2024-03-24T12:06:47.373 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T12:06:47.373 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999 --verbose --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi --description orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999/orig.config.yaml 2024-03-24T12:06:47.408 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T12:06:47.507 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999 branch: squid description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617999' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr1: false ms bind msgr2: true ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCL40OM2DBOoWWg9YLz+PMTjeo1NtoyPYRIzlHs+Xs6di299FV0gjNRv8vZznlwC129g1h+CqMQEaWtG5tcPhPM= smithi190.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDemyXDXAATj1s5F0ACac++1oAZMyNmqRLmaZbHcx6bjd1Fy2d+wcDa4wN6WmHh17bY4eAAEiWlxYgPOo1/zppo= tasks: - pexec: all: - sudo dnf install runc -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-24T12:06:47.507 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T12:06:47.508 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T12:06:47.508 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T12:06:47.510 INFO:teuthology.task.internal:Checking packages... 2024-03-24T12:06:47.533 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T12:06:47.533 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T12:06:47.533 INFO:teuthology.packaging:ref: None 2024-03-24T12:06:47.533 INFO:teuthology.packaging:tag: None 2024-03-24T12:06:47.533 INFO:teuthology.packaging:branch: squid 2024-03-24T12:06:47.533 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T12:06:47.534 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T12:06:47.685 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T12:06:47.686 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T12:06:47.693 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T12:06:47.693 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T12:06:47.737 INFO:teuthology.task.internal:Saving configuration 2024-03-24T12:06:47.750 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T12:06:47.763 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:06:47.786 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 12:00:39.711460', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCL40OM2DBOoWWg9YLz+PMTjeo1NtoyPYRIzlHs+Xs6di299FV0gjNRv8vZznlwC129g1h+CqMQEaWtG5tcPhPM='} 2024-03-24T12:06:47.804 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 12:00:39.713185', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDemyXDXAATj1s5F0ACac++1oAZMyNmqRLmaZbHcx6bjd1Fy2d+wcDa4wN6WmHh17bY4eAAEiWlxYgPOo1/zppo='} 2024-03-24T12:06:47.804 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T12:06:47.811 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-24T12:06:47.811 INFO:teuthology.task.internal:roles: ubuntu@smithi190.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-24T12:06:47.811 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T12:06:47.891 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb1294770d0>, signals=[15]) 2024-03-24T12:06:47.893 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T12:06:47.901 INFO:teuthology.task.internal:Opening connections... 2024-03-24T12:06:47.901 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2024-03-24T12:06:47.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:06:48.009 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi190.front.sepia.ceph.com 2024-03-24T12:06:48.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:06:48.099 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T12:06:48.105 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2024-03-24T12:06:48.123 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2024-03-24T12:06:48.123 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2024-03-24T12:06:48.179 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2024-03-24T12:06:48.179 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2024-03-24T12:06:48.179 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2024-03-24T12:06:48.179 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2024-03-24T12:06:48.179 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2024-03-24T12:06:48.179 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2024-03-24T12:06:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T12:06:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2024-03-24T12:06:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2024-03-24T12:06:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T12:06:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2024-03-24T12:06:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T12:06:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T12:06:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T12:06:48.181 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2024-03-24T12:06:48.210 DEBUG:teuthology.orchestra.run.smithi190:> uname -m 2024-03-24T12:06:48.227 INFO:teuthology.orchestra.run.smithi190.stdout:x86_64 2024-03-24T12:06:48.227 DEBUG:teuthology.orchestra.run.smithi190:> cat /etc/os-release 2024-03-24T12:06:48.283 INFO:teuthology.orchestra.run.smithi190.stdout:NAME="CentOS Stream" 2024-03-24T12:06:48.283 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION="9" 2024-03-24T12:06:48.283 INFO:teuthology.orchestra.run.smithi190.stdout:ID="centos" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:ID_LIKE="rhel fedora" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:VERSION_ID="9" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:PLATFORM_ID="platform:el9" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:ANSI_COLOR="0;31" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:LOGO="fedora-logo-icon" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:HOME_URL="https://centos.org/" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T12:06:48.284 INFO:teuthology.orchestra.run.smithi190.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T12:06:48.284 INFO:teuthology.lock.ops:Updating smithi190.front.sepia.ceph.com on lock server 2024-03-24T12:06:48.318 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T12:06:48.327 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T12:06:48.346 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T12:06:48.346 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2024-03-24T12:06:48.349 DEBUG:teuthology.orchestra.run.smithi190:> test '!' -e /home/ubuntu/cephtest 2024-03-24T12:06:48.365 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T12:06:48.371 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T12:06:48.371 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2024-03-24T12:06:48.406 DEBUG:teuthology.orchestra.run.smithi190:> test -z $(ls -A /var/lib/ceph) 2024-03-24T12:06:48.432 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T12:06:48.479 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T12:06:48.493 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T12:06:48.493 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T12:06:48.493 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T12:06:48.494 DEBUG:teuthology.orchestra.run.smithi190:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T12:06:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:48.509 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-03-24T12:06:48.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:06:48.511 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-03-24T12:06:48.565 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-361.el9.x86_64 2024-03-24T12:06:48.565 INFO:teuthology.task.kernel:Running kernel on smithi190: 5.14.0-361.el9.x86_64 2024-03-24T12:06:48.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum install -y kernel 2024-03-24T12:06:48.569 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2024-03-24T12:06:48.569 INFO:teuthology.task.kernel:Running kernel on smithi029: 5.14.0-361.el9.x86_64 2024-03-24T12:06:48.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum install -y kernel 2024-03-24T12:06:50.712 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T12:06:51.868 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-24T12:06:52.104 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 6.9 MB/s | 2.0 MB 00:00 2024-03-24T12:06:53.105 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T12:06:53.527 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-03-24T12:06:55.004 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-24T12:06:55.029 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 7.0 MB/s | 8.0 MB 00:01 2024-03-24T12:06:59.088 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.1 MB 00:00 2024-03-24T12:07:00.514 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 16 kB 00:00 2024-03-24T12:07:01.365 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 28 MB/s | 21 MB 00:00 2024-03-24T12:07:02.904 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 2.8 MB/s | 19 MB 00:06 2024-03-24T12:07:06.365 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 6.4 kB/s | 1.7 kB 00:00 2024-03-24T12:07:07.258 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.1 MB 00:00 2024-03-24T12:07:08.455 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T12:07:08.456 INFO:teuthology.orchestra.run.smithi190.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T12:07:08.538 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:07:08.546 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:07:08.546 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-24T12:07:08.546 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:07:08.546 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:07:08.546 INFO:teuthology.orchestra.run.smithi190.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T12:07:08.546 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-24T12:07:08.546 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T12:07:08.546 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T12:07:08.547 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T12:07:08.547 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:07:08.547 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:07:08.547 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:07:08.547 INFO:teuthology.orchestra.run.smithi190.stdout:Install 4 Packages 2024-03-24T12:07:08.547 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:07:08.547 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 76 M 2024-03-24T12:07:08.548 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 123 M 2024-03-24T12:07:08.548 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:07:08.734 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-03-24T12:07:08.808 INFO:teuthology.orchestra.run.smithi190.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 321 kB/s | 44 kB 00:00 2024-03-24T12:07:15.833 INFO:teuthology.orchestra.run.smithi190.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 2.1 MB/s | 15 MB 00:07 2024-03-24T12:07:34.622 INFO:teuthology.orchestra.run.smithi190.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 1.3 MB/s | 33 MB 00:25 2024-03-24T12:07:40.765 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 664 kB/s | 21 MB 00:31 2024-03-24T12:07:41.055 INFO:teuthology.orchestra.run.smithi190.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 872 kB/s | 27 MB 00:32 2024-03-24T12:07:41.056 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:07:41.056 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.3 MB/s | 76 MB 00:32 2024-03-24T12:07:41.718 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:07:42.088 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:07:42.088 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:07:43.108 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:07:43.108 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:07:45.583 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:07:45.838 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-24T12:07:46.244 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T12:07:46.330 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T12:07:48.063 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T12:07:48.063 INFO:teuthology.orchestra.run.smithi029.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T12:07:48.169 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-24T12:07:48.184 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout:Install 4 Packages 2024-03-24T12:07:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 76 M 2024-03-24T12:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 123 M 2024-03-24T12:07:48.186 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:07:48.424 INFO:teuthology.orchestra.run.smithi029.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 229 kB/s | 44 kB 00:00 2024-03-24T12:07:49.063 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T12:07:49.199 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T12:07:53.900 INFO:teuthology.orchestra.run.smithi029.stdout:(2/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 5.9 MB/s | 33 MB 00:05 2024-03-24T12:07:53.917 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T12:07:53.957 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:07:54.383 INFO:teuthology.orchestra.run.smithi029.stdout:(3/4): kernel-core-5.14.0-430.el9.x86_64.rpm 2.4 MB/s | 15 MB 00:06 2024-03-24T12:07:58.617 INFO:teuthology.orchestra.run.smithi029.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 2.7 MB/s | 27 MB 00:10 2024-03-24T12:07:58.617 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:07:58.617 INFO:teuthology.orchestra.run.smithi029.stdout:Total 7.3 MB/s | 76 MB 00:10 2024-03-24T12:07:58.688 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:07:59.306 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:07:59.755 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:07:59.755 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:08:00.780 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:08:00.780 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:08:03.393 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:08:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T12:08:04.222 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T12:08:07.082 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T12:08:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T12:08:12.355 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T12:08:12.413 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:08:17.504 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:08:27.554 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:08:27.554 INFO:teuthology.orchestra.run.smithi190.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T12:08:27.554 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:08:27.599 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:08:27.987 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:08:27.987 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T12:08:27.987 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T12:08:27.987 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T12:08:28.406 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:08:28.407 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:08:28.407 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:08:28.407 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T12:08:28.407 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T12:08:28.407 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T12:08:28.407 INFO:teuthology.orchestra.run.smithi190.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T12:08:28.407 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:08:28.407 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:08:28.752 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T12:08:28.868 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T12:08:28.868 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T12:08:28.868 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T12:08:28.868 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T12:08:28.869 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T12:08:28.869 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-24T12:08:28.869 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi190.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T12:08:28.869 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T12:08:28.947 INFO:teuthology.orchestra.run.smithi190.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T12:08:28.947 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T12:08:28.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo rpm -qi grub2-tools 2024-03-24T12:08:28.981 INFO:teuthology.orchestra.run.smithi190.stdout:Name : grub2-tools 2024-03-24T12:08:28.981 INFO:teuthology.orchestra.run.smithi190.stdout:Epoch : 1 2024-03-24T12:08:28.981 INFO:teuthology.orchestra.run.smithi190.stdout:Version : 2.06 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Release : 68.el9 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Architecture: x86_64 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Group : Unspecified 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Size : 8264254 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:License : GPLv3+ 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Packager : builder@centos.org 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:Vendor : CentOS 2024-03-24T12:08:28.982 INFO:teuthology.orchestra.run.smithi190.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T12:08:28.983 INFO:teuthology.orchestra.run.smithi190.stdout:Summary : Support tools for GRUB. 2024-03-24T12:08:28.983 INFO:teuthology.orchestra.run.smithi190.stdout:Description : 2024-03-24T12:08:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:08:28.983 INFO:teuthology.orchestra.run.smithi190.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T12:08:28.983 INFO:teuthology.orchestra.run.smithi190.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T12:08:28.983 INFO:teuthology.orchestra.run.smithi190.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T12:08:28.983 INFO:teuthology.orchestra.run.smithi190.stdout:hardware devices. 2024-03-24T12:08:28.983 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:08:28.983 INFO:teuthology.orchestra.run.smithi190.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T12:08:28.984 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T12:08:28.984 INFO:teuthology.task.kernel:Updating grub on smithi190 to boot 5.14.0-430.el9.x86_64 2024-03-24T12:08:28.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T12:08:29.597 INFO:teuthology.orchestra.run.smithi190.stderr:Generating grub configuration file ... 2024-03-24T12:08:31.093 INFO:teuthology.orchestra.run.smithi190.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T12:08:31.127 INFO:teuthology.orchestra.run.smithi190.stderr:done 2024-03-24T12:08:31.129 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-03-24T12:08:31.144 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.68hku3s5F0 2024-03-24T12:08:31.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.68hku3s5F0 2024-03-24T12:08:31.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.68hku3s5F0 2024-03-24T12:08:31.356 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.68hku3s5F0 is 6KB 2024-03-24T12:08:31.405 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.68hku3s5F0 2024-03-24T12:08:31.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T12:08:31.483 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T12:08:31.483 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T12:08:31.484 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T12:08:31.484 INFO:teuthology.orchestra.run.smithi190.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T12:08:31.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T12:08:31.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo shutdown -r now 2024-03-24T12:08:50.647 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:08:50.647 INFO:teuthology.orchestra.run.smithi029.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T12:08:50.648 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:08:50.693 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:08:51.136 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:08:51.136 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T12:08:51.136 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T12:08:51.136 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T12:08:51.588 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T12:08:51.588 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:08:51.588 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:08:51.588 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T12:08:51.588 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T12:08:51.589 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T12:08:51.589 INFO:teuthology.orchestra.run.smithi029.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T12:08:51.589 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:08:51.589 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:08:52.016 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T12:08:52.130 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T12:08:52.131 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T12:08:52.131 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T12:08:52.131 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T12:08:52.131 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T12:08:52.131 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-24T12:08:52.131 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T12:08:52.131 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T12:08:52.213 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T12:08:52.214 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T12:08:52.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2024-03-24T12:08:52.251 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2024-03-24T12:08:52.251 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2024-03-24T12:08:52.251 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2024-03-24T12:08:52.251 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2024-03-24T12:08:52.251 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2024-03-24T12:08:52.251 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T12:08:52.251 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2024-03-24T12:08:52.251 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2024-03-24T12:08:52.251 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2024-03-24T12:08:52.251 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T12:08:52.252 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T12:08:52.253 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2024-03-24T12:08:52.253 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:08:52.253 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T12:08:52.253 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T12:08:52.254 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 5.14.0-430.el9.x86_64 2024-03-24T12:08:52.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T12:08:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2024-03-24T12:08:54.668 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T12:08:54.693 INFO:teuthology.orchestra.run.smithi029.stderr:done 2024-03-24T12:08:54.695 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2024-03-24T12:08:54.711 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.MdU6VEDTGs 2024-03-24T12:08:54.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MdU6VEDTGs 2024-03-24T12:08:54.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.MdU6VEDTGs 2024-03-24T12:08:54.932 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.MdU6VEDTGs is 6KB 2024-03-24T12:08:54.953 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.MdU6VEDTGs 2024-03-24T12:08:54.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T12:08:55.042 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T12:08:55.042 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T12:08:55.042 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T12:08:55.042 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T12:08:55.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T12:08:55.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2024-03-24T12:09:01.634 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-24T12:09:01.634 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-24T12:09:01.635 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:09:20.158 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-03-24T12:09:25.204 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-24T12:09:25.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-24T12:09:25.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:09:29.164 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-24T12:09:29.164 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:09:32.222 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.190 2024-03-24T12:09:43.710 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2024-03-24T12:09:44.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-24T12:09:44.224 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:09:52.718 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-24T12:09:52.719 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:10:16.130 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-03-24T12:10:16.712 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-24T12:10:16.712 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T12:10:16.712 DEBUG:teuthology.orchestra.run.smithi190:> uname -r 2024-03-24T12:10:16.726 INFO:teuthology.orchestra.run.smithi190.stdout:5.14.0-430.el9.x86_64 2024-03-24T12:10:16.727 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T12:10:16.727 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T12:10:16.727 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T12:10:17.728 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-24T12:10:17.729 DEBUG:teuthology.orchestra.run.smithi190:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T12:10:17.815 INFO:teuthology.orchestra.run.smithi190.stdout:ttyS1 2024-03-24T12:10:17.847 DEBUG:teuthology.parallel:result is None 2024-03-24T12:10:52.722 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T12:11:04.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-24T12:11:04.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:11:05.107 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-24T12:11:05.528 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-24T12:11:05.529 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T12:11:05.529 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2024-03-24T12:11:05.544 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-430.el9.x86_64 2024-03-24T12:11:05.544 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T12:11:05.544 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T12:11:05.545 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T12:11:06.546 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-24T12:11:06.546 DEBUG:teuthology.orchestra.run.smithi029:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T12:11:06.636 INFO:teuthology.orchestra.run.smithi029.stdout:ttyS1 2024-03-24T12:11:06.675 DEBUG:teuthology.parallel:result is None 2024-03-24T12:11:06.676 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T12:11:06.684 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T12:11:06.685 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T12:11:06.687 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T12:11:06.707 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T12:11:06.732 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T12:11:06.767 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T12:11:06.768 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T12:11:06.770 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T12:11:06.802 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T12:11:06.811 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T12:11:06.811 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T12:11:06.832 DEBUG:teuthology.orchestra.run.smithi190:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T12:11:06.866 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T12:11:06.878 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T12:11:06.880 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T12:11:06.892 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T12:11:06.893 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T12:11:06.900 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T12:11:06.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T12:11:06.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T12:11:06.964 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T12:11:06.974 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T12:11:06.974 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T12:11:06.994 DEBUG:teuthology.orchestra.run.smithi190:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T12:11:07.022 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T12:11:07.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T12:11:07.143 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T12:11:07.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T12:11:07.272 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:11:07.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T12:11:07.341 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T12:11:07.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T12:11:07.444 DEBUG:teuthology.orchestra.run.smithi190:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T12:11:07.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T12:11:07.572 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:11:07.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T12:11:07.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2024-03-24T12:11:07.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo service rsyslog restart 2024-03-24T12:11:07.688 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T12:11:07.742 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T12:11:08.179 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T12:11:08.187 INFO:teuthology.task.internal:Starting timer... 2024-03-24T12:11:08.187 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T12:11:08.212 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T12:11:08.243 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-24T12:11:08.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2024-03-24T12:11:08.323 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2024-03-24T12:11:08.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo service auditd rotate 2024-03-24T12:11:08.382 INFO:teuthology.orchestra.run.smithi190.stdout:Rotating logs: 2024-03-24T12:11:08.385 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T12:11:08.385 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2024-03-24T12:11:08.409 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2024-03-24T12:11:08.409 DEBUG:teuthology.orchestra.run.smithi190:> /usr/sbin/getenforce 2024-03-24T12:11:08.443 INFO:teuthology.orchestra.run.smithi190.stdout:Permissive 2024-03-24T12:11:08.443 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi190.front.sepia.ceph.com': 'permissive'} 2024-03-24T12:11:08.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T12:11:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:08.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T12:11:08.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:11:08.512 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T12:11:08.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2024-03-24T12:11:08.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo /usr/sbin/setenforce permissive 2024-03-24T12:11:08.587 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T12:11:08.597 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T12:11:08.613 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T12:11:08.615 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi190.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T12:15:35.913 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi190.front.sepia.ceph.com')] 2024-03-24T12:15:35.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-24T12:15:35.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:15:36.007 DEBUG:teuthology.orchestra.run.smithi029:> true 2024-03-24T12:15:36.092 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2024-03-24T12:15:36.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-24T12:15:36.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi190.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:15:36.182 DEBUG:teuthology.orchestra.run.smithi190:> true 2024-03-24T12:15:36.261 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi190.front.sepia.ceph.com' 2024-03-24T12:15:36.261 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T12:15:36.272 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T12:15:36.272 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T12:15:36.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T12:15:36.275 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T12:15:36.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T12:15:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T12:15:36.320 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T12:15:36.333 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T12:15:36.349 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2024-03-24T12:15:36.350 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T12:15:36.363 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2024-03-24T12:15:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T12:15:36.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: ntpd: command not found 2024-03-24T12:15:36.393 INFO:teuthology.orchestra.run.smithi190.stdout:506 Cannot talk to daemon 2024-03-24T12:15:36.396 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T12:15:36.410 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T12:15:36.426 INFO:teuthology.orchestra.run.smithi190.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T12:15:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-03-24T12:15:36.456 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T12:15:36.456 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-03-24T12:15:36.457 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:15:36.457 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:15:36.457 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:15:36.457 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:15:36.479 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-03-24T12:15:36.483 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T12:15:36.483 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-03-24T12:15:36.483 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:15:36.483 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:15:36.483 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:15:36.483 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T12:15:36.483 INFO:teuthology.run_tasks:Running task pexec... 2024-03-24T12:15:36.493 INFO:teuthology.task.pexec:Executing custom commands... 2024-03-24T12:15:36.494 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi029.front.sepia.ceph.com 2024-03-24T12:15:36.494 DEBUG:teuthology.orchestra.run.smithi029:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T12:15:36.494 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi190.front.sepia.ceph.com 2024-03-24T12:15:36.494 DEBUG:teuthology.orchestra.run.smithi190:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-03-24T12:15:37.088 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:48 ago on Sun 24 Mar 2024 12:14:48 PM UTC. 2024-03-24T12:15:37.108 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:49 ago on Sun 24 Mar 2024 12:14:47 PM UTC. 2024-03-24T12:15:37.239 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:15:37.254 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:15:37.261 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:15:37.261 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-24T12:15:37.261 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:15:37.261 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:15:37.261 INFO:teuthology.orchestra.run.smithi029.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T12:15:37.261 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:15:37.261 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:15:37.261 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:15:37.261 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-24T12:15:37.261 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:15:37.266 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:15:37.266 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-24T12:15:37.266 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:15:37.266 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:15:37.266 INFO:teuthology.orchestra.run.smithi190.stdout: runc x86_64 4:1.1.12-2.el9 appstream 3.1 M 2024-03-24T12:15:37.266 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:15:37.267 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:15:37.267 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:15:37.267 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-24T12:15:37.267 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:15:37.272 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.1 M 2024-03-24T12:15:37.272 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 9.8 M 2024-03-24T12:15:37.272 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:15:37.273 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.1 M 2024-03-24T12:15:37.273 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 9.8 M 2024-03-24T12:15:37.273 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:15:37.622 INFO:teuthology.orchestra.run.smithi029.stdout:runc-1.1.12-2.el9.x86_64.rpm 14 MB/s | 3.1 MB 00:00 2024-03-24T12:15:37.622 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:15:37.622 INFO:teuthology.orchestra.run.smithi029.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-03-24T12:15:37.659 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:15:37.665 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:15:37.665 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:15:37.747 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:15:37.748 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:15:37.984 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:15:38.087 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T12:15:38.657 INFO:teuthology.orchestra.run.smithi190.stdout:runc-1.1.12-2.el9.x86_64.rpm 2.8 MB/s | 3.1 MB 00:01 2024-03-24T12:15:38.657 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:15:38.657 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.3 MB/s | 3.1 MB 00:01 2024-03-24T12:15:38.694 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:15:38.700 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:15:38.700 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:15:38.782 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:15:38.783 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:15:38.807 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T12:15:38.994 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:15:39.094 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T12:15:39.127 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T12:15:39.128 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:15:39.128 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:15:39.128 INFO:teuthology.orchestra.run.smithi029.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T12:15:39.128 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:15:39.128 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:15:39.421 DEBUG:teuthology.parallel:result is None 2024-03-24T12:15:39.731 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T12:15:40.031 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : runc-4:1.1.12-2.el9.x86_64 1/1 2024-03-24T12:15:40.031 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:15:40.031 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:15:40.031 INFO:teuthology.orchestra.run.smithi190.stdout: runc-4:1.1.12-2.el9.x86_64 2024-03-24T12:15:40.031 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:15:40.031 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:15:40.294 DEBUG:teuthology.parallel:result is None 2024-03-24T12:15:40.294 INFO:teuthology.run_tasks:Running task install... 2024-03-24T12:15:40.304 DEBUG:teuthology.task.install:project ceph 2024-03-24T12:15:40.304 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}} 2024-03-24T12:15:40.304 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T12:15:40.304 INFO:teuthology.task.install:Using flavor: default 2024-03-24T12:15:40.309 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T12:15:40.310 INFO:teuthology.task.install:extra packages: [] 2024-03-24T12:15:40.310 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'tag': None, 'wait_for_package': False} 2024-03-24T12:15:40.310 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T12:15:40.313 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'tag': None, 'wait_for_package': False} 2024-03-24T12:15:40.313 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T12:15:40.458 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/ 2024-03-24T12:15:40.458 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1667.gdb0330b1 2024-03-24T12:15:40.542 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/ 2024-03-24T12:15:40.543 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1667.gdb0330b1 2024-03-24T12:15:40.566 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T12:15:40.566 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:15:40.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T12:15:40.598 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T12:15:40.598 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/db0330b1e4e2470d52b750e251e55a522b4f7d69/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T12:15:40.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T12:15:40.713 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T12:15:40.714 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:15:40.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T12:15:40.742 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T12:15:40.742 DEBUG:teuthology.orchestra.run.smithi190:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/db0330b1e4e2470d52b750e251e55a522b4f7d69/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T12:15:40.752 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T12:15:40.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T12:15:40.819 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2024-03-24T12:15:40.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2024-03-24T12:15:40.890 DEBUG:teuthology.orchestra.run.smithi190:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T12:15:40.957 INFO:teuthology.orchestra.run.smithi190.stdout:check_obsoletes = 1 2024-03-24T12:15:40.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum clean all 2024-03-24T12:15:41.072 INFO:teuthology.orchestra.run.smithi029.stdout:74 files removed 2024-03-24T12:15:41.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw 2024-03-24T12:15:41.194 INFO:teuthology.orchestra.run.smithi190.stdout:74 files removed 2024-03-24T12:15:41.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-radosgw 2024-03-24T12:15:41.836 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 189 kB/s | 82 kB 00:00 2024-03-24T12:15:41.978 INFO:teuthology.orchestra.run.smithi190.stdout:ceph packages for x86_64 180 kB/s | 82 kB 00:00 2024-03-24T12:15:42.370 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-24T12:15:42.603 INFO:teuthology.orchestra.run.smithi190.stdout:ceph noarch packages 17 kB/s | 10 kB 00:00 2024-03-24T12:15:43.105 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-03-24T12:15:43.471 INFO:teuthology.orchestra.run.smithi190.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-24T12:15:43.750 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-24T12:15:43.981 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T12:15:45.319 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-24T12:15:45.365 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS-9 - BaseOS 7.5 MB/s | 2.0 MB 00:00 2024-03-24T12:15:46.228 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-03-24T12:15:46.964 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - BaseOS 6.3 MB/s | 8.0 MB 00:01 2024-03-24T12:15:48.972 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-24T12:15:49.401 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-24T12:15:53.193 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 9.5 MB/s | 6.1 MB 00:00 2024-03-24T12:15:53.932 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - CRB 5.6 MB/s | 6.1 MB 00:01 2024-03-24T12:15:54.644 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-03-24T12:15:55.818 INFO:teuthology.orchestra.run.smithi190.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-24T12:15:57.431 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 7.8 MB/s | 21 MB 00:02 2024-03-24T12:15:58.058 INFO:teuthology.orchestra.run.smithi190.stdout:Extra Packages for Enterprise Linux 9.7 MB/s | 21 MB 00:02 2024-03-24T12:16:02.592 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-03-24T12:16:03.101 INFO:teuthology.orchestra.run.smithi190.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-24T12:16:05.243 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:16:05.245 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:16:05.246 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-24T12:16:05.246 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:16:05.246 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:16:05.246 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 9.8 M 2024-03-24T12:16:05.246 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2024-03-24T12:16:05.247 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.4 M 2024-03-24T12:16:05.247 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T12:16:05.247 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-24T12:16:05.247 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options 2024-03-24T12:16:05.247 INFO:teuthology.orchestra.run.smithi029.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T12:16:05.247 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.3 M 2024-03-24T12:16:05.247 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 20 M 2024-03-24T12:16:05.247 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 25 k 2024-03-24T12:16:05.248 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T12:16:05.248 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T12:16:05.248 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T12:16:05.248 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 742 k 2024-03-24T12:16:05.248 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T12:16:05.248 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T12:16:05.249 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T12:16:05.249 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 498 k 2024-03-24T12:16:05.249 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T12:16:05.249 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.1 M 2024-03-24T12:16:05.249 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T12:16:05.249 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T12:16:05.249 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T12:16:05.249 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 45 k 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 131 k 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 161 k 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 320 k 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 298 k 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 100 k 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T12:16:05.250 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T12:16:05.251 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2024-03-24T12:16:05.251 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T12:16:05.251 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:05.251 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:16:05.251 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:16:05.251 INFO:teuthology.orchestra.run.smithi029.stdout:Install 30 Packages 2024-03-24T12:16:05.251 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 2 Packages 2024-03-24T12:16:05.251 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:05.252 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 58 M 2024-03-24T12:16:05.252 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:16:05.491 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:16:05.493 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:16:05.494 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-24T12:16:05.494 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:16:05.494 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:16:05.494 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 9.8 M 2024-03-24T12:16:05.494 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrading: 2024-03-24T12:16:05.494 INFO:teuthology.orchestra.run.smithi190.stdout: librados2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.4 M 2024-03-24T12:16:05.494 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T12:16:05.494 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-24T12:16:05.494 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options 2024-03-24T12:16:05.494 INFO:teuthology.orchestra.run.smithi190.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T12:16:05.495 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.3 M 2024-03-24T12:16:05.495 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 20 M 2024-03-24T12:16:05.495 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 25 k 2024-03-24T12:16:05.495 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T12:16:05.495 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T12:16:05.495 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T12:16:05.495 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 742 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 498 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2 x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 5.1 M 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 45 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 131 k 2024-03-24T12:16:05.496 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 161 k 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 320 k 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 298 k 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 100 k 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout:Installing weak dependencies: 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:16:05.497 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:16:05.498 INFO:teuthology.orchestra.run.smithi190.stdout:Install 30 Packages 2024-03-24T12:16:05.498 INFO:teuthology.orchestra.run.smithi190.stdout:Upgrade 2 Packages 2024-03-24T12:16:05.498 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:05.498 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 58 M 2024-03-24T12:16:05.498 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:16:06.328 INFO:teuthology.orchestra.run.smithi029.stdout:(1/32): ceph-base-19.0.0-1667.gdb0330b1.el9.x86 5.4 MB/s | 5.3 MB 00:00 2024-03-24T12:16:06.429 INFO:teuthology.orchestra.run.smithi029.stdout:(2/32): ceph-selinux-19.0.0-1667.gdb0330b1.el9. 251 kB/s | 25 kB 00:00 2024-03-24T12:16:06.629 INFO:teuthology.orchestra.run.smithi029.stdout:(3/32): libcephfs2-19.0.0-1667.gdb0330b1.el9.x8 3.6 MB/s | 742 kB 00:00 2024-03-24T12:16:06.830 INFO:teuthology.orchestra.run.smithi029.stdout:(4/32): libradosstriper1-19.0.0-1667.gdb0330b1. 2.4 MB/s | 498 kB 00:00 2024-03-24T12:16:07.281 INFO:teuthology.orchestra.run.smithi029.stdout:(5/32): ceph-common-19.0.0-1667.gdb0330b1.el9.x 10 MB/s | 20 MB 00:01 2024-03-24T12:16:07.364 INFO:teuthology.orchestra.run.smithi029.stdout:(6/32): python3-ceph-argparse-19.0.0-1667.gdb03 541 kB/s | 45 kB 00:00 2024-03-24T12:16:07.448 INFO:teuthology.orchestra.run.smithi029.stdout:(7/32): python3-ceph-common-19.0.0-1667.gdb0330 1.5 MB/s | 131 kB 00:00 2024-03-24T12:16:07.532 INFO:teuthology.orchestra.run.smithi029.stdout:(8/32): python3-cephfs-19.0.0-1667.gdb0330b1.el 1.9 MB/s | 161 kB 00:00 2024-03-24T12:16:07.759 INFO:teuthology.orchestra.run.smithi029.stdout:(9/32): ceph-radosgw-19.0.0-1667.gdb0330b1.el9. 4.0 MB/s | 9.8 MB 00:02 2024-03-24T12:16:07.876 INFO:teuthology.orchestra.run.smithi029.stdout:(10/32): librgw2-19.0.0-1667.gdb0330b1.el9.x86_ 4.9 MB/s | 5.1 MB 00:01 2024-03-24T12:16:07.910 INFO:teuthology.orchestra.run.smithi029.stdout:(11/32): python3-rados-19.0.0-1667.gdb0330b1.el 847 kB/s | 320 kB 00:00 2024-03-24T12:16:07.944 INFO:teuthology.orchestra.run.smithi029.stdout:(12/32): python3-rbd-19.0.0-1667.gdb0330b1.el9. 1.6 MB/s | 298 kB 00:00 2024-03-24T12:16:07.969 INFO:teuthology.orchestra.run.smithi029.stdout:(13/32): python3-rgw-19.0.0-1667.gdb0330b1.el9. 1.0 MB/s | 100 kB 00:00 2024-03-24T12:16:08.095 INFO:teuthology.orchestra.run.smithi029.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 352 kB/s | 44 kB 00:00 2024-03-24T12:16:08.129 INFO:teuthology.orchestra.run.smithi029.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 863 kB/s | 159 kB 00:00 2024-03-24T12:16:08.188 INFO:teuthology.orchestra.run.smithi029.stdout:(16/32): boost-program-options-1.75.0-8.el9.x86 435 kB/s | 106 kB 00:00 2024-03-24T12:16:08.272 INFO:teuthology.orchestra.run.smithi029.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2024-03-24T12:16:08.306 INFO:teuthology.orchestra.run.smithi029.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 282 kB 00:00 2024-03-24T12:16:08.331 INFO:teuthology.orchestra.run.smithi029.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 289 kB/s | 41 kB 00:00 2024-03-24T12:16:08.443 INFO:teuthology.orchestra.run.smithi190.stdout:(1/32): ceph-common-19.0.0-1667.gdb0330b1.el9.x 7.0 MB/s | 20 MB 00:02 2024-03-24T12:16:08.465 INFO:teuthology.orchestra.run.smithi029.stdout:(20/32): qatlib-service-23.11.0-1.el9.x86_64.rp 261 kB/s | 35 kB 00:00 2024-03-24T12:16:08.490 INFO:teuthology.orchestra.run.smithi029.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 280 kB/s | 44 kB 00:00 2024-03-24T12:16:08.494 INFO:teuthology.orchestra.run.smithi190.stdout:(2/32): ceph-selinux-19.0.0-1667.gdb0330b1.el9. 506 kB/s | 25 kB 00:00 2024-03-24T12:16:08.558 INFO:teuthology.orchestra.run.smithi029.stdout:(22/32): qatlib-23.11.0-1.el9.x86_64.rpm 759 kB/s | 216 kB 00:00 2024-03-24T12:16:08.577 INFO:teuthology.orchestra.run.smithi190.stdout:(3/32): libcephfs2-19.0.0-1667.gdb0330b1.el9.x8 8.7 MB/s | 742 kB 00:00 2024-03-24T12:16:08.642 INFO:teuthology.orchestra.run.smithi029.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-03-24T12:16:08.644 INFO:teuthology.orchestra.run.smithi190.stdout:(4/32): libradosstriper1-19.0.0-1667.gdb0330b1. 7.3 MB/s | 498 kB 00:00 2024-03-24T12:16:08.667 INFO:teuthology.orchestra.run.smithi029.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.5 MB/s | 309 kB 00:00 2024-03-24T12:16:08.719 INFO:teuthology.orchestra.run.smithi029.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-24T12:16:08.745 INFO:teuthology.orchestra.run.smithi029.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 472 kB/s | 49 kB 00:00 2024-03-24T12:16:08.887 INFO:teuthology.orchestra.run.smithi029.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.9 MB/s | 838 kB 00:00 2024-03-24T12:16:08.912 INFO:teuthology.orchestra.run.smithi029.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-24T12:16:08.953 INFO:teuthology.orchestra.run.smithi190.stdout:(5/32): librgw2-19.0.0-1667.gdb0330b1.el9.x86_6 17 MB/s | 5.1 MB 00:00 2024-03-24T12:16:09.003 INFO:teuthology.orchestra.run.smithi190.stdout:(6/32): python3-ceph-argparse-19.0.0-1667.gdb03 909 kB/s | 45 kB 00:00 2024-03-24T12:16:09.053 INFO:teuthology.orchestra.run.smithi190.stdout:(7/32): python3-ceph-common-19.0.0-1667.gdb0330 2.6 MB/s | 131 kB 00:00 2024-03-24T12:16:09.103 INFO:teuthology.orchestra.run.smithi190.stdout:(8/32): python3-cephfs-19.0.0-1667.gdb0330b1.el 3.2 MB/s | 161 kB 00:00 2024-03-24T12:16:09.146 INFO:teuthology.orchestra.run.smithi029.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-03-24T12:16:09.154 INFO:teuthology.orchestra.run.smithi190.stdout:(9/32): python3-rados-19.0.0-1667.gdb0330b1.el9 6.3 MB/s | 320 kB 00:00 2024-03-24T12:16:09.247 INFO:teuthology.orchestra.run.smithi190.stdout:(10/32): ceph-base-19.0.0-1667.gdb0330b1.el9.x8 1.5 MB/s | 5.3 MB 00:03 2024-03-24T12:16:09.271 INFO:teuthology.orchestra.run.smithi190.stdout:(11/32): python3-rbd-19.0.0-1667.gdb0330b1.el9. 2.5 MB/s | 298 kB 00:00 2024-03-24T12:16:09.338 INFO:teuthology.orchestra.run.smithi190.stdout:(12/32): python3-rgw-19.0.0-1667.gdb0330b1.el9. 1.1 MB/s | 100 kB 00:00 2024-03-24T12:16:09.397 INFO:teuthology.orchestra.run.smithi190.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 847 kB/s | 106 kB 00:00 2024-03-24T12:16:09.422 INFO:teuthology.orchestra.run.smithi190.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-24T12:16:09.447 INFO:teuthology.orchestra.run.smithi190.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-24T12:16:09.456 INFO:teuthology.orchestra.run.smithi029.stdout:(30/32): librados2-19.0.0-1667.gdb0330b1.el9.x8 6.2 MB/s | 3.4 MB 00:00 2024-03-24T12:16:09.481 INFO:teuthology.orchestra.run.smithi190.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-24T12:16:09.515 INFO:teuthology.orchestra.run.smithi190.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.2 MB/s | 282 kB 00:00 2024-03-24T12:16:09.540 INFO:teuthology.orchestra.run.smithi190.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 709 kB/s | 41 kB 00:00 2024-03-24T12:16:09.631 INFO:teuthology.orchestra.run.smithi029.stdout:(31/32): librbd1-19.0.0-1667.gdb0330b1.el9.x86_ 6.5 MB/s | 3.1 MB 00:00 2024-03-24T12:16:09.749 INFO:teuthology.orchestra.run.smithi190.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 167 kB/s | 35 kB 00:00 2024-03-24T12:16:09.773 INFO:teuthology.orchestra.run.smithi029.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-03-24T12:16:09.774 INFO:teuthology.orchestra.run.smithi190.stdout:(20/32): qatlib-23.11.0-1.el9.x86_64.rpm 837 kB/s | 216 kB 00:00 2024-03-24T12:16:09.778 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:16:09.778 INFO:teuthology.orchestra.run.smithi029.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-24T12:16:09.807 INFO:teuthology.orchestra.run.smithi190.stdout:(21/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 764 kB/s | 44 kB 00:00 2024-03-24T12:16:09.812 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:16:09.853 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:16:09.853 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:16:10.016 INFO:teuthology.orchestra.run.smithi190.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-24T12:16:10.058 INFO:teuthology.orchestra.run.smithi190.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 625 kB/s | 26 kB 00:00 2024-03-24T12:16:10.108 INFO:teuthology.orchestra.run.smithi190.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 972 kB/s | 49 kB 00:00 2024-03-24T12:16:10.200 INFO:teuthology.orchestra.run.smithi190.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-03-24T12:16:10.225 INFO:teuthology.orchestra.run.smithi190.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 575 kB/s | 67 kB 00:00 2024-03-24T12:16:10.334 INFO:teuthology.orchestra.run.smithi190.stdout:(27/32): ceph-radosgw-19.0.0-1667.gdb0330b1.el9 2.1 MB/s | 9.8 MB 00:04 2024-03-24T12:16:10.359 INFO:teuthology.orchestra.run.smithi190.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.4 MB/s | 192 kB 00:00 2024-03-24T12:16:10.362 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:16:10.363 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:16:10.393 INFO:teuthology.orchestra.run.smithi190.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 838 kB 00:00 2024-03-24T12:16:10.677 INFO:teuthology.orchestra.run.smithi190.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.6 MB/s | 1.6 MB 00:00 2024-03-24T12:16:10.743 INFO:teuthology.orchestra.run.smithi190.stdout:(31/32): librbd1-19.0.0-1667.gdb0330b1.el9.x86_ 8.9 MB/s | 3.1 MB 00:00 2024-03-24T12:16:11.485 INFO:teuthology.orchestra.run.smithi190.stdout:(32/32): librados2-19.0.0-1667.gdb0330b1.el9.x8 3.0 MB/s | 3.4 MB 00:01 2024-03-24T12:16:11.490 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:16:11.491 INFO:teuthology.orchestra.run.smithi190.stdout:Total 9.7 MB/s | 58 MB 00:05 2024-03-24T12:16:11.525 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:16:11.561 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:16:11.561 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:16:11.818 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:16:11.937 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-24T12:16:11.982 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-24T12:16:12.016 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-24T12:16:12.048 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:16:12.048 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:16:12.055 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T12:16:12.072 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T12:16:12.277 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T12:16:12.347 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T12:16:12.689 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-24T12:16:12.831 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T12:16:12.919 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T12:16:13.014 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 8/34 2024-03-24T12:16:13.051 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T12:16:13.114 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T12:16:13.152 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-24T12:16:13.213 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 11/34 2024-03-24T12:16:13.293 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T12:16:13.324 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T12:16:13.388 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T12:16:13.423 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-24T12:16:13.460 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:16:13.534 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-24T12:16:13.554 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-24T12:16:13.575 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-24T12:16:13.590 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-24T12:16:13.624 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-24T12:16:13.631 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-24T12:16:13.646 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T12:16:13.662 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-24T12:16:13.667 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-24T12:16:13.706 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-24T12:16:13.740 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-24T12:16:13.881 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T12:16:13.934 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-24T12:16:14.064 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-24T12:16:14.104 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T12:16:14.195 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T12:16:14.235 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 23/34 2024-03-24T12:16:14.256 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-24T12:16:14.291 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-24T12:16:14.325 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T12:16:14.418 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T12:16:14.507 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 8/34 2024-03-24T12:16:14.535 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T12:16:14.598 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 9/34 2024-03-24T12:16:14.626 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-24T12:16:14.671 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 11/34 2024-03-24T12:16:14.739 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T12:16:14.762 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T12:16:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-24T12:16:14.808 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 13/34 2024-03-24T12:16:14.836 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-24T12:16:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T12:16:14.916 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T12:16:14.946 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-24T12:16:14.995 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-24T12:16:15.004 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 27/34 2024-03-24T12:16:15.040 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 28/34 2024-03-24T12:16:15.044 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-24T12:16:15.076 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-24T12:16:15.105 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-24T12:16:15.138 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-24T12:16:15.451 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-24T12:16:15.476 INFO:teuthology.orchestra.run.smithi190.stdout: Upgrading : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T12:16:15.567 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 22/34 2024-03-24T12:16:15.597 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 23/34 2024-03-24T12:16:15.636 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-24T12:16:16.087 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-24T12:16:16.112 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T12:16:16.183 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 26/34 2024-03-24T12:16:16.260 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 27/34 2024-03-24T12:16:16.306 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 28/34 2024-03-24T12:16:17.358 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T12:16:17.431 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T12:16:18.012 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T12:16:18.065 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T12:16:18.145 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T12:16:18.145 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T12:16:18.145 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T12:16:18.145 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:18.192 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T12:16:18.481 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T12:16:18.519 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T12:16:19.089 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 29/34 2024-03-24T12:16:19.138 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T12:16:19.230 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 30/34 2024-03-24T12:16:19.230 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T12:16:19.230 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T12:16:19.230 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:19.273 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T12:16:28.862 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T12:16:28.862 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /sys 2024-03-24T12:16:28.862 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /proc 2024-03-24T12:16:28.862 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /mnt 2024-03-24T12:16:28.863 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /var/tmp 2024-03-24T12:16:28.863 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /home 2024-03-24T12:16:28.863 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /root 2024-03-24T12:16:28.863 INFO:teuthology.orchestra.run.smithi190.stdout:skipping the directory /tmp 2024-03-24T12:16:28.863 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:29.804 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T12:16:29.829 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T12:16:29.830 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:16:29.830 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T12:16:29.830 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T12:16:29.830 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T12:16:29.830 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:29.831 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T12:16:29.907 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T12:16:29.907 INFO:teuthology.orchestra.run.smithi190.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T12:16:30.183 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T12:16:30.183 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2024-03-24T12:16:30.183 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2024-03-24T12:16:30.183 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2024-03-24T12:16:30.183 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2024-03-24T12:16:30.183 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2024-03-24T12:16:30.183 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2024-03-24T12:16:30.183 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2024-03-24T12:16:30.183 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:31.088 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 6/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 8/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 9/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 10/34 2024-03-24T12:16:31.089 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 11/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 13/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-24T12:16:31.090 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-24T12:16:31.091 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-24T12:16:31.091 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-24T12:16:31.091 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-24T12:16:31.091 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-24T12:16:31.091 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-24T12:16:31.091 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-24T12:16:31.092 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-24T12:16:31.092 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T12:16:31.092 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-24T12:16:31.092 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/34 2024-03-24T12:16:31.138 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T12:16:31.162 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/34 2024-03-24T12:16:31.162 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:16:31.163 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T12:16:31.163 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T12:16:31.163 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T12:16:31.163 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:31.164 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T12:16:31.259 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-24T12:16:31.260 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T12:16:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T12:16:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:31.990 INFO:teuthology.orchestra.run.smithi190.stdout:Upgraded: 2024-03-24T12:16:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.990 INFO:teuthology.orchestra.run.smithi190.stdout: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T12:16:31.991 INFO:teuthology.orchestra.run.smithi190.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.992 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T12:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T12:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T12:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T12:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T12:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:31.993 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:16:32.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-test 2024-03-24T12:16:32.777 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_6 6/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 7/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9. 8/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x8 9/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 10/34 2024-03-24T12:16:32.778 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 11/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 12/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 13/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-24T12:16:32.779 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-24T12:16:32.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-24T12:16:32.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-24T12:16:32.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-24T12:16:32.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-24T12:16:32.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-24T12:16:32.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-24T12:16:32.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 31/34 2024-03-24T12:16:32.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-24T12:16:32.780 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/34 2024-03-24T12:16:32.903 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:29 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:16:33.050 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:16:33.051 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:16:33.051 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-24T12:16:33.051 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:16:33.051 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:16:33.051 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 47 M 2024-03-24T12:16:33.051 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-24T12:16:33.051 INFO:teuthology.orchestra.run.smithi190.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T12:16:33.052 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 163 k 2024-03-24T12:16:33.052 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T12:16:33.052 INFO:teuthology.orchestra.run.smithi190.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T12:16:33.052 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T12:16:33.052 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:33.052 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:16:33.052 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:16:33.052 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-03-24T12:16:33.052 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:33.053 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 48 M 2024-03-24T12:16:33.053 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 205 M 2024-03-24T12:16:33.053 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:16:33.479 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-24T12:16:33.530 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T12:16:33.555 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-24T12:16:33.605 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): libcephsqlite-19.0.0-1667.gdb0330b1.el9. 567 kB/s | 163 kB 00:00 2024-03-24T12:16:33.946 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T12:16:33.947 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.948 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:33.949 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:16:33.989 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 432 kB/s | 187 kB 00:00 2024-03-24T12:16:34.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-test 2024-03-24T12:16:34.957 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:32 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:16:35.121 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:16:35.122 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:16:35.122 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-24T12:16:35.122 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:16:35.122 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:16:35.122 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 47 M 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 163 k 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-03-24T12:16:35.123 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:35.124 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 48 M 2024-03-24T12:16:35.124 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 205 M 2024-03-24T12:16:35.124 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:16:35.833 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 572 kB/s | 217 kB 00:00 2024-03-24T12:16:35.867 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): libcephsqlite-19.0.0-1667.gdb0330b1.el9. 395 kB/s | 163 kB 00:00 2024-03-24T12:16:36.043 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-24T12:16:36.068 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 311 kB/s | 63 kB 00:00 2024-03-24T12:16:36.661 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 303 kB/s | 187 kB 00:00 2024-03-24T12:16:39.569 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): ceph-test-19.0.0-1667.gdb0330b1.el9.x86_ 12 MB/s | 47 MB 00:04 2024-03-24T12:16:39.572 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:16:39.572 INFO:teuthology.orchestra.run.smithi029.stdout:Total 11 MB/s | 48 MB 00:04 2024-03-24T12:16:39.587 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:16:39.610 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:16:39.610 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:16:40.003 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:16:40.003 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:16:40.590 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:16:40.680 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T12:16:40.726 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T12:16:40.770 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T12:16:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T12:16:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T12:16:42.513 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): ceph-test-19.0.0-1667.gdb0330b1.el9.x86_ 5.1 MB/s | 47 MB 00:09 2024-03-24T12:16:42.516 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:16:42.516 INFO:teuthology.orchestra.run.smithi190.stdout:Total 5.1 MB/s | 48 MB 00:09 2024-03-24T12:16:42.530 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:16:42.551 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:16:42.551 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:16:42.930 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:16:42.931 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:16:43.498 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:16:43.582 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T12:16:43.617 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T12:16:43.648 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T12:16:43.698 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T12:16:43.733 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T12:16:45.612 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T12:16:45.709 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T12:16:48.193 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/6 2024-03-24T12:16:48.282 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T12:16:49.832 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T12:16:49.832 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/6 2024-03-24T12:16:49.832 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/6 2024-03-24T12:16:49.832 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T12:16:49.832 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T12:16:49.832 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T12:16:50.305 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T12:16:50.305 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:50.306 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:16:50.306 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:50.306 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-16.el9.x86_64 2024-03-24T12:16:50.306 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:50.306 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T12:16:50.306 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T12:16:50.306 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T12:16:50.306 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:50.306 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:16:50.568 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph 2024-03-24T12:16:50.739 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 6/6 2024-03-24T12:16:50.739 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/6 2024-03-24T12:16:50.739 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/6 2024-03-24T12:16:50.740 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T12:16:50.740 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T12:16:50.740 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T12:16:51.135 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:00:49 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:16:51.151 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T12:16:51.151 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:51.151 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:16:51.151 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-test-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:51.151 INFO:teuthology.orchestra.run.smithi190.stdout: jq-1.6-16.el9.x86_64 2024-03-24T12:16:51.151 INFO:teuthology.orchestra.run.smithi190.stdout: libcephsqlite-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:16:51.151 INFO:teuthology.orchestra.run.smithi190.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T12:16:51.151 INFO:teuthology.orchestra.run.smithi190.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T12:16:51.152 INFO:teuthology.orchestra.run.smithi190.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T12:16:51.152 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:51.152 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:16:51.300 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:16:51.302 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2024-03-24T12:16:51.302 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-24T12:16:51.302 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2024-03-24T12:16:51.302 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:16:51.302 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 6.6 k 2024-03-24T12:16:51.302 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 2.1 M 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 1.5 M 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 247 k 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 4.7 M 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 17 M 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T12:16:51.303 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T12:16:51.304 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:16:51.305 INFO:teuthology.orchestra.run.smithi029.stdout:========================================================================================= 2024-03-24T12:16:51.306 INFO:teuthology.orchestra.run.smithi029.stdout:Install 36 Packages 2024-03-24T12:16:51.306 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:16:51.306 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 30 M 2024-03-24T12:16:51.306 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 107 M 2024-03-24T12:16:51.306 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:16:51.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph 2024-03-24T12:16:51.981 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:00:48 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:16:52.132 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:16:52.134 INFO:teuthology.orchestra.run.smithi190.stdout:========================================================================================= 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout:========================================================================================= 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: ceph x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 6.6 k 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 2.1 M 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 1.5 M 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 247 k 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 4.7 M 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 17 M 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T12:16:52.135 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T12:16:52.136 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:16:52.137 INFO:teuthology.orchestra.run.smithi190.stdout:========================================================================================= 2024-03-24T12:16:52.138 INFO:teuthology.orchestra.run.smithi190.stdout:Install 36 Packages 2024-03-24T12:16:52.138 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:16:52.138 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 30 M 2024-03-24T12:16:52.138 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 107 M 2024-03-24T12:16:52.138 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:16:52.157 INFO:teuthology.orchestra.run.smithi029.stdout:(1/36): ceph-19.0.0-1667.gdb0330b1.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-24T12:16:52.908 INFO:teuthology.orchestra.run.smithi190.stdout:(1/36): ceph-19.0.0-1667.gdb0330b1.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-24T12:16:53.100 INFO:teuthology.orchestra.run.smithi190.stdout:(2/36): ceph-mgr-19.0.0-1667.gdb0330b1.el9.x86_ 2.6 MB/s | 1.5 MB 00:00 2024-03-24T12:16:53.141 INFO:teuthology.orchestra.run.smithi029.stdout:(2/36): ceph-mgr-19.0.0-1667.gdb0330b1.el9.x86_ 1.1 MB/s | 1.5 MB 00:01 2024-03-24T12:16:53.225 INFO:teuthology.orchestra.run.smithi029.stdout:(3/36): ceph-mds-19.0.0-1667.gdb0330b1.el9.x86_ 1.5 MB/s | 2.1 MB 00:01 2024-03-24T12:16:53.326 INFO:teuthology.orchestra.run.smithi029.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1667.gdb03 2.4 MB/s | 247 kB 00:00 2024-03-24T12:16:53.502 INFO:teuthology.orchestra.run.smithi029.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-24T12:16:53.569 INFO:teuthology.orchestra.run.smithi029.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.1 MB/s | 144 kB 00:00 2024-03-24T12:16:53.619 INFO:teuthology.orchestra.run.smithi029.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2024-03-24T12:16:53.670 INFO:teuthology.orchestra.run.smithi029.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 940 kB/s | 47 kB 00:00 2024-03-24T12:16:53.729 INFO:teuthology.orchestra.run.smithi029.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 660 kB/s | 39 kB 00:00 2024-03-24T12:16:53.797 INFO:teuthology.orchestra.run.smithi029.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.4 MB/s | 241 kB 00:00 2024-03-24T12:16:53.898 INFO:teuthology.orchestra.run.smithi029.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-24T12:16:53.948 INFO:teuthology.orchestra.run.smithi029.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-24T12:16:53.999 INFO:teuthology.orchestra.run.smithi029.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-24T12:16:54.066 INFO:teuthology.orchestra.run.smithi029.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.7 MB/s | 187 kB 00:00 2024-03-24T12:16:54.559 INFO:teuthology.orchestra.run.smithi190.stdout:(3/36): ceph-osd-19.0.0-1667.gdb0330b1.el9.x86_ 12 MB/s | 17 MB 00:01 2024-03-24T12:16:54.625 INFO:teuthology.orchestra.run.smithi190.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1667.gdb03 3.7 MB/s | 247 kB 00:00 2024-03-24T12:16:54.667 INFO:teuthology.orchestra.run.smithi190.stdout:(5/36): ceph-mds-19.0.0-1667.gdb0330b1.el9.x86_ 1.0 MB/s | 2.1 MB 00:02 2024-03-24T12:16:54.775 INFO:teuthology.orchestra.run.smithi029.stdout:(15/36): ceph-mon-19.0.0-1667.gdb0330b1.el9.x86 1.8 MB/s | 4.7 MB 00:02 2024-03-24T12:16:54.926 INFO:teuthology.orchestra.run.smithi190.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 622 kB/s | 187 kB 00:00 2024-03-24T12:16:54.951 INFO:teuthology.orchestra.run.smithi190.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 558 kB/s | 144 kB 00:00 2024-03-24T12:16:54.977 INFO:teuthology.orchestra.run.smithi190.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 629 kB/s | 32 kB 00:00 2024-03-24T12:16:55.010 INFO:teuthology.orchestra.run.smithi190.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 806 kB/s | 47 kB 00:00 2024-03-24T12:16:55.044 INFO:teuthology.orchestra.run.smithi190.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 578 kB/s | 39 kB 00:00 2024-03-24T12:16:55.109 INFO:teuthology.orchestra.run.smithi029.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 67 kB/s | 22 kB 00:00 2024-03-24T12:16:55.114 INFO:teuthology.orchestra.run.smithi190.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.4 MB/s | 241 kB 00:00 2024-03-24T12:16:55.203 INFO:teuthology.orchestra.run.smithi190.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 7.2 MB/s | 1.1 MB 00:00 2024-03-24T12:16:55.228 INFO:teuthology.orchestra.run.smithi190.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 882 kB/s | 103 kB 00:00 2024-03-24T12:16:55.262 INFO:teuthology.orchestra.run.smithi190.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.1 MB/s | 124 kB 00:00 2024-03-24T12:16:55.287 INFO:teuthology.orchestra.run.smithi190.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-24T12:16:55.429 INFO:teuthology.orchestra.run.smithi190.stdout:(16/36): ceph-mon-19.0.0-1667.gdb0330b1.el9.x86 1.9 MB/s | 4.7 MB 00:02 2024-03-24T12:16:55.446 INFO:teuthology.orchestra.run.smithi190.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-24T12:16:55.668 INFO:teuthology.orchestra.run.smithi029.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 270 kB/s | 151 kB 00:00 2024-03-24T12:16:55.771 INFO:teuthology.orchestra.run.smithi190.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 134 kB/s | 43 kB 00:00 2024-03-24T12:16:55.794 INFO:teuthology.orchestra.run.smithi029.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 347 kB/s | 43 kB 00:00 2024-03-24T12:16:55.897 INFO:teuthology.orchestra.run.smithi190.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 323 kB/s | 151 kB 00:00 2024-03-24T12:16:55.997 INFO:teuthology.orchestra.run.smithi190.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 762 kB/s | 172 kB 00:00 2024-03-24T12:16:56.064 INFO:teuthology.orchestra.run.smithi190.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.1 MB/s | 358 kB 00:00 2024-03-24T12:16:56.081 INFO:teuthology.orchestra.run.smithi190.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-24T12:16:56.111 INFO:teuthology.orchestra.run.smithi029.stdout:(19/36): ceph-osd-19.0.0-1667.gdb0330b1.el9.x86 5.7 MB/s | 17 MB 00:02 2024-03-24T12:16:56.162 INFO:teuthology.orchestra.run.smithi029.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 467 kB/s | 172 kB 00:00 2024-03-24T12:16:56.164 INFO:teuthology.orchestra.run.smithi190.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 176 kB/s | 18 kB 00:00 2024-03-24T12:16:56.181 INFO:teuthology.orchestra.run.smithi190.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-24T12:16:56.257 INFO:teuthology.orchestra.run.smithi190.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 211 kB/s | 19 kB 00:00 2024-03-24T12:16:56.270 INFO:teuthology.orchestra.run.smithi029.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-03-24T12:16:56.273 INFO:teuthology.orchestra.run.smithi190.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-24T12:16:56.357 INFO:teuthology.orchestra.run.smithi190.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 462 kB/s | 46 kB 00:00 2024-03-24T12:16:56.379 INFO:teuthology.orchestra.run.smithi029.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 162 kB/s | 18 kB 00:00 2024-03-24T12:16:56.382 INFO:teuthology.orchestra.run.smithi190.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 727 kB/s | 79 kB 00:00 2024-03-24T12:16:56.497 INFO:teuthology.orchestra.run.smithi029.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 198 kB/s | 23 kB 00:00 2024-03-24T12:16:56.599 INFO:teuthology.orchestra.run.smithi029.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 191 kB/s | 19 kB 00:00 2024-03-24T12:16:56.683 INFO:teuthology.orchestra.run.smithi190.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 837 kB/s | 272 kB 00:00 2024-03-24T12:16:56.699 INFO:teuthology.orchestra.run.smithi029.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-24T12:16:56.708 INFO:teuthology.orchestra.run.smithi190.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 50 kB/s | 16 kB 00:00 2024-03-24T12:16:56.741 INFO:teuthology.orchestra.run.smithi029.stdout:(26/36): python3-cherrypy-18.6.1-2.el9.noarch.r 569 kB/s | 358 kB 00:00 2024-03-24T12:16:56.767 INFO:teuthology.orchestra.run.smithi190.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-03-24T12:16:56.792 INFO:teuthology.orchestra.run.smithi029.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 500 kB/s | 46 kB 00:00 2024-03-24T12:16:56.792 INFO:teuthology.orchestra.run.smithi190.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 428 kB/s | 36 kB 00:00 2024-03-24T12:16:56.851 INFO:teuthology.orchestra.run.smithi029.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 723 kB/s | 79 kB 00:00 2024-03-24T12:16:56.884 INFO:teuthology.orchestra.run.smithi190.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-24T12:16:56.968 INFO:teuthology.orchestra.run.smithi029.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-03-24T12:16:56.992 INFO:teuthology.orchestra.run.smithi190.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 184 kB/s | 20 kB 00:00 2024-03-24T12:16:57.051 INFO:teuthology.orchestra.run.smithi190.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.6 MB/s | 427 kB 00:00 2024-03-24T12:16:57.077 INFO:teuthology.orchestra.run.smithi029.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 827 kB/s | 90 kB 00:00 2024-03-24T12:16:57.194 INFO:teuthology.orchestra.run.smithi029.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 306 kB/s | 36 kB 00:00 2024-03-24T12:16:57.253 INFO:teuthology.orchestra.run.smithi029.stdout:(32/36): python3-pecan-1.4.2-3.el9.noarch.rpm 591 kB/s | 272 kB 00:00 2024-03-24T12:16:57.312 INFO:teuthology.orchestra.run.smithi029.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-24T12:16:57.412 INFO:teuthology.orchestra.run.smithi029.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-24T12:16:57.863 INFO:teuthology.orchestra.run.smithi029.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 701 kB/s | 427 kB 00:00 2024-03-24T12:17:58.509 INFO:teuthology.orchestra.run.smithi029.stdout:(36/36): python3-requests-2.25.1-8.el9.noarch.r 1.9 kB/s | 125 kB 01:04 2024-03-24T12:17:58.512 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:17:58.512 INFO:teuthology.orchestra.run.smithi029.stdout:Total 452 kB/s | 30 MB 01:07 2024-03-24T12:17:58.533 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:17:58.580 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:17:58.580 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:17:59.053 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:17:59.053 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:17:59.767 INFO:teuthology.orchestra.run.smithi190.stdout:(36/36): python3-requests-2.25.1-8.el9.noarch.r 1.9 kB/s | 125 kB 01:04 2024-03-24T12:17:59.771 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:17:59.771 INFO:teuthology.orchestra.run.smithi190.stdout:Total 449 kB/s | 30 MB 01:07 2024-03-24T12:17:59.793 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:17:59.836 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:17:59.836 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:00.246 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:00.246 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:00.422 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:18:00.504 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-24T12:18:00.554 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-24T12:18:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-24T12:18:00.682 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-24T12:18:00.734 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-24T12:18:00.776 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-24T12:18:00.881 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-24T12:18:00.968 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-24T12:18:01.016 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-24T12:18:01.098 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-24T12:18:01.160 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-24T12:18:01.208 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-24T12:18:01.284 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-24T12:18:01.358 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-24T12:18:01.577 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-24T12:18:01.592 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:01.641 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-24T12:18:01.653 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-24T12:18:01.686 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-24T12:18:01.696 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-24T12:18:01.714 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-24T12:18:01.742 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-24T12:18:01.780 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-24T12:18:01.784 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-24T12:18:01.829 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-24T12:18:01.861 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-24T12:18:01.862 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-24T12:18:01.909 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T12:18:01.951 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-24T12:18:01.952 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-24T12:18:01.992 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-24T12:18:02.038 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-24T12:18:02.040 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-24T12:18:02.082 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-24T12:18:02.090 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-24T12:18:02.153 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-24T12:18:02.192 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-24T12:18:02.208 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-24T12:18:02.239 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-24T12:18:02.246 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-24T12:18:02.303 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-24T12:18:02.332 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-24T12:18:02.370 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-24T12:18:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-24T12:18:02.558 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-24T12:18:02.600 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-24T12:18:02.617 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T12:18:02.645 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-24T12:18:02.682 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-24T12:18:02.712 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-24T12:18:02.788 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-24T12:18:02.795 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 31/36 2024-03-24T12:18:02.826 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T12:18:02.831 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T12:18:02.849 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T12:18:02.849 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:02.849 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T12:18:02.849 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T12:18:02.849 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T12:18:02.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:02.864 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-24T12:18:02.905 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-24T12:18:02.941 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-24T12:18:03.158 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-24T12:18:03.246 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-24T12:18:03.289 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-24T12:18:03.361 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-24T12:18:03.497 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-24T12:18:03.618 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T12:18:03.777 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 31/36 2024-03-24T12:18:03.805 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T12:18:03.829 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 32/36 2024-03-24T12:18:03.830 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:03.830 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T12:18:03.830 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T12:18:03.830 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T12:18:03.830 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:04.361 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T12:18:04.386 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T12:18:04.386 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:04.386 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T12:18:04.386 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T12:18:04.386 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T12:18:04.386 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:04.854 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T12:18:04.876 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T12:18:04.876 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:04.876 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T12:18:04.876 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T12:18:04.876 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T12:18:04.876 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:05.099 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T12:18:05.121 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T12:18:05.121 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:05.121 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T12:18:05.121 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T12:18:05.121 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T12:18:05.121 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:05.191 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T12:18:05.290 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T12:18:05.317 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 33/36 2024-03-24T12:18:05.317 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:05.317 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T12:18:05.317 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T12:18:05.317 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T12:18:05.317 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:05.766 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T12:18:05.789 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 34/36 2024-03-24T12:18:05.789 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:05.789 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T12:18:05.789 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T12:18:05.789 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T12:18:05.789 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:06.000 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T12:18:06.022 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 35/36 2024-03-24T12:18:06.023 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:06.023 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T12:18:06.023 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T12:18:06.023 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T12:18:06.023 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:06.090 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T12:18:07.642 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T12:18:07.642 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/36 2024-03-24T12:18:07.642 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/36 2024-03-24T12:18:07.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/36 2024-03-24T12:18:07.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/36 2024-03-24T12:18:07.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/36 2024-03-24T12:18:07.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 6/36 2024-03-24T12:18:07.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-24T12:18:07.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-24T12:18:07.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-24T12:18:07.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-24T12:18:07.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-24T12:18:07.643 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-24T12:18:07.644 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-24T12:18:07.645 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-24T12:18:07.645 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-24T12:18:07.645 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-24T12:18:07.645 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-24T12:18:07.645 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-24T12:18:07.645 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-24T12:18:07.645 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T12:18:07.645 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-24T12:18:07.645 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-24T12:18:07.645 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-24T12:18:07.646 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-24T12:18:07.646 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-24T12:18:07.710 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 36/36 2024-03-24T12:18:07.710 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/36 2024-03-24T12:18:07.710 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/36 2024-03-24T12:18:07.710 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 3/36 2024-03-24T12:18:07.710 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 4/36 2024-03-24T12:18:07.710 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 5/36 2024-03-24T12:18:07.710 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9. 6/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-24T12:18:07.711 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-24T12:18:07.712 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-24T12:18:07.712 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-24T12:18:07.712 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-24T12:18:07.712 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-24T12:18:07.712 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-24T12:18:07.712 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-24T12:18:07.713 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-24T12:18:07.713 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-24T12:18:07.713 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-24T12:18:07.713 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-24T12:18:07.713 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-24T12:18:07.713 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-24T12:18:07.714 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-24T12:18:07.714 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-24T12:18:07.714 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-24T12:18:07.714 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-24T12:18:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-24T12:18:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:08.582 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:18:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:08.582 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T12:18:08.583 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T12:18:08.584 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T12:18:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T12:18:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T12:18:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T12:18:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T12:18:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T12:18:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T12:18:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T12:18:08.585 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:08.585 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-24T12:18:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:08.726 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mds-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-modules-core-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:08.726 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mon-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-osd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T12:18:08.727 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T12:18:08.728 INFO:teuthology.orchestra.run.smithi190.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T12:18:08.729 INFO:teuthology.orchestra.run.smithi190.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T12:18:08.729 INFO:teuthology.orchestra.run.smithi190.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T12:18:08.729 INFO:teuthology.orchestra.run.smithi190.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T12:18:08.729 INFO:teuthology.orchestra.run.smithi190.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T12:18:08.729 INFO:teuthology.orchestra.run.smithi190.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T12:18:08.729 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:08.729 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:09.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-base 2024-03-24T12:18:09.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-base 2024-03-24T12:18:09.635 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:07 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:09.645 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:06 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:09.734 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:18:09.741 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-base-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:18:09.788 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:09.789 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-24T12:18:09.789 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:09.792 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:09.793 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-24T12:18:09.793 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:09.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install cephadm 2024-03-24T12:18:09.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install cephadm 2024-03-24T12:18:10.420 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:07 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:10.434 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:08 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:10.571 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:10.572 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:10.572 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-24T12:18:10.572 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:10.572 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:18:10.573 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 748 k 2024-03-24T12:18:10.573 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:10.573 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:18:10.573 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:10.573 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-24T12:18:10.573 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:10.573 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 748 k 2024-03-24T12:18:10.573 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 754 k 2024-03-24T12:18:10.573 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:18:10.586 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:10.587 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:10.587 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-24T12:18:10.587 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:10.587 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:18:10.587 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 748 k 2024-03-24T12:18:10.587 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:10.587 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:18:10.588 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:10.588 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-24T12:18:10.588 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:10.588 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 748 k 2024-03-24T12:18:10.588 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 754 k 2024-03-24T12:18:10.588 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:18:10.883 INFO:teuthology.orchestra.run.smithi190.stdout:cephadm-19.0.0-1667.gdb0330b1.el9.noarch.rpm 2.4 MB/s | 748 kB 00:00 2024-03-24T12:18:10.884 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:10.884 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.3 MB/s | 748 kB 00:00 2024-03-24T12:18:10.884 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:18:10.885 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm-19.0.0-1667.gdb0330b1.el9.noarch.rpm 2.5 MB/s | 748 kB 00:00 2024-03-24T12:18:10.885 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:10.885 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.5 MB/s | 748 kB 00:00 2024-03-24T12:18:10.885 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:18:10.889 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:18:10.889 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:18:10.891 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:18:10.895 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:10.895 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:10.897 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:18:10.897 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:18:10.971 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:10.975 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:18:11.374 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T12:18:11.431 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T12:18:11.446 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T12:18:11.530 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T12:18:11.833 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T12:18:11.984 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T12:18:12.163 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T12:18:12.163 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:12.163 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:12.163 INFO:teuthology.orchestra.run.smithi190.stdout: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:12.163 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:12.163 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:12.230 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/1 2024-03-24T12:18:12.230 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:12.230 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:18:12.230 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:12.230 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:12.230 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:12.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T12:18:12.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T12:18:12.955 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:09 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:13.037 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:11 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:13.106 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:13.107 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:13.107 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-24T12:18:13.107 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:13.107 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:18:13.107 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 145 k 2024-03-24T12:18:13.107 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:13.107 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:18:13.107 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:13.107 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-24T12:18:13.108 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:13.108 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 145 k 2024-03-24T12:18:13.108 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 436 k 2024-03-24T12:18:13.108 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:18:13.193 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 145 k 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:13.194 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 145 k 2024-03-24T12:18:13.195 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 436 k 2024-03-24T12:18:13.195 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:18:13.409 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-immutable-object-cache-19.0.0-1667.gdb0330 677 kB/s | 145 kB 00:00 2024-03-24T12:18:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:Total 672 kB/s | 145 kB 00:00 2024-03-24T12:18:13.410 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:18:13.418 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:18:13.418 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:18:13.447 INFO:teuthology.orchestra.run.smithi190.stdout:ceph-immutable-object-cache-19.0.0-1667.gdb0330 427 kB/s | 145 kB 00:00 2024-03-24T12:18:13.448 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:13.448 INFO:teuthology.orchestra.run.smithi190.stdout:Total 426 kB/s | 145 kB 00:00 2024-03-24T12:18:13.449 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:18:13.456 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:18:13.456 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:13.472 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:18:13.473 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:18:13.508 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:13.508 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:13.621 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:18:13.644 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:13.678 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T12:18:13.698 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T12:18:13.701 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T12:18:13.702 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:13.702 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T12:18:13.702 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:13.723 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T12:18:13.723 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:13.723 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T12:18:13.724 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T12:18:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:14.730 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:14.730 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:14.730 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:14.758 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1. 1/1 2024-03-24T12:18:14.758 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:14.758 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:18:14.758 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:14.758 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:14.758 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:14.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr 2024-03-24T12:18:15.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr 2024-03-24T12:18:15.571 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:12 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:15.621 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:13 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:15.669 INFO:teuthology.orchestra.run.smithi190.stdout:Package ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:18:15.721 INFO:teuthology.orchestra.run.smithi029.stdout:Package ceph-mgr-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:18:15.721 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:15.722 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-24T12:18:15.723 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:15.776 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:15.777 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-24T12:18:15.777 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:15.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T12:18:15.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T12:18:16.351 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:13 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:16.423 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:14 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:16.503 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:16.504 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-03-24T12:18:16.504 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-24T12:18:16.504 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-03-24T12:18:16.504 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:18:16.504 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 1.8 M 2024-03-24T12:18:16.504 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-24T12:18:16.504 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T12:18:16.504 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 31 k 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 15 k 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout:====================================================================================== 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-03-24T12:18:16.505 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:16.506 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 6.9 M 2024-03-24T12:18:16.506 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-03-24T12:18:16.506 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:18:16.579 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:16.580 INFO:teuthology.orchestra.run.smithi029.stdout:====================================================================================== 2024-03-24T12:18:16.580 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout:====================================================================================== 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 1.8 M 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 31 k 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 15 k 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T12:18:16.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T12:18:16.582 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T12:18:16.582 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:16.582 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:18:16.582 INFO:teuthology.orchestra.run.smithi029.stdout:====================================================================================== 2024-03-24T12:18:16.582 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2024-03-24T12:18:16.582 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:16.582 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 6.9 M 2024-03-24T12:18:16.583 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2024-03-24T12:18:16.583 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:18:16.969 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1667.gdb0 58 kB/s | 15 kB 00:00 2024-03-24T12:18:16.971 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1667.gdb 154 kB/s | 31 kB 00:00 2024-03-24T12:18:16.985 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1667.gdb 112 kB/s | 31 kB 00:00 2024-03-24T12:18:17.072 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1667.gdb0 50 kB/s | 15 kB 00:00 2024-03-24T12:18:17.094 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1667.gdb0330b 4.6 MB/s | 1.8 MB 00:00 2024-03-24T12:18:17.169 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-03-24T12:18:17.205 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-24T12:18:17.264 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.4 MB/s | 1.0 MB 00:00 2024-03-24T12:18:17.311 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.9 MB/s | 1.0 MB 00:00 2024-03-24T12:18:17.323 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1667.gdb0330b 3.2 MB/s | 1.8 MB 00:00 2024-03-24T12:18:17.391 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-03-24T12:18:17.420 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.6 MB/s | 863 kB 00:00 2024-03-24T12:18:17.458 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.8 MB/s | 548 kB 00:00 2024-03-24T12:18:17.500 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 3.4 MB/s | 144 kB 00:00 2024-03-24T12:18:17.533 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 915 kB/s | 31 kB 00:00 2024-03-24T12:18:17.601 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.7 MB/s | 2.0 MB 00:00 2024-03-24T12:18:17.635 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-24T12:18:17.737 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 46 kB/s | 19 kB 00:00 2024-03-24T12:18:17.920 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 730 kB/s | 548 kB 00:00 2024-03-24T12:18:18.029 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-24T12:18:18.054 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 455 kB/s | 144 kB 00:00 2024-03-24T12:18:18.118 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 947 kB/s | 863 kB 00:00 2024-03-24T12:18:18.122 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:18.122 INFO:teuthology.orchestra.run.smithi029.stdout:Total 4.5 MB/s | 6.9 MB 00:01 2024-03-24T12:18:18.142 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:18:18.154 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-24T12:18:18.162 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:18:18.162 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:18:18.340 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:18:18.340 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:18:18.380 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.2 MB/s | 2.0 MB 00:00 2024-03-24T12:18:18.386 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:18.386 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.6 MB/s | 6.9 MB 00:01 2024-03-24T12:18:18.404 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:18:18.422 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:18:18.422 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:18.590 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:18.590 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:18.866 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:18:18.953 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-24T12:18:19.007 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-24T12:18:19.063 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-24T12:18:19.104 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:19.110 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-24T12:18:19.189 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-24T12:18:19.225 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-24T12:18:19.232 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-24T12:18:19.278 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-24T12:18:19.310 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-24T12:18:19.378 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-24T12:18:19.411 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-24T12:18:19.489 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-24T12:18:19.544 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-24T12:18:19.547 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-24T12:18:19.593 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-24T12:18:19.625 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 10/12 2024-03-24T12:18:19.639 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-24T12:18:19.689 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-24T12:18:19.726 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-24T12:18:19.758 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 10/12 2024-03-24T12:18:19.888 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 11/12 2024-03-24T12:18:19.928 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T12:18:20.001 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 11/12 2024-03-24T12:18:20.041 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T12:18:20.911 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T12:18:20.911 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 1/12 2024-03-24T12:18:20.912 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 2/12 2024-03-24T12:18:20.912 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 3/12 2024-03-24T12:18:20.912 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-24T12:18:20.912 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-24T12:18:20.912 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-24T12:18:20.912 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-24T12:18:20.912 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-24T12:18:20.912 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-24T12:18:20.912 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-24T12:18:20.912 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-24T12:18:21.100 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 12/12 2024-03-24T12:18:21.100 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el 1/12 2024-03-24T12:18:21.100 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noa 2/12 2024-03-24T12:18:21.101 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9 3/12 2024-03-24T12:18:21.101 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-24T12:18:21.101 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-24T12:18:21.101 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-24T12:18:21.101 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-24T12:18:21.101 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-24T12:18:21.101 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-24T12:18:21.101 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-24T12:18:21.101 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-24T12:18:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-24T12:18:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:21.270 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T12:18:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:21.270 INFO:teuthology.orchestra.run.smithi190.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T12:18:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T12:18:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T12:18:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T12:18:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T12:18:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T12:18:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T12:18:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T12:18:21.271 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:21.271 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:21.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-24T12:18:21.463 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:21.463 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:18:21.463 INFO:teuthology.orchestra.run.smithi029.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T12:18:21.463 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:21.463 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:21.464 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:21.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T12:18:21.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T12:18:22.088 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:19 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:22.239 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:22.240 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-24T12:18:22.240 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-24T12:18:22.240 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-24T12:18:22.240 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:18:22.240 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 7.4 M 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T12:18:22.241 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T12:18:22.242 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:22.242 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:18:22.242 INFO:teuthology.orchestra.run.smithi190.stdout:============================================================================================ 2024-03-24T12:18:22.242 INFO:teuthology.orchestra.run.smithi190.stdout:Install 12 Packages 2024-03-24T12:18:22.242 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:22.242 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 37 M 2024-03-24T12:18:22.243 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 209 M 2024-03-24T12:18:22.243 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:18:22.324 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:20 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:22.484 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:22.485 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-24T12:18:22.485 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-24T12:18:22.485 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-24T12:18:22.485 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:18:22.485 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 7.4 M 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T12:18:22.486 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T12:18:22.487 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T12:18:22.487 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:22.487 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:18:22.487 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2024-03-24T12:18:22.487 INFO:teuthology.orchestra.run.smithi029.stdout:Install 12 Packages 2024-03-24T12:18:22.487 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:22.488 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 37 M 2024-03-24T12:18:22.488 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 209 M 2024-03-24T12:18:22.488 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:18:22.637 INFO:teuthology.orchestra.run.smithi190.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 124 kB/s | 32 kB 00:00 2024-03-24T12:18:22.695 INFO:teuthology.orchestra.run.smithi190.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 294 kB/s | 17 kB 00:00 2024-03-24T12:18:22.754 INFO:teuthology.orchestra.run.smithi190.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 585 kB/s | 34 kB 00:00 2024-03-24T12:18:22.854 INFO:teuthology.orchestra.run.smithi190.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.3 MB/s | 3.0 MB 00:00 2024-03-24T12:18:23.279 INFO:teuthology.orchestra.run.smithi029.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 97 kB/s | 32 kB 00:00 2024-03-24T12:18:23.321 INFO:teuthology.orchestra.run.smithi029.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 409 kB/s | 17 kB 00:00 2024-03-24T12:18:23.321 INFO:teuthology.orchestra.run.smithi190.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 8.4 MB/s | 4.7 MB 00:00 2024-03-24T12:18:23.363 INFO:teuthology.orchestra.run.smithi029.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 818 kB/s | 34 kB 00:00 2024-03-24T12:18:23.380 INFO:teuthology.orchestra.run.smithi190.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 9.1 MB/s | 4.8 MB 00:00 2024-03-24T12:18:23.405 INFO:teuthology.orchestra.run.smithi190.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-24T12:18:24.106 INFO:teuthology.orchestra.run.smithi190.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-03-24T12:18:24.131 INFO:teuthology.orchestra.run.smithi190.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-24T12:18:24.214 INFO:teuthology.orchestra.run.smithi029.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 5.6 MB/s | 4.7 MB 00:00 2024-03-24T12:18:24.281 INFO:teuthology.orchestra.run.smithi190.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-24T12:18:24.398 INFO:teuthology.orchestra.run.smithi190.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 669 kB/s | 195 kB 00:00 2024-03-24T12:18:24.522 INFO:teuthology.orchestra.run.smithi029.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 1.9 MB/s | 3.0 MB 00:01 2024-03-24T12:18:24.648 INFO:teuthology.orchestra.run.smithi029.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 4.8 MB 00:00 2024-03-24T12:18:24.681 INFO:teuthology.orchestra.run.smithi029.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.4 MB/s | 217 kB 00:00 2024-03-24T12:18:25.182 INFO:teuthology.orchestra.run.smithi029.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-24T12:18:25.258 INFO:teuthology.orchestra.run.smithi029.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.6 MB/s | 195 kB 00:00 2024-03-24T12:18:25.400 INFO:teuthology.orchestra.run.smithi029.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-24T12:18:25.842 INFO:teuthology.orchestra.run.smithi029.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-03-24T12:18:27.948 INFO:teuthology.orchestra.run.smithi190.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.3 MB/s | 7.4 MB 00:05 2024-03-24T12:18:27.950 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:27.951 INFO:teuthology.orchestra.run.smithi190.stdout:Total 6.5 MB/s | 37 MB 00:05 2024-03-24T12:18:27.994 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:18:28.019 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:18:28.019 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:28.342 INFO:teuthology.orchestra.run.smithi029.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.4 MB/s | 7.4 MB 00:05 2024-03-24T12:18:28.344 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:28.345 INFO:teuthology.orchestra.run.smithi029.stdout:Total 6.3 MB/s | 37 MB 00:05 2024-03-24T12:18:28.372 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:28.373 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:28.389 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:18:28.415 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:18:28.415 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:18:28.795 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:18:28.795 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:18:29.070 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:29.146 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T12:18:29.185 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T12:18:29.286 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T12:18:29.330 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T12:18:29.508 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T12:18:29.520 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:18:29.537 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T12:18:29.601 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T12:18:29.619 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T12:18:29.641 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T12:18:29.742 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T12:18:29.787 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T12:18:29.971 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T12:18:30.037 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T12:18:30.136 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T12:18:30.278 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T12:18:30.392 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T12:18:30.861 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T12:18:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T12:18:31.623 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T12:18:32.293 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T12:18:32.653 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T12:18:32.679 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T12:18:33.392 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T12:18:33.420 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T12:18:35.242 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T12:18:35.242 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 1/12 2024-03-24T12:18:35.242 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T12:18:35.242 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T12:18:35.242 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T12:18:35.242 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T12:18:35.242 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T12:18:35.242 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T12:18:35.243 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T12:18:35.243 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T12:18:35.243 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T12:18:35.243 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T12:18:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T12:18:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:35.630 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:35.630 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:35.631 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:36.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-rook 2024-03-24T12:18:36.593 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:33 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:36.755 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:36.756 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-24T12:18:36.756 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-24T12:18:36.756 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-24T12:18:36.756 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:18:36.756 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 49 k 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T12:18:36.757 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T12:18:36.758 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T12:18:36.758 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:36.758 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:18:36.758 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-24T12:18:36.758 INFO:teuthology.orchestra.run.smithi190.stdout:Install 13 Packages 2024-03-24T12:18:36.758 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:36.758 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 2.0 M 2024-03-24T12:18:36.759 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 26 M 2024-03-24T12:18:36.759 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:18:36.939 INFO:teuthology.orchestra.run.smithi190.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 135 kB/s | 18 kB 00:00 2024-03-24T12:18:36.964 INFO:teuthology.orchestra.run.smithi190.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 153 kB/s | 24 kB 00:00 2024-03-24T12:18:37.015 INFO:teuthology.orchestra.run.smithi190.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-24T12:18:37.048 INFO:teuthology.orchestra.run.smithi190.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-24T12:18:37.074 INFO:teuthology.orchestra.run.smithi190.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-24T12:18:37.099 INFO:teuthology.orchestra.run.smithi190.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 859 kB/s | 43 kB 00:00 2024-03-24T12:18:37.141 INFO:teuthology.orchestra.run.smithi190.stdout:(7/13): ceph-mgr-rook-19.0.0-1667.gdb0330b1.el9 147 kB/s | 49 kB 00:00 2024-03-24T12:18:37.191 INFO:teuthology.orchestra.run.smithi190.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 275 kB/s | 32 kB 00:00 2024-03-24T12:18:37.217 INFO:teuthology.orchestra.run.smithi190.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-24T12:18:37.258 INFO:teuthology.orchestra.run.smithi190.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-24T12:18:37.292 INFO:teuthology.orchestra.run.smithi190.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 1.3 MB/s | 203 kB 00:00 2024-03-24T12:18:37.317 INFO:teuthology.orchestra.run.smithi190.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-24T12:18:37.359 INFO:teuthology.orchestra.run.smithi190.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-03-24T12:18:37.361 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:37.362 INFO:teuthology.orchestra.run.smithi190.stdout:Total 3.4 MB/s | 2.0 MB 00:00 2024-03-24T12:18:37.362 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:18:37.374 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:18:37.375 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:37.468 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:37.468 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 12/12 2024-03-24T12:18:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb033 1/12 2024-03-24T12:18:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T12:18:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T12:18:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T12:18:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T12:18:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T12:18:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T12:18:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T12:18:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T12:18:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T12:18:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T12:18:37.985 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:38.157 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T12:18:38.218 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T12:18:38.282 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T12:18:38.309 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T12:18:38.309 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:38.309 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:18:38.309 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:38.310 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:38.320 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T12:18:38.361 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T12:18:38.454 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T12:18:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T12:18:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T12:18:38.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-rook 2024-03-24T12:18:39.334 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T12:18:39.402 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T12:18:39.416 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:37 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:39.436 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T12:18:39.484 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T12:18:39.506 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T12:18:39.572 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:39.573 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-24T12:18:39.573 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-24T12:18:39.573 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-24T12:18:39.573 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:18:39.573 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 49 k 2024-03-24T12:18:39.573 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-24T12:18:39.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T12:18:39.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T12:18:39.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T12:18:39.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T12:18:39.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T12:18:39.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T12:18:39.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T12:18:39.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T12:18:39.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T12:18:39.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T12:18:39.575 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T12:18:39.575 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T12:18:39.575 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:39.575 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:18:39.575 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-24T12:18:39.575 INFO:teuthology.orchestra.run.smithi029.stdout:Install 13 Packages 2024-03-24T12:18:39.575 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:39.576 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 2.0 M 2024-03-24T12:18:39.576 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 26 M 2024-03-24T12:18:39.576 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:18:39.819 INFO:teuthology.orchestra.run.smithi029.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 122 kB/s | 24 kB 00:00 2024-03-24T12:18:39.844 INFO:teuthology.orchestra.run.smithi029.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 80 kB/s | 18 kB 00:00 2024-03-24T12:18:39.987 INFO:teuthology.orchestra.run.smithi029.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.0 MB/s | 174 kB 00:00 2024-03-24T12:18:40.012 INFO:teuthology.orchestra.run.smithi029.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 793 kB/s | 132 kB 00:00 2024-03-24T12:18:40.046 INFO:teuthology.orchestra.run.smithi029.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-24T12:18:40.072 INFO:teuthology.orchestra.run.smithi029.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 725 kB/s | 43 kB 00:00 2024-03-24T12:18:40.222 INFO:teuthology.orchestra.run.smithi029.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 100 kB/s | 15 kB 00:00 2024-03-24T12:18:40.248 INFO:teuthology.orchestra.run.smithi029.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 160 kB/s | 32 kB 00:00 2024-03-24T12:18:40.275 INFO:teuthology.orchestra.run.smithi029.stdout:(9/13): ceph-mgr-rook-19.0.0-1667.gdb0330b1.el9 75 kB/s | 49 kB 00:00 2024-03-24T12:18:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T12:18:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 1/13 2024-03-24T12:18:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T12:18:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T12:18:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T12:18:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T12:18:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T12:18:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T12:18:40.337 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T12:18:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T12:18:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T12:18:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T12:18:40.338 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T12:18:40.342 INFO:teuthology.orchestra.run.smithi029.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.7 MB/s | 203 kB 00:00 2024-03-24T12:18:40.368 INFO:teuthology.orchestra.run.smithi029.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 640 kB/s | 59 kB 00:00 2024-03-24T12:18:40.393 INFO:teuthology.orchestra.run.smithi029.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-24T12:18:40.427 INFO:teuthology.orchestra.run.smithi029.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.7 MB/s | 1.0 MB 00:00 2024-03-24T12:18:40.429 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:40.429 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-24T12:18:40.430 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:18:40.445 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:18:40.445 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:18:40.544 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:18:40.544 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:18:40.709 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T12:18:40.709 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:40.709 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:40.709 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:40.709 INFO:teuthology.orchestra.run.smithi190.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T12:18:40.710 INFO:teuthology.orchestra.run.smithi190.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T12:18:40.711 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:40.711 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:40.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T12:18:41.061 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:18:41.233 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T12:18:41.290 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T12:18:41.363 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T12:18:41.409 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T12:18:41.452 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T12:18:41.512 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:38 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:41.551 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T12:18:41.663 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:41.663 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-24T12:18:41.663 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-24T12:18:41.663 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 138 k 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout:======================================================================================== 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout:Install 6 Packages 2024-03-24T12:18:41.664 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:41.665 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 6.8 M 2024-03-24T12:18:41.665 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 33 M 2024-03-24T12:18:41.665 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:18:41.677 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T12:18:41.747 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T12:18:41.937 INFO:teuthology.orchestra.run.smithi190.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-24T12:18:42.229 INFO:teuthology.orchestra.run.smithi190.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 5.8 MB 00:00 2024-03-24T12:18:42.271 INFO:teuthology.orchestra.run.smithi190.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1667.gdb0330b1.e 266 kB/s | 138 kB 00:00 2024-03-24T12:18:42.438 INFO:teuthology.orchestra.run.smithi190.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 277 kB/s | 58 kB 00:00 2024-03-24T12:18:42.463 INFO:teuthology.orchestra.run.smithi190.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 266 kB/s | 51 kB 00:00 2024-03-24T12:18:42.505 INFO:teuthology.orchestra.run.smithi190.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 968 kB/s | 548 kB 00:00 2024-03-24T12:18:42.507 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:42.508 INFO:teuthology.orchestra.run.smithi190.stdout:Total 8.1 MB/s | 6.8 MB 00:00 2024-03-24T12:18:42.520 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:18:42.529 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T12:18:42.531 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:18:42.531 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:42.599 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T12:18:42.621 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:42.621 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:42.633 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T12:18:42.676 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T12:18:42.705 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T12:18:42.917 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:43.096 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T12:18:43.159 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T12:18:43.697 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T12:18:43.778 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T12:18:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 13/13 2024-03-24T12:18:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 1/13 2024-03-24T12:18:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T12:18:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T12:18:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T12:18:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T12:18:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T12:18:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T12:18:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T12:18:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T12:18:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T12:18:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T12:18:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T12:18:43.840 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T12:18:43.863 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T12:18:44.156 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T12:18:44.156 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:44.156 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:18:44.156 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:44.156 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T12:18:44.156 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T12:18:44.156 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T12:18:44.156 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T12:18:44.156 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T12:18:44.156 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T12:18:44.157 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T12:18:44.157 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T12:18:44.157 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T12:18:44.157 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T12:18:44.158 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T12:18:44.158 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T12:18:44.158 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:44.158 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:44.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T12:18:44.734 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T12:18:44.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/6 2024-03-24T12:18:44.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T12:18:44.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T12:18:44.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T12:18:44.734 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T12:18:45.047 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:43 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:45.048 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T12:18:45.048 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:45.048 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:45.048 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:45.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T12:18:45.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T12:18:45.048 INFO:teuthology.orchestra.run.smithi190.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T12:18:45.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T12:18:45.049 INFO:teuthology.orchestra.run.smithi190.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T12:18:45.049 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:45.049 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:45.210 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:45.256 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-24T12:18:45.256 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 138 k 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout:======================================================================================== 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout:Install 6 Packages 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:45.257 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 6.8 M 2024-03-24T12:18:45.258 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 33 M 2024-03-24T12:18:45.258 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:18:45.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-fuse 2024-03-24T12:18:45.860 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:42 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:45.928 INFO:teuthology.orchestra.run.smithi029.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-24T12:18:46.010 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:46.010 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:46.010 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-24T12:18:46.011 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:46.011 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:18:46.011 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 886 k 2024-03-24T12:18:46.011 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-24T12:18:46.011 INFO:teuthology.orchestra.run.smithi190.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T12:18:46.011 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:46.011 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:18:46.011 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:46.011 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-03-24T12:18:46.011 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:46.012 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 964 k 2024-03-24T12:18:46.012 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 2.8 M 2024-03-24T12:18:46.012 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:18:46.036 INFO:teuthology.orchestra.run.smithi029.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1667.gdb0330b1.e 297 kB/s | 138 kB 00:00 2024-03-24T12:18:46.143 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 602 kB/s | 79 kB 00:00 2024-03-24T12:18:46.229 INFO:teuthology.orchestra.run.smithi029.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 377 kB/s | 248 kB 00:00 2024-03-24T12:18:46.379 INFO:teuthology.orchestra.run.smithi029.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 168 kB/s | 58 kB 00:00 2024-03-24T12:18:46.443 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): ceph-fuse-19.0.0-1667.gdb0330b1.el9.x86_ 2.0 MB/s | 886 kB 00:00 2024-03-24T12:18:46.445 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:46.445 INFO:teuthology.orchestra.run.smithi190.stdout:Total 2.2 MB/s | 964 kB 00:00 2024-03-24T12:18:46.445 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:18:46.466 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:18:46.467 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:46.541 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:46.541 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:46.571 INFO:teuthology.orchestra.run.smithi029.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 149 kB/s | 51 kB 00:00 2024-03-24T12:18:46.630 INFO:teuthology.orchestra.run.smithi029.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 781 kB/s | 548 kB 00:00 2024-03-24T12:18:46.632 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:46.632 INFO:teuthology.orchestra.run.smithi029.stdout:Total 4.8 MB/s | 6.8 MB 00:01 2024-03-24T12:18:46.646 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:18:46.657 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:18:46.657 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:18:46.733 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:46.751 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:18:46.751 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:18:46.869 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T12:18:46.942 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T12:18:47.047 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:18:47.239 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T12:18:47.302 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T12:18:47.667 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T12:18:47.667 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/2 2024-03-24T12:18:47.881 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T12:18:47.977 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T12:18:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T12:18:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:48.024 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T12:18:48.024 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:48.024 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:48.056 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T12:18:48.094 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T12:18:48.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install ceph-volume 2024-03-24T12:18:48.861 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:45 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:49.012 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:49.012 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repository Size 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 264 k 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout:Installing dependencies: 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout:Install 2 Packages 2024-03-24T12:18:49.013 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:49.014 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 332 k 2024-03-24T12:18:49.014 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 1.5 M 2024-03-24T12:18:49.014 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:18:49.139 INFO:teuthology.orchestra.run.smithi190.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 549 kB/s | 69 kB 00:00 2024-03-24T12:18:49.299 INFO:teuthology.orchestra.run.smithi190.stdout:(2/2): ceph-volume-19.0.0-1667.gdb0330b1.el9.no 929 kB/s | 264 kB 00:00 2024-03-24T12:18:49.299 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:49.300 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-24T12:18:49.300 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:18:49.308 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:18:49.309 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:49.320 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 6/6 2024-03-24T12:18:49.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 1/6 2024-03-24T12:18:49.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T12:18:49.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T12:18:49.320 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T12:18:49.321 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T12:18:49.346 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:49.347 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:49.628 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:49.696 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-24T12:18:49.743 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T12:18:49.758 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T12:18:49.759 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:49.759 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T12:18:49.759 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:49.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-fuse 2024-03-24T12:18:50.507 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:48 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:50.522 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 1/2 2024-03-24T12:18:50.662 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:50.663 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:50.663 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 886 k 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 964 k 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 2.8 M 2024-03-24T12:18:50.664 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:18:50.804 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 562 kB/s | 79 kB 00:00 2024-03-24T12:18:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-24T12:18:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:50.873 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T12:18:50.873 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:50.874 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:51.130 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-fuse-19.0.0-1667.gdb0330b1.el9.x86_ 1.9 MB/s | 886 kB 00:00 2024-03-24T12:18:51.132 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:51.132 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.0 MB/s | 964 kB 00:00 2024-03-24T12:18:51.132 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:18:51.153 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:18:51.153 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:18:51.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados-devel 2024-03-24T12:18:51.229 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:18:51.229 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:18:51.412 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:18:51.561 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T12:18:51.634 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T12:18:51.757 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:48 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:51.907 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:51.908 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:51.908 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-24T12:18:51.908 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:51.908 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:18:51.908 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 127 k 2024-03-24T12:18:51.908 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:51.908 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:18:51.909 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:51.909 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-24T12:18:51.909 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:51.909 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 127 k 2024-03-24T12:18:51.909 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 456 k 2024-03-24T12:18:51.909 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:18:52.221 INFO:teuthology.orchestra.run.smithi190.stdout:librados-devel-19.0.0-1667.gdb0330b1.el9.x86_64 405 kB/s | 127 kB 00:00 2024-03-24T12:18:52.222 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:52.222 INFO:teuthology.orchestra.run.smithi190.stdout:Total 403 kB/s | 127 kB 00:00 2024-03-24T12:18:52.222 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:18:52.228 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:18:52.228 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:52.282 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:52.283 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2/2 2024-03-24T12:18:52.393 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/2 2024-03-24T12:18:52.480 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:52.586 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:18:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T12:18:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:52.709 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:18:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T12:18:52.709 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:52.709 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:53.015 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:18:53.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-volume 2024-03-24T12:18:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:18:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:53.322 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:53.322 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:53.322 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:53.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs2 2024-03-24T12:18:53.657 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:51 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:53.817 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:53.817 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:19.0.0-1667.gdb0330b1.el9 ceph-noarch 264 k 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout:Install 2 Packages 2024-03-24T12:18:53.818 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:53.819 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 332 k 2024-03-24T12:18:53.819 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 1.5 M 2024-03-24T12:18:53.819 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:18:53.951 INFO:teuthology.orchestra.run.smithi029.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 520 kB/s | 69 kB 00:00 2024-03-24T12:18:54.145 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:51 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:54.247 INFO:teuthology.orchestra.run.smithi190.stdout:Package libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:18:54.299 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:54.300 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-24T12:18:54.300 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:54.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install libcephfs-devel 2024-03-24T12:18:54.394 INFO:teuthology.orchestra.run.smithi029.stdout:(2/2): ceph-volume-19.0.0-1667.gdb0330b1.el9.no 459 kB/s | 264 kB 00:00 2024-03-24T12:18:54.395 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:54.395 INFO:teuthology.orchestra.run.smithi029.stdout:Total 576 kB/s | 332 kB 00:00 2024-03-24T12:18:54.395 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:18:54.405 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:18:54.405 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:18:54.450 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:18:54.450 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:18:54.606 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:18:54.793 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-24T12:18:54.842 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T12:18:54.857 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2/2 2024-03-24T12:18:54.857 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:18:54.857 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T12:18:54.857 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:54.918 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:51 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:55.067 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:55.067 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:55.067 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-24T12:18:55.067 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:55.067 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:18:55.067 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 31 k 2024-03-24T12:18:55.067 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:55.067 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:18:55.068 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:18:55.068 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-24T12:18:55.068 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:55.068 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 31 k 2024-03-24T12:18:55.068 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 140 k 2024-03-24T12:18:55.068 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:18:55.233 INFO:teuthology.orchestra.run.smithi190.stdout:libcephfs-devel-19.0.0-1667.gdb0330b1.el9.x86_6 189 kB/s | 31 kB 00:00 2024-03-24T12:18:55.234 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:55.234 INFO:teuthology.orchestra.run.smithi190.stdout:Total 187 kB/s | 31 kB 00:00 2024-03-24T12:18:55.234 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:18:55.238 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:18:55.238 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:18:55.266 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:18:55.266 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:18:55.354 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:18:55.457 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:18:55.756 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 1/2 2024-03-24T12:18:55.869 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:18:56.120 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-24T12:18:56.120 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:56.120 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:18:56.120 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:19.0.0-1667.gdb0330b1.el9.noarch 2024-03-24T12:18:56.120 INFO:teuthology.orchestra.run.smithi029.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T12:18:56.120 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:56.120 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:56.138 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:18:56.138 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:56.138 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:18:56.138 INFO:teuthology.orchestra.run.smithi190.stdout: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:56.139 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:18:56.139 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:56.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librados2 2024-03-24T12:18:56.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados-devel 2024-03-24T12:18:56.870 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:53 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:56.978 INFO:teuthology.orchestra.run.smithi190.stdout:Package librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:18:57.029 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:57.030 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-24T12:18:57.030 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:57.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install librbd1 2024-03-24T12:18:57.093 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:55 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:57.251 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:57.252 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:57.252 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-24T12:18:57.252 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:57.252 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:18:57.252 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 127 k 2024-03-24T12:18:57.252 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:57.252 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:18:57.252 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:18:57.253 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-24T12:18:57.253 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:57.253 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 127 k 2024-03-24T12:18:57.253 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 456 k 2024-03-24T12:18:57.253 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:18:57.647 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:54 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:57.678 INFO:teuthology.orchestra.run.smithi029.stdout:librados-devel-19.0.0-1667.gdb0330b1.el9.x86_64 298 kB/s | 127 kB 00:00 2024-03-24T12:18:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:18:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:Total 296 kB/s | 127 kB 00:00 2024-03-24T12:18:57.679 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:18:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:18:57.685 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:18:57.743 INFO:teuthology.orchestra.run.smithi190.stdout:Package librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:18:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:18:57.744 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:18:57.793 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:57.794 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-24T12:18:57.795 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:57.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rados 2024-03-24T12:18:57.953 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:18:58.064 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:18:58.416 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:55 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:58.512 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:18:58.536 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:18:58.563 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:58.564 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-24T12:18:58.564 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:58.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rgw 2024-03-24T12:18:58.849 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:18:58.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:58.849 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:18:58.849 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:18:58.849 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:18:58.849 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:59.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs2 2024-03-24T12:18:59.183 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:56 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:18:59.279 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:18:59.331 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:18:59.332 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-24T12:18:59.332 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:18:59.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-cephfs 2024-03-24T12:18:59.700 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:57 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:18:59.802 INFO:teuthology.orchestra.run.smithi029.stdout:Package libcephfs2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:18:59.857 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:18:59.858 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-24T12:18:59.858 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:18:59.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install libcephfs-devel 2024-03-24T12:18:59.943 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:56 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:19:00.040 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:19:00.091 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:19:00.092 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-24T12:19:00.092 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:19:00.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install python3-rbd 2024-03-24T12:19:00.503 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:02:58 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:19:00.658 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:19:00.659 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:00.659 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-24T12:19:00.659 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:00.659 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:19:00.659 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 31 k 2024-03-24T12:19:00.659 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:00.659 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:19:00.659 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:00.660 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-24T12:19:00.660 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:00.660 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 31 k 2024-03-24T12:19:00.660 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 140 k 2024-03-24T12:19:00.660 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:19:00.704 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:57 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:19:00.800 INFO:teuthology.orchestra.run.smithi190.stdout:Package python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:19:00.813 INFO:teuthology.orchestra.run.smithi029.stdout:libcephfs-devel-19.0.0-1667.gdb0330b1.el9.x86_6 203 kB/s | 31 kB 00:00 2024-03-24T12:19:00.814 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:19:00.814 INFO:teuthology.orchestra.run.smithi029.stdout:Total 201 kB/s | 31 kB 00:00 2024-03-24T12:19:00.814 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:19:00.818 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:19:00.818 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:19:00.850 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:19:00.850 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:19:00.852 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:19:00.853 INFO:teuthology.orchestra.run.smithi190.stdout:Nothing to do. 2024-03-24T12:19:00.853 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:19:00.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-fuse 2024-03-24T12:19:00.951 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:19:01.076 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:01.461 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:02:58 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:19:01.491 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:01.613 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:19:01.614 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:19:01.614 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-24T12:19:01.614 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:19:01.614 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:19:01.614 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 85 k 2024-03-24T12:19:01.614 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:01.615 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:19:01.615 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:19:01.615 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-24T12:19:01.615 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:01.615 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 85 k 2024-03-24T12:19:01.615 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 231 k 2024-03-24T12:19:01.615 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:19:01.770 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:01.770 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:01.770 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:19:01.770 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:19:01.770 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:01.770 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:19:01.854 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-fuse-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 356 kB/s | 85 kB 00:00 2024-03-24T12:19:01.855 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:19:01.855 INFO:teuthology.orchestra.run.smithi190.stdout:Total 354 kB/s | 85 kB 00:00 2024-03-24T12:19:01.855 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:19:01.862 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:19:01.862 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:19:01.917 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:19:01.917 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:19:01.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librados2 2024-03-24T12:19:02.058 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:19:02.159 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:02.557 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:02.560 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:00 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:19:02.659 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:19:02.714 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:19:02.715 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-24T12:19:02.715 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:19:02.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install librbd1 2024-03-24T12:19:02.871 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:02.871 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:02.872 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:19:02.872 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:19:02.872 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:02.872 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:19:03.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-mirror 2024-03-24T12:19:03.363 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:01 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:19:03.465 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:19:03.520 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:19:03.521 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-24T12:19:03.521 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:19:03.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rados 2024-03-24T12:19:03.705 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:03:00 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:19:03.858 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:19:03.858 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:19:03.858 INFO:teuthology.orchestra.run.smithi190.stdout: Package Arch Version Repo Size 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 3.1 M 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 13 M 2024-03-24T12:19:03.859 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:19:04.177 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:02 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:19:04.276 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rados-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:19:04.331 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:19:04.332 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-24T12:19:04.332 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:19:04.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rgw 2024-03-24T12:19:04.980 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:02 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:19:05.079 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rgw-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:19:05.134 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:19:05.135 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-24T12:19:05.135 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:19:05.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-cephfs 2024-03-24T12:19:05.779 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:03 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:19:05.879 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-cephfs-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:19:05.897 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-mirror-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 1.5 MB/s | 3.1 MB 00:02 2024-03-24T12:19:05.898 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:19:05.899 INFO:teuthology.orchestra.run.smithi190.stdout:Total 1.5 MB/s | 3.1 MB 00:02 2024-03-24T12:19:05.899 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:19:05.910 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:19:05.910 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:19:05.933 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:19:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-24T12:19:05.934 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:19:05.985 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:19:05.986 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:19:06.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install python3-rbd 2024-03-24T12:19:06.425 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:19:06.490 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:06.514 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:06.515 INFO:teuthology.orchestra.run.smithi190.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:19:06.515 INFO:teuthology.orchestra.run.smithi190.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T12:19:06.515 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T12:19:06.515 INFO:teuthology.orchestra.run.smithi190.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T12:19:06.515 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:06.582 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:04 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:19:06.680 INFO:teuthology.orchestra.run.smithi029.stdout:Package python3-rbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 is already installed. 2024-03-24T12:19:06.734 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:19:06.735 INFO:teuthology.orchestra.run.smithi029.stdout:Nothing to do. 2024-03-24T12:19:06.735 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:19:06.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-fuse 2024-03-24T12:19:07.380 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:05 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:19:07.534 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:19:07.535 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:07.535 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-24T12:19:07.535 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:07.535 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:19:07.535 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 85 k 2024-03-24T12:19:07.535 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:07.535 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:19:07.536 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:07.536 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-24T12:19:07.536 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:07.536 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 85 k 2024-03-24T12:19:07.536 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 231 k 2024-03-24T12:19:07.536 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:19:07.689 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:07.689 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:07.689 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:19:07.690 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:19:07.690 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:07.690 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:19:07.747 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-fuse-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 403 kB/s | 85 kB 00:00 2024-03-24T12:19:07.748 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:19:07.748 INFO:teuthology.orchestra.run.smithi029.stdout:Total 401 kB/s | 85 kB 00:00 2024-03-24T12:19:07.748 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:19:07.756 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:19:07.756 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:19:07.814 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:19:07.814 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:19:07.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo yum -y install rbd-nbd 2024-03-24T12:19:07.956 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:19:08.076 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:08.526 INFO:teuthology.orchestra.run.smithi190.stdout:Last metadata expiration check: 0:03:05 ago on Sun 24 Mar 2024 12:16:03 PM UTC. 2024-03-24T12:19:08.568 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:08.682 INFO:teuthology.orchestra.run.smithi190.stdout:Dependencies resolved. 2024-03-24T12:19:08.683 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:19:08.683 INFO:teuthology.orchestra.run.smithi190.stdout: Package Architecture Version Repository Size 2024-03-24T12:19:08.683 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:19:08.683 INFO:teuthology.orchestra.run.smithi190.stdout:Installing: 2024-03-24T12:19:08.683 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 171 k 2024-03-24T12:19:08.683 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:08.683 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction Summary 2024-03-24T12:19:08.683 INFO:teuthology.orchestra.run.smithi190.stdout:================================================================================ 2024-03-24T12:19:08.684 INFO:teuthology.orchestra.run.smithi190.stdout:Install 1 Package 2024-03-24T12:19:08.684 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:08.684 INFO:teuthology.orchestra.run.smithi190.stdout:Total download size: 171 k 2024-03-24T12:19:08.684 INFO:teuthology.orchestra.run.smithi190.stdout:Installed size: 494 k 2024-03-24T12:19:08.684 INFO:teuthology.orchestra.run.smithi190.stdout:Downloading Packages: 2024-03-24T12:19:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:08.866 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:19:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:19:08.866 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:08.866 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:19:08.927 INFO:teuthology.orchestra.run.smithi190.stdout:rbd-nbd-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 705 kB/s | 171 kB 00:00 2024-03-24T12:19:08.928 INFO:teuthology.orchestra.run.smithi190.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:19:08.928 INFO:teuthology.orchestra.run.smithi190.stdout:Total 699 kB/s | 171 kB 00:00 2024-03-24T12:19:08.928 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction check 2024-03-24T12:19:08.937 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction check succeeded. 2024-03-24T12:19:08.937 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction test 2024-03-24T12:19:08.994 INFO:teuthology.orchestra.run.smithi190.stdout:Transaction test succeeded. 2024-03-24T12:19:08.994 INFO:teuthology.orchestra.run.smithi190.stdout:Running transaction 2024-03-24T12:19:09.154 INFO:teuthology.orchestra.run.smithi190.stdout: Preparing : 1/1 2024-03-24T12:19:09.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-mirror 2024-03-24T12:19:09.261 INFO:teuthology.orchestra.run.smithi190.stdout: Installing : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:09.705 INFO:teuthology.orchestra.run.smithi190.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:09.744 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:07 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:19:09.897 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:19:09.898 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repo Size 2024-03-24T12:19:09.898 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:09.898 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:19:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 3.1 M 2024-03-24T12:19:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:09.898 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:19:09.898 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:09.898 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-24T12:19:09.898 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:09.899 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 3.1 M 2024-03-24T12:19:09.899 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 13 M 2024-03-24T12:19:09.899 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:19:10.036 INFO:teuthology.orchestra.run.smithi190.stdout: Verifying : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:10.036 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:10.036 INFO:teuthology.orchestra.run.smithi190.stdout:Installed: 2024-03-24T12:19:10.036 INFO:teuthology.orchestra.run.smithi190.stdout: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:19:10.036 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:19:10.036 INFO:teuthology.orchestra.run.smithi190.stdout:Complete! 2024-03-24T12:19:10.281 DEBUG:teuthology.parallel:result is None 2024-03-24T12:19:11.417 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-mirror-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 2.0 MB/s | 3.1 MB 00:01 2024-03-24T12:19:11.418 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:19:11.418 INFO:teuthology.orchestra.run.smithi029.stdout:Total 2.0 MB/s | 3.1 MB 00:01 2024-03-24T12:19:11.419 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:19:11.430 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:19:11.430 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:19:11.510 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:19:11.510 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:19:11.951 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:19:12.027 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:12.052 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:12.052 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T12:19:12.052 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T12:19:12.052 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T12:19:12.052 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T12:19:12.052 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:13.339 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:13.339 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:13.340 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:19:13.340 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:19:13.340 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:13.340 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:19:13.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install rbd-nbd 2024-03-24T12:19:14.219 INFO:teuthology.orchestra.run.smithi029.stdout:Last metadata expiration check: 0:03:12 ago on Sun 24 Mar 2024 12:16:02 PM UTC. 2024-03-24T12:19:14.373 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2024-03-24T12:19:14.374 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:14.374 INFO:teuthology.orchestra.run.smithi029.stdout: Package Architecture Version Repository Size 2024-03-24T12:19:14.374 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:14.374 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2024-03-24T12:19:14.374 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:19.0.0-1667.gdb0330b1.el9 ceph 171 k 2024-03-24T12:19:14.374 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:14.374 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2024-03-24T12:19:14.375 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2024-03-24T12:19:14.375 INFO:teuthology.orchestra.run.smithi029.stdout:Install 1 Package 2024-03-24T12:19:14.375 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:14.375 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 171 k 2024-03-24T12:19:14.375 INFO:teuthology.orchestra.run.smithi029.stdout:Installed size: 494 k 2024-03-24T12:19:14.375 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2024-03-24T12:19:14.587 INFO:teuthology.orchestra.run.smithi029.stdout:rbd-nbd-19.0.0-1667.gdb0330b1.el9.x86_64.rpm 807 kB/s | 171 kB 00:00 2024-03-24T12:19:14.588 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:19:14.588 INFO:teuthology.orchestra.run.smithi029.stdout:Total 800 kB/s | 171 kB 00:00 2024-03-24T12:19:14.588 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2024-03-24T12:19:14.597 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2024-03-24T12:19:14.597 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2024-03-24T12:19:14.654 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2024-03-24T12:19:14.654 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2024-03-24T12:19:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2024-03-24T12:19:14.923 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:15.426 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:15.733 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 1/1 2024-03-24T12:19:15.733 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:15.733 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2024-03-24T12:19:15.733 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:19.0.0-1667.gdb0330b1.el9.x86_64 2024-03-24T12:19:15.733 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:19:15.733 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2024-03-24T12:19:16.008 DEBUG:teuthology.parallel:result is None 2024-03-24T12:19:16.009 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T12:19:16.148 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T12:19:16.168 INFO:teuthology.orchestra.run.smithi029.stdout:19.0.0-1667.gdb0330b1.el9 2024-03-24T12:19:16.169 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1667.gdb0330b1.el9 2024-03-24T12:19:16.169 INFO:teuthology.task.install:The correct ceph version 19.0.0-1667.gdb0330b1 is installed. 2024-03-24T12:19:16.170 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T12:19:16.386 DEBUG:teuthology.orchestra.run.smithi190:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T12:19:16.411 INFO:teuthology.orchestra.run.smithi190.stdout:19.0.0-1667.gdb0330b1.el9 2024-03-24T12:19:16.411 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1667.gdb0330b1.el9 2024-03-24T12:19:16.411 INFO:teuthology.task.install:The correct ceph version 19.0.0-1667.gdb0330b1 is installed. 2024-03-24T12:19:16.413 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T12:19:16.413 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:19:16.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T12:19:16.440 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:19:16.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T12:19:16.483 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T12:19:16.483 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:19:16.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T12:19:16.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T12:19:16.576 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:19:16.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T12:19:16.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T12:19:16.676 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T12:19:16.676 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:19:16.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T12:19:16.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T12:19:16.769 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:19:16.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T12:19:16.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T12:19:16.867 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T12:19:16.867 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:19:16.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T12:19:16.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T12:19:16.959 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:19:16.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T12:19:16.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T12:19:17.055 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T12:19:17.147 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'cephadm_mode': 'root'} 2024-03-24T12:19:17.147 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T12:19:17.148 INFO:tasks.cephadm:Cluster fsid is bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:19:17.148 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T12:19:17.148 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.29', 'mon.c': '[v2:172.21.15.29:3301,v1:172.21.15.29:6790]', 'mon.b': '172.21.15.190'} 2024-03-24T12:19:17.148 INFO:tasks.cephadm:First mon is mon.a on smithi029 2024-03-24T12:19:17.148 INFO:tasks.cephadm:First mgr is y 2024-03-24T12:19:17.149 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T12:19:17.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo hostname $(hostname -s) 2024-03-24T12:19:17.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo hostname $(hostname -s) 2024-03-24T12:19:17.203 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T12:19:17.203 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T12:19:17.415 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77704/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1667-gdb0330b1', 'node_name': '172.21.2.14+braggi14', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1667.gdb0330b1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-16 11:55:36.797747', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-24T12:19:17.628 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref squid, sha1 db0330b1e4e2470d52b750e251e55a522b4f7d69 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T12:19:17.629 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm 2024-03-24T12:19:17.629 DEBUG:teuthology.orchestra.run.smithi029:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T12:19:18.156 INFO:teuthology.orchestra.run.smithi029.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 12:19 /home/ubuntu/cephtest/cephadm 2024-03-24T12:19:18.157 DEBUG:teuthology.orchestra.run.smithi190:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/squid/db0330b1e4e2470d52b750e251e55a522b4f7d69/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T12:19:18.814 INFO:teuthology.orchestra.run.smithi190.stdout:-rw-r--r--. 1 ubuntu ubuntu 766879 Mar 24 12:19 /home/ubuntu/cephtest/cephadm 2024-03-24T12:19:18.814 DEBUG:teuthology.orchestra.run.smithi029:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T12:19:18.832 DEBUG:teuthology.orchestra.run.smithi190:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T12:19:18.863 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 on all hosts... 2024-03-24T12:19:18.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T12:19:18.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 pull 2024-03-24T12:19:19.794 INFO:teuthology.orchestra.run.smithi029.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T12:19:19.846 INFO:teuthology.orchestra.run.smithi190.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T12:19:46.515 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:19:46.515 INFO:teuthology.orchestra.run.smithi029.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T12:19:46.515 INFO:teuthology.orchestra.run.smithi029.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T12:19:46.515 INFO:teuthology.orchestra.run.smithi029.stdout: "repo_digests": [ 2024-03-24T12:19:46.515 INFO:teuthology.orchestra.run.smithi029.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T12:19:46.515 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:19:46.516 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:19:52.359 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:19:52.359 INFO:teuthology.orchestra.run.smithi190.stdout: "ceph_version": "ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev)", 2024-03-24T12:19:52.359 INFO:teuthology.orchestra.run.smithi190.stdout: "image_id": "4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d", 2024-03-24T12:19:52.359 INFO:teuthology.orchestra.run.smithi190.stdout: "repo_digests": [ 2024-03-24T12:19:52.359 INFO:teuthology.orchestra.run.smithi190.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10" 2024-03-24T12:19:52.359 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:19:52.359 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:19:52.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph 2024-03-24T12:19:52.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo mkdir -p /etc/ceph 2024-03-24T12:19:52.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /etc/ceph 2024-03-24T12:19:52.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /etc/ceph 2024-03-24T12:19:52.517 INFO:tasks.cephadm:Writing seed config... 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [global] ms bind msgr1 = False 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [global] ms bind msgr2 = True 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T12:19:52.518 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T12:19:52.519 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-24T12:19:52.520 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:19:52.520 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T12:19:52.546 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = bc017092-e9d8-11ee-95cd-87774f69a715 ms bind msgr1 = False ms bind msgr2 = True ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-24T12:19:52.547 DEBUG:teuthology.orchestra.run.smithi029:mon.a> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T12:19:52.589 DEBUG:teuthology.orchestra.run.smithi029:mgr.y> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T12:19:52.631 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T12:19:52.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 -v bootstrap --fsid bc017092-e9d8-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.29 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:19:52.829 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2024-03-24T12:19:52.829 INFO:teuthology.orchestra.run.smithi029.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69', '-v', 'bootstrap', '--fsid', 'bc017092-e9d8-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.29', '--skip-admin-label'] 2024-03-24T12:19:52.854 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T12:19:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T12:19:52.855 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying podman|docker is present... 2024-03-24T12:19:52.878 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T12:19:52.878 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying lvm2 is present... 2024-03-24T12:19:52.878 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying time synchronization is in place... 2024-03-24T12:19:52.885 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T12:19:52.886 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T12:19:52.891 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T12:19:52.891 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-24T12:19:52.898 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-03-24T12:19:52.903 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-03-24T12:19:52.904 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-03-24T12:19:52.904 INFO:teuthology.orchestra.run.smithi029.stdout:Repeating the final host check... 2024-03-24T12:19:52.926 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T12:19:52.926 INFO:teuthology.orchestra.run.smithi029.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T12:19:52.926 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl is present 2024-03-24T12:19:52.927 INFO:teuthology.orchestra.run.smithi029.stdout:lvcreate is present 2024-03-24T12:19:52.932 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T12:19:52.933 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T12:19:52.938 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T12:19:52.938 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-24T12:19:52.944 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout enabled 2024-03-24T12:19:52.950 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout active 2024-03-24T12:19:52.951 INFO:teuthology.orchestra.run.smithi029.stdout:Unit chronyd.service is enabled and running 2024-03-24T12:19:52.951 INFO:teuthology.orchestra.run.smithi029.stdout:Host looks OK 2024-03-24T12:19:52.951 INFO:teuthology.orchestra.run.smithi029.stdout:Cluster fsid: bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:19:52.951 INFO:teuthology.orchestra.run.smithi029.stdout:Acquiring lock 139649602344560 on /run/cephadm/bc017092-e9d8-11ee-95cd-87774f69a715.lock 2024-03-24T12:19:52.951 INFO:teuthology.orchestra.run.smithi029.stdout:Lock 139649602344560 acquired on /run/cephadm/bc017092-e9d8-11ee-95cd-87774f69a715.lock 2024-03-24T12:19:52.951 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 3300 ... 2024-03-24T12:19:52.952 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying IP 172.21.15.29 port 6789 ... 2024-03-24T12:19:52.952 INFO:teuthology.orchestra.run.smithi029.stdout:Base mon IP(s) is [172.21.15.29:3300, 172.21.15.29:6789], mon addrv is [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-03-24T12:19:52.954 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.29 metric 100 2024-03-24T12:19:52.954 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.29 metric 100 2024-03-24T12:19:52.957 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T12:19:52.957 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-24T12:19:52.957 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-24T12:19:52.959 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T12:19:52.959 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T12:19:52.960 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T12:19:52.960 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-24T12:19:52.960 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2024-03-24T12:19:52.960 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T12:19:52.960 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-03-24T12:19:52.960 INFO:teuthology.orchestra.run.smithi029.stdout:Mon IP `172.21.15.29` is in CIDR network `172.21.0.0/20` 2024-03-24T12:19:52.961 INFO:teuthology.orchestra.run.smithi029.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T12:19:52.961 INFO:teuthology.orchestra.run.smithi029.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T12:19:52.961 INFO:teuthology.orchestra.run.smithi029.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T12:19:58.112 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stdout 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T12:19:58.112 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69... 2024-03-24T12:19:58.112 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T12:19:58.113 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-24T12:19:58.113 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying blob sha256:989de1c398e87dbcb4906791a352373dc0e0415d619bc5b36f1f1823df929cb1 2024-03-24T12:19:58.113 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Copying config sha256:4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d 2024-03-24T12:19:58.113 INFO:teuthology.orchestra.run.smithi029.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T12:20:08.025 INFO:teuthology.orchestra.run.smithi029.stdout:ceph: stdout ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T12:20:08.025 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph version: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev) 2024-03-24T12:20:08.025 INFO:teuthology.orchestra.run.smithi029.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T12:20:09.078 INFO:teuthology.orchestra.run.smithi029.stdout:stat: stdout 167 167 2024-03-24T12:20:09.078 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial keys... 2024-03-24T12:20:10.133 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQB5GgBmEiF2GRAA2AdoA47Bb5tK1jPj73i7DA== 2024-03-24T12:20:11.121 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQB6GgBmQrA2IBAAykyG2y4aHopv+1ElFDmBmw== 2024-03-24T12:20:15.109 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph-authtool: stdout AQB9GgBmLeypCBAAXasrv5yoy44rnOORTerLwg== 2024-03-24T12:20:15.110 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial monmap... 2024-03-24T12:20:16.106 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T12:20:16.106 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T12:20:16.106 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:20:16.106 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T12:20:16.107 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool for a [v2:172.21.15.29:3300,v1:172.21.15.29:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T12:20:16.107 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = quincy 2024-03-24T12:20:16.107 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: set fsid to bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:20:16.107 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T12:20:16.107 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:20:16.107 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mon... 2024-03-24T12:20:17.278 INFO:teuthology.orchestra.run.smithi029.stdout:create mon.a on 2024-03-24T12:20:17.486 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T12:20:17.633 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T12:20:17.826 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-bc017092-e9d8-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-bc017092-e9d8-11ee-95cd-87774f69a715.target. 2024-03-24T12:20:17.826 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-bc017092-e9d8-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-bc017092-e9d8-11ee-95cd-87774f69a715.target. 2024-03-24T12:20:18.050 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.a 2024-03-24T12:20:18.051 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T12:20:18.211 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-bc017092-e9d8-11ee-95cd-87774f69a715.target.wants/ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-bc017092-e9d8-11ee-95cd-87774f69a715@.service. 2024-03-24T12:20:18.749 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T12:20:18.750 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-24T12:20:18.756 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T12:20:18.756 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-24T12:20:18.756 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-24T12:20:18.756 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T12:20:18.757 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon to start... 2024-03-24T12:20:18.757 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mon... 2024-03-24T12:20:20.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:19 smithi029 ceph-mon[29430]: from='client.? 172.21.15.29:0/3393891511' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T12:20:20.974 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T12:20:20.974 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout id: bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:20:20.974 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T12:20:20.974 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:20.974 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout services: 2024-03-24T12:20:20.974 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.589186s) 2024-03-24T12:20:20.974 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T12:20:20.974 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T12:20:20.974 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:20.975 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout data: 2024-03-24T12:20:20.975 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T12:20:20.975 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T12:20:20.975 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T12:20:20.975 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T12:20:20.975 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:20.975 INFO:teuthology.orchestra.run.smithi029.stdout:mon is available 2024-03-24T12:20:20.975 INFO:teuthology.orchestra.run.smithi029.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T12:20:21.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:21 smithi029 ceph-mon[29430]: from='client.? 172.21.15.29:0/1646310796' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T12:20:21.806 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:21 smithi029 ceph-mon[29430]: from='client.? 172.21.15.29:0/1646310796' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T12:20:22.103 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:22.103 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T12:20:22.103 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:20:22.103 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.29:3300,v1:172.21.15.29:6789] 2024-03-24T12:20:22.103 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T12:20:22.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T12:20:22.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T12:20:22.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T12:20:22.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:22.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T12:20:22.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T12:20:22.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:22.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T12:20:22.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T12:20:22.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T12:20:22.105 INFO:teuthology.orchestra.run.smithi029.stdout:Generating new minimal ceph.conf... 2024-03-24T12:20:22.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:22 smithi029 ceph-mon[29430]: from='client.? 172.21.15.29:0/1925534063' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:20:23.175 INFO:teuthology.orchestra.run.smithi029.stdout:Restarting the monitor... 2024-03-24T12:20:23.560 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:23 smithi029 systemd[1]: Stopping Ceph mon.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:20:23.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:23 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a[29407]: 2024-03-24T12:20:23.438+0000 7f0b94099700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:20:23.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:23 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a[29407]: 2024-03-24T12:20:23.438+0000 7f0b94099700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T12:20:23.561 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:23 smithi029 podman[29765]: 2024-03-24 12:20:23.561603385 +0000 UTC m=+0.224335781 container died c658f8f30dba25f6ff4514da56ce4e06e8aa8c870c6a832b2aa10036ca739981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-24T12:20:24.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:23 smithi029 podman[29765]: 2024-03-24 12:20:23.912298124 +0000 UTC m=+0.575030521 container cleanup c658f8f30dba25f6ff4514da56ce4e06e8aa8c870c6a832b2aa10036ca739981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-24T12:20:24.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:23 smithi029 bash[29765]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a 2024-03-24T12:20:24.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 podman[29784]: 2024-03-24 12:20:24.157181344 +0000 UTC m=+0.600233653 container remove c658f8f30dba25f6ff4514da56ce4e06e8aa8c870c6a832b2aa10036ca739981 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T12:20:24.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T12:20:24.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 systemd[1]: Stopped Ceph mon.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:20:24.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 systemd[1]: Starting Ceph mon.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:20:24.775 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 podman[29881]: 2024-03-24 12:20:24.404848758 +0000 UTC m=+0.017682415 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T12:20:24.775 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 podman[29881]: 2024-03-24 12:20:24.517392006 +0000 UTC m=+0.130225655 container create b6e668dccf1e914c2f306eb4b072c2251b20a99e15ab23df7c74e9eed397f327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T12:20:24.775 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 podman[29881]: 2024-03-24 12:20:24.746598809 +0000 UTC m=+0.359432471 container init b6e668dccf1e914c2f306eb4b072c2251b20a99e15ab23df7c74e9eed397f327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T12:20:24.775 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 podman[29881]: 2024-03-24 12:20:24.752294364 +0000 UTC m=+0.365128014 container start b6e668dccf1e914c2f306eb4b072c2251b20a99e15ab23df7c74e9eed397f327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T12:20:24.775 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T12:20:24.775 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T12:20:24.775 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: pidfile_write: ignore empty --pid-file 2024-03-24T12:20:24.867 INFO:teuthology.orchestra.run.smithi029.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T12:20:25.081 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: load: jerasure load: lrc 2024-03-24T12:20:25.081 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: RocksDB version: 7.9.2 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Git sha 0 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: DB SUMMARY 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: DB Session ID: PJ9ROYKCRCTHFB6UEFFO 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: CURRENT file: CURRENT 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 91703 ; 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.error_if_exists: 0 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.create_if_missing: 0 2024-03-24T12:20:25.082 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.paranoid_checks: 1 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.env: 0x5576bf56eae0 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.info_log: 0x5576c13f58c0 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.statistics: (nil) 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.use_fsync: 0 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_log_file_size: 0 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T12:20:25.083 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.allow_fallocate: 1 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.use_direct_reads: 0 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.db_log_dir: 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.wal_dir: 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T12:20:25.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.write_buffer_manager: 0x5576c13c9400 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T12:20:25.085 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.unordered_write: 0 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.row_cache: None 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.wal_filter: None 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.two_write_queues: 0 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T12:20:25.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.wal_compression: 0 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.atomic_flush: 0 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.log_readahead_size: 0 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_background_jobs: 2 2024-03-24T12:20:25.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_background_compactions: -1 2024-03-24T12:20:25.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_subcompactions: 1 2024-03-24T12:20:25.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T12:20:25.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T12:20:25.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T12:20:25.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T12:20:25.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T12:20:25.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T12:20:25.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T12:20:25.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T12:20:25.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_open_files: -1 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_background_flushes: -1 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Compression algorithms supported: 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: kZSTD supported: 0 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: kXpressCompression supported: 0 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: kZlibCompression supported: 1 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: kSnappyCompression supported: 1 2024-03-24T12:20:25.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: kLZ4Compression supported: 1 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: kBZip2Compression supported: 0 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.merge_operator: 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_filter: None 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T12:20:25.091 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5576c13f59c0) 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_type: 0 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_index_type: 0 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_shortening: 1 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: checksum: 4 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: no_block_cache: 0 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache: 0x5576c13f9090 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_options: 2024-03-24T12:20:25.092 INFO:journalctl@ceph.mon.a.smithi029.stdout: capacity : 536870912 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_shard_bits : 4 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: strict_capacity_limit : 0 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_cache_compressed: (nil) 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: persistent_cache: (nil) 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size: 4096 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_size_deviation: 10 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_restart_interval: 16 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: index_block_restart_interval: 1 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: metadata_block_size: 4096 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: partition_filters: 0 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: use_delta_encoding: 1 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: filter_policy: bloomfilter 2024-03-24T12:20:25.093 INFO:journalctl@ceph.mon.a.smithi029.stdout: whole_key_filtering: 1 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout: verify_compression: 0 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout: format_version: 5 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_index_compression: 1 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout: block_align: 0 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout: max_auto_readahead_size: 262144 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout: prepopulate_block_cache: 0 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout: initial_auto_readahead_size: 8192 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T12:20:25.094 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compression: NoCompression 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.num_levels: 7 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T12:20:25.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T12:20:25.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T12:20:25.097 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T12:20:25.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.table_properties_collectors: 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.inplace_update_support: 0 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T12:20:25.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.bloom_locality: 0 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.max_successive_merges: 0 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.ttl: 2592000 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.enable_blob_files: false 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.min_blob_size: 0 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T12:20:25.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ea3d9b4c-9501-49a8-a985-5d903e348eab 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711282824782408, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711282824783350, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 87576, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 282, "table_properties": {"data_size": 85664, "index_size": 258, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11802, "raw_average_key_size": 48, "raw_value_size": 79199, "raw_average_value_size": 328, "num_data_blocks": 11, "num_entries": 241, "num_filter_entries": 241, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711282824, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ea3d9b4c-9501-49a8-a985-5d903e348eab", "db_session_id": "PJ9ROYKCRCTHFB6UEFFO", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711282824783437, "job": 1, "event": "recovery_finished"} 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T12:20:25.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5576c14ea000 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: DB pointer 0x5576c14d4000 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** DB Stats ** 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-24T12:20:25.102 INFO:journalctl@ceph.mon.a.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: L0 2/0 87.36 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: Sum 2/0 87.36 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 136.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 136.0 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-24T12:20:25.103 INFO:journalctl@ceph.mon.a.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: Cumulative compaction: 0.00 GB write, 27.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: Interval compaction: 0.00 GB write, 27.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache BinnedLRUCache@0x5576c13f9090#7 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-24T12:20:25.104 INFO:journalctl@ceph.mon.a.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: starting mon.a rank 0 at public addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] at bind addrs [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: mon.a@-1(???) e1 preinit fsid bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: mon.a@-1(???).mds e1 new map 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: mon.a@-1(???).mds e1 print_map 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout: e1 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout: legacy client fscid: -1 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout: 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout: No filesystems configured 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T12:20:25.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T12:20:25.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T12:20:25.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: monmap e1: 1 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T12:20:25.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: fsmap 2024-03-24T12:20:25.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T12:20:25.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 ceph-mon[29919]: mgrmap e1: no daemons active 2024-03-24T12:20:25.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 bash[29881]: b6e668dccf1e914c2f306eb4b072c2251b20a99e15ab23df7c74e9eed397f327 2024-03-24T12:20:25.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:24 smithi029 systemd[1]: Started Ceph mon.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:20:25.930 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T12:20:25.955 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:20:25.955 INFO:teuthology.orchestra.run.smithi029.stdout:Creating mgr... 2024-03-24T12:20:25.955 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T12:20:25.955 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T12:20:26.199 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mgr.y 2024-03-24T12:20:26.200 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Failed to reset failed state of unit ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mgr.y.service: Unit ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mgr.y.service not loaded. 2024-03-24T12:20:26.350 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-bc017092-e9d8-11ee-95cd-87774f69a715.target.wants/ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-bc017092-e9d8-11ee-95cd-87774f69a715@.service. 2024-03-24T12:20:26.581 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:26 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2905320564' entity='client.admin' 2024-03-24T12:20:26.869 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:26 smithi029 podman[30162]: 2024-03-24 12:20:26.801633704 +0000 UTC m=+0.328127723 container start aa0b22dc45d5467f6fee9d8fef69cd9f37752f2f6088002ec615ded241947914 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T12:20:26.883 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T12:20:26.883 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-24T12:20:26.890 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T12:20:26.890 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-24T12:20:26.890 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-24T12:20:26.890 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T12:20:26.899 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T12:20:26.899 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-24T12:20:26.905 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T12:20:26.905 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-24T12:20:26.905 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-24T12:20:26.905 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T12:20:26.905 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr to start... 2024-03-24T12:20:26.905 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr... 2024-03-24T12:20:27.169 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:26 smithi029 bash[30162]: aa0b22dc45d5467f6fee9d8fef69cd9f37752f2f6088002ec615ded241947914 2024-03-24T12:20:27.170 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:26 smithi029 systemd[1]: Started Ceph mgr.y for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:20:27.420 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:27.364+0000 7f0c0823f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T12:20:27.798 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:27 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/578554915' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T12:20:27.799 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:27.461+0000 7f0c0823f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T12:20:28.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:28.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-24T12:20:28.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:20:28.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T12:20:28.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T12:20:28.090 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T12:20:28.091 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:28.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T12:20:28.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:28.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T12:20:28.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T12:20:28.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T12:20:28.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T12:20:28.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T12:20:28.092 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T12:20:28.095 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T12:20:28.096 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:28.097 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T12:20:28.098 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:28.098 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T12:20:18.713466+0000", 2024-03-24T12:20:28.098 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T12:20:28.098 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:28.098 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T12:20:28.098 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-24T12:20:28.098 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (1/15)... 2024-03-24T12:20:28.367 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:28 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:28.131+0000 7f0c0823f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T12:20:28.367 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:28 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:28.225+0000 7f0c0823f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T12:20:28.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:28 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:28.368+0000 7f0c0823f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T12:20:28.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:28 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:28.464+0000 7f0c0823f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T12:20:28.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:28 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:28.548+0000 7f0c0823f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T12:20:29.107 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:28 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:28.815+0000 7f0c0823f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T12:20:29.107 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:28 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:28.925+0000 7f0c0823f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T12:20:29.107 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:29 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:29.012+0000 7f0c0823f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T12:20:29.107 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:29 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:29.107+0000 7f0c0823f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T12:20:29.420 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:29 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:29.285+0000 7f0c0823f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T12:20:30.524 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:30 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:30.279+0000 7f0c0823f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T12:20:30.524 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:30 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:30.419+0000 7f0c0823f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T12:20:30.853 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:30 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4282586470' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T12:20:30.853 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:30 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:30.763+0000 7f0c0823f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T12:20:31.170 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:30 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:30.854+0000 7f0c0823f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T12:20:31.170 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:31.052+0000 7f0c0823f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:20:31.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:31.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-24T12:20:31.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:20:31.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T12:20:31.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T12:20:31.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T12:20:31.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T12:20:31.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:31.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T12:20:31.241 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T12:20:31.242 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T12:20:31.243 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T12:20:31.244 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T12:20:18.713466+0000", 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-24T12:20:31.245 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (2/15)... 2024-03-24T12:20:31.529 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:31.241+0000 7f0c0823f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T12:20:31.790 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:31.530+0000 7f0c0823f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:20:31.791 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:31.791+0000 7f0c0823f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T12:20:32.170 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:32 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:32.034+0000 7f0c0823f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T12:20:33.170 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:32 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:32.854+0000 7f0c0823f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T12:20:33.653 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:33.446+0000 7f0c0823f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T12:20:33.653 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:33.552+0000 7f0c0823f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T12:20:33.920 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:33.789+0000 7f0c0823f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T12:20:33.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: Activating manager daemon y 2024-03-24T12:20:33.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: mgrmap e2: y(active, starting, since 0.00317374s) 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='mgr.14100 172.21.15.29:0/493112147' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='mgr.14100 172.21.15.29:0/493112147' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='mgr.14100 172.21.15.29:0/493112147' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='mgr.14100 172.21.15.29:0/493112147' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='mgr.14100 172.21.15.29:0/493112147' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: Manager daemon y is now available 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='mgr.14100 172.21.15.29:0/493112147' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='mgr.14100 172.21.15.29:0/493112147' entity='mgr.y' 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='mgr.14100 172.21.15.29:0/493112147' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='mgr.14100 172.21.15.29:0/493112147' entity='mgr.y' 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2447340618' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T12:20:33.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:33 smithi029 ceph-mon[29919]: from='mgr.14100 172.21.15.29:0/493112147' entity='mgr.y' 2024-03-24T12:20:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-24T12:20:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:20:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T12:20:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T12:20:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T12:20:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T12:20:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:34.442 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:34.443 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T12:20:34.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:34.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T12:20:34.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T12:20:34.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T12:20:34.444 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T12:20:34.448 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:34.449 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T12:20:34.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:34.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T12:20:34.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:34.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T12:20:18.713466+0000", 2024-03-24T12:20:34.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T12:20:34.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:34.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T12:20:34.450 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-24T12:20:34.450 INFO:teuthology.orchestra.run.smithi029.stdout:mgr not available, waiting (3/15)... 2024-03-24T12:20:36.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:35 smithi029 ceph-mon[29919]: mgrmap e3: y(active, since 1.00765s) 2024-03-24T12:20:37.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:37.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-24T12:20:37.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:20:37.816 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 0 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T12:20:37.817 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T12:20:37.818 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:37.822 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ], 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T12:20:37.823 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:37.824 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T12:20:37.824 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T12:20:37.824 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T12:20:18.713466+0000", 2024-03-24T12:20:37.824 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T12:20:37.824 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout }, 2024-03-24T12:20:37.824 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T12:20:37.824 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-24T12:20:37.824 INFO:teuthology.orchestra.run.smithi029.stdout:mgr is available 2024-03-24T12:20:38.064 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:37 smithi029 ceph-mon[29919]: mgrmap e4: y(active, since 3s) 2024-03-24T12:20:38.112 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:37 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4121256783' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T12:20:39.100 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:38 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/582353802' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout fsid = bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T12:20:39.104 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T12:20:39.105 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T12:20:39.105 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling cephadm module... 2024-03-24T12:20:40.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:39 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1214568410' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T12:20:40.171 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:39 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ignoring --setuser ceph since I am not root 2024-03-24T12:20:40.171 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:39 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ignoring --setgroup ceph since I am not root 2024-03-24T12:20:40.546 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:40 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:40.450+0000 7efdc463a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T12:20:40.800 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:40 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:40.547+0000 7efdc463a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T12:20:41.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:40 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1214568410' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T12:20:41.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:40 smithi029 ceph-mon[29919]: mgrmap e5: y(active, since 6s) 2024-03-24T12:20:41.341 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:41.239+0000 7efdc463a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T12:20:41.341 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:41.341+0000 7efdc463a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T12:20:41.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:41.488+0000 7efdc463a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T12:20:41.671 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:41.590+0000 7efdc463a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T12:20:41.865 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-24T12:20:41.866 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T12:20:41.866 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T12:20:41.866 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T12:20:41.866 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T12:20:41.866 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-24T12:20:41.866 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-03-24T12:20:41.866 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 5... 2024-03-24T12:20:41.954 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:41 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1116738642' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T12:20:41.955 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:41.677+0000 7efdc463a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T12:20:41.955 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:41.954+0000 7efdc463a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T12:20:42.252 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:42 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:42.071+0000 7efdc463a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T12:20:42.252 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:42 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:42.165+0000 7efdc463a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T12:20:42.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:42 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:42.264+0000 7efdc463a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T12:20:42.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:42 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:42.446+0000 7efdc463a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T12:20:43.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:43.445+0000 7efdc463a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T12:20:43.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:43.579+0000 7efdc463a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T12:20:43.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:43.918+0000 7efdc463a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T12:20:44.170 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:44.011+0000 7efdc463a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T12:20:44.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:44.210+0000 7efdc463a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:20:44.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:44.400+0000 7efdc463a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T12:20:44.955 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:44.693+0000 7efdc463a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:20:44.955 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:44.956+0000 7efdc463a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T12:20:45.420 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:45 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:45.198+0000 7efdc463a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T12:20:46.420 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:46.024+0000 7efdc463a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T12:20:46.920 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:46.612+0000 7efdc463a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T12:20:46.920 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:46.713+0000 7efdc463a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T12:20:47.420 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:20:46.934+0000 7efdc463a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T12:20:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: Active manager daemon y restarted 2024-03-24T12:20:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: Activating manager daemon y 2024-03-24T12:20:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: mgrmap e6: y(active, starting, since 0.0029939s) 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: Manager daemon y is now available 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:20:47.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:47 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:20:48.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:48 smithi029 ceph-mon[29919]: Found migration_current of "None". Setting to last migration. 2024-03-24T12:20:48.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:48 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:20:48.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:48 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T12:20:48.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:48 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T12:20:48.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:48 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:20:48.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:48 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:20:48.300 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:48 smithi029 ceph-mon[29919]: mgrmap e7: y(active, since 1.00539s) 2024-03-24T12:20:48.622 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-24T12:20:48.624 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T12:20:48.624 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T12:20:48.624 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-24T12:20:48.624 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 5 is available 2024-03-24T12:20:48.624 INFO:teuthology.orchestra.run.smithi029.stdout:Setting orchestrator backend to cephadm... 2024-03-24T12:20:49.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:49 smithi029 ceph-mon[29919]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T12:20:49.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:49 smithi029 ceph-mon[29919]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T12:20:49.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:49 smithi029 ceph-mon[29919]: [24/Mar/2024:12:20:48] ENGINE Bus STARTING 2024-03-24T12:20:49.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:49 smithi029 ceph-mon[29919]: [24/Mar/2024:12:20:48] ENGINE Serving on https://172.21.15.29:7150 2024-03-24T12:20:49.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:49 smithi029 ceph-mon[29919]: [24/Mar/2024:12:20:48] ENGINE Serving on http://172.21.15.29:8765 2024-03-24T12:20:49.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:49 smithi029 ceph-mon[29919]: [24/Mar/2024:12:20:48] ENGINE Bus STARTED 2024-03-24T12:20:49.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:49 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:20:50.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:50 smithi029 ceph-mon[29919]: mgrmap e8: y(active, since 2s) 2024-03-24T12:20:50.215 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:50 smithi029 ceph-mon[29919]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:20:50.215 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:50 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:20:50.215 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:50 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:20:51.148 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:51 smithi029 ceph-mon[29919]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:20:51.151 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T12:20:51.151 INFO:teuthology.orchestra.run.smithi029.stdout:Generating ssh key... 2024-03-24T12:20:52.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-mon[29919]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:20:52.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-mon[29919]: Generating ssh key... 2024-03-24T12:20:52.420 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: Generating public/private rsa key pair. 2024-03-24T12:20:52.420 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: Your identification has been saved in /tmp/tmp8qxjzzwe/key. 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: Your public key has been saved in /tmp/tmp8qxjzzwe/key.pub. 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: The key fingerprint is: 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: SHA256:or3JfOwwQY7U8G4TjKzWYrgd0pRQeDMRiqKcIZx6Ud8 ceph-bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: The key's randomart image is: 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: +---[RSA 3072]----+ 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: |.o+oo | 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: |+o=+ B . | 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: |==+o+ B E | 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: |*+o= = . | 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: |=oO o B S | 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: | B o + + | 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: |. . . +. | 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: | o =o | 2024-03-24T12:20:52.421 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: | =o. | 2024-03-24T12:20:52.422 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:20:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: +----[SHA256]-----+ 2024-03-24T12:20:53.493 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:53 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:20:53.493 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:53 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:20:54.122 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEnUzcXK12/3jrEazwrLTsCaOh/KPG5n6lufyT+vVN+WB/NtEpldiYeuwhAdywY9e0lJMBrk8vexSY/K03ACDswLc+/aO1zmSWLVWlBzmFupMSZYfVJ3j36s+PF9eLBWVStojqTBLCTK/lyk1+himFgmUOY4FFJGr4BJcDK3NlIejxjyTSWZHAIyztIr4j2P+z3Pfob1geOYE4xlmqe+E6XTFoaCB4LBweiFOcP4TCNDiSszDfjX/xCcnm0tNu+sU18h8wnKt5paJmn96dfHDsWFT244N8MzynjgxC0nAe9wXqDkCcBhdlNfJLIjCo8ffjQyD40gdvv43CJYIvZfQiEAdePCcj+OJh9tNyAc7kb8FeJLDxzu8gfyr+xlxgOblFQef0yJqpTwC7VDupvZHwT0JFhpXERnNOtFqzeCxjiCPAVJnB9qs+YsKTQdN7aOGGQg7/fnK/qMRAkhBzSVgzRGJNYtbwvY6WU2ygPJrIVXtYVP3HUl0n5IkF93Znxs8= ceph-bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:20:54.122 INFO:teuthology.orchestra.run.smithi029.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T12:20:54.122 INFO:teuthology.orchestra.run.smithi029.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T12:20:54.123 INFO:teuthology.orchestra.run.smithi029.stdout:Adding host smithi029... 2024-03-24T12:20:54.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:54 smithi029 ceph-mon[29919]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:20:55.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:55 smithi029 ceph-mon[29919]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi029", "addr": "172.21.15.29", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:20:56.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:20:56 smithi029 ceph-mon[29919]: Deploying cephadm binary to smithi029 2024-03-24T12:20:59.820 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Added host 'smithi029' with addr '172.21.15.29' 2024-03-24T12:20:59.820 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mon service... 2024-03-24T12:21:00.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:00 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:21:00.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:00 smithi029 ceph-mon[29919]: Added host smithi029 2024-03-24T12:21:00.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:00 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:21:01.329 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T12:21:01.330 INFO:teuthology.orchestra.run.smithi029.stdout:Deploying unmanaged mgr service... 2024-03-24T12:21:01.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:01 smithi029 ceph-mon[29919]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:21:01.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:01 smithi029 ceph-mon[29919]: Saving service mon spec with placement count:5 2024-03-24T12:21:01.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:01 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:21:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:03 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:21:03.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:03 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:21:03.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:03 smithi029 ceph-mon[29919]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:21:03.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:03 smithi029 ceph-mon[29919]: Saving service mgr spec with placement count:2 2024-03-24T12:21:03.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:03 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:21:03.459 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T12:21:05.423 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:05 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3635368308' entity='client.admin' 2024-03-24T12:21:06.345 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling the dashboard module... 2024-03-24T12:21:06.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:06 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1779840613' entity='client.admin' 2024-03-24T12:21:06.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:06 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:21:08.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:08 smithi029 ceph-mon[29919]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:08.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:08 smithi029 ceph-mon[29919]: from='mgr.14120 172.21.15.29:0/3872160467' entity='mgr.y' 2024-03-24T12:21:08.264 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:08 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/39191920' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T12:21:08.264 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:08 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ignoring --setuser ceph since I am not root 2024-03-24T12:21:08.264 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:08 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ignoring --setgroup ceph since I am not root 2024-03-24T12:21:08.867 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:08 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:08.650+0000 7f9d705b7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T12:21:08.867 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:08 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:08.746+0000 7f9d705b7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T12:21:09.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:09 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/39191920' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T12:21:09.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:09 smithi029 ceph-mon[29919]: mgrmap e9: y(active, since 21s) 2024-03-24T12:21:09.569 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:09 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:09.416+0000 7f9d705b7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T12:21:09.569 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:09 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:09.513+0000 7f9d705b7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T12:21:09.850 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:09 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:09.661+0000 7f9d705b7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T12:21:09.851 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:09 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:09.762+0000 7f9d705b7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T12:21:09.851 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:09 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:09.850+0000 7f9d705b7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T12:21:10.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:10 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3411878155' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T12:21:10.385 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:10 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:10.132+0000 7f9d705b7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T12:21:10.385 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:10 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:10.247+0000 7f9d705b7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T12:21:10.386 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:10 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:10.340+0000 7f9d705b7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T12:21:10.661 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:10 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:10.443+0000 7f9d705b7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T12:21:10.662 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:10 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:10.627+0000 7f9d705b7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T12:21:10.664 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-24T12:21:10.665 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T12:21:10.665 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T12:21:10.665 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-24T12:21:10.665 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T12:21:10.665 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-24T12:21:10.665 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for the mgr to restart... 2024-03-24T12:21:10.665 INFO:teuthology.orchestra.run.smithi029.stdout:Waiting for mgr epoch 9... 2024-03-24T12:21:11.920 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:11 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:11.652+0000 7f9d705b7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T12:21:11.920 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:11 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:11.786+0000 7f9d705b7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T12:21:12.418 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:12 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:12.126+0000 7f9d705b7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T12:21:12.418 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:12 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:12.218+0000 7f9d705b7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T12:21:12.419 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:12 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:12.419+0000 7f9d705b7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:21:12.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:12 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:12.608+0000 7f9d705b7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T12:21:13.161 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:12 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:12.900+0000 7f9d705b7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:21:13.162 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:13 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:13.162+0000 7f9d705b7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T12:21:13.420 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:13 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:13.403+0000 7f9d705b7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T12:21:14.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:14 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:14.231+0000 7f9d705b7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T12:21:15.137 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:14 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:14.814+0000 7f9d705b7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T12:21:15.137 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:14 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:14.915+0000 7f9d705b7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T12:21:15.137 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:21:15.137+0000 7f9d705b7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T12:21:15.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: Active manager daemon y restarted 2024-03-24T12:21:15.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: Activating manager daemon y 2024-03-24T12:21:15.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T12:21:15.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: mgrmap e10: y(active, starting, since 0.0031588s) 2024-03-24T12:21:15.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:21:15.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T12:21:15.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T12:21:15.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T12:21:15.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T12:21:15.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: Manager daemon y is now available 2024-03-24T12:21:15.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:21:15.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:21:16.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T12:21:16.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T12:21:16.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:16 smithi029 ceph-mon[29919]: mgrmap e11: y(active, since 1.0054s) 2024-03-24T12:21:16.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout { 2024-03-24T12:21:16.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T12:21:16.800 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T12:21:16.801 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout } 2024-03-24T12:21:16.801 INFO:teuthology.orchestra.run.smithi029.stdout:mgr epoch 9 is available 2024-03-24T12:21:16.801 INFO:teuthology.orchestra.run.smithi029.stdout:Generating a dashboard self-signed certificate... 2024-03-24T12:21:17.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:17 smithi029 ceph-mon[29919]: [24/Mar/2024:12:21:16] ENGINE Bus STARTING 2024-03-24T12:21:17.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:17 smithi029 ceph-mon[29919]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T12:21:17.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:17 smithi029 ceph-mon[29919]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T12:21:17.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:17 smithi029 ceph-mon[29919]: [24/Mar/2024:12:21:16] ENGINE Serving on http://172.21.15.29:8765 2024-03-24T12:21:17.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:17 smithi029 ceph-mon[29919]: [24/Mar/2024:12:21:16] ENGINE Serving on https://172.21.15.29:7150 2024-03-24T12:21:17.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:17 smithi029 ceph-mon[29919]: [24/Mar/2024:12:21:16] ENGINE Bus STARTED 2024-03-24T12:21:18.347 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:18 smithi029 ceph-mon[29919]: mgrmap e12: y(active, since 2s) 2024-03-24T12:21:18.347 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:18 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:18.348 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:18 smithi029 ceph-mon[29919]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:21:18.348 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:18 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:18.348 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:18 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:18.636 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T12:21:18.636 INFO:teuthology.orchestra.run.smithi029.stdout:Creating initial admin user... 2024-03-24T12:21:19.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:19 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:19.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:19 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:20.468 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$JlPqgbYcKnfiQqNuVRuL3OPZLV5edN.Q/ssAD1ute79Ba.mCAjsjK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711282879, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T12:21:20.468 INFO:teuthology.orchestra.run.smithi029.stdout:Fetching dashboard port number... 2024-03-24T12:21:20.721 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:20 smithi029 ceph-mon[29919]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:21:21.893 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:21 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2324068216' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T12:21:21.896 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T12:21:21.905 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T12:21:21.905 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout disabled 2024-03-24T12:21:21.913 INFO:teuthology.orchestra.run.smithi029.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T12:21:21.913 INFO:teuthology.orchestra.run.smithi029.stdout:systemctl: stdout inactive 2024-03-24T12:21:21.913 INFO:teuthology.orchestra.run.smithi029.stdout:firewalld.service is not enabled 2024-03-24T12:21:21.913 INFO:teuthology.orchestra.run.smithi029.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T12:21:21.914 INFO:teuthology.orchestra.run.smithi029.stdout:Ceph Dashboard is now available at: 2024-03-24T12:21:21.914 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:21.914 INFO:teuthology.orchestra.run.smithi029.stdout: URL: https://smithi029.front.sepia.ceph.com:8443/ 2024-03-24T12:21:21.914 INFO:teuthology.orchestra.run.smithi029.stdout: User: admin 2024-03-24T12:21:21.914 INFO:teuthology.orchestra.run.smithi029.stdout: Password: d2q9o9mdxp 2024-03-24T12:21:21.915 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:21.915 INFO:teuthology.orchestra.run.smithi029.stdout:Saving cluster configuration to /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config directory 2024-03-24T12:21:21.915 INFO:teuthology.orchestra.run.smithi029.stdout:Enabling autotune for osd_memory_target 2024-03-24T12:21:22.896 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:22 smithi029 ceph-mon[29919]: mgrmap e13: y(active, since 6s) 2024-03-24T12:21:24.609 INFO:teuthology.orchestra.run.smithi029.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T12:21:24.609 INFO:teuthology.orchestra.run.smithi029.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T12:21:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:21:24.609 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: ceph telemetry on 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout:For more information see: 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:24.610 INFO:teuthology.orchestra.run.smithi029.stdout:Bootstrap complete. 2024-03-24T12:21:24.657 INFO:tasks.cephadm:Fetching config... 2024-03-24T12:21:24.657 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:21:24.657 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T12:21:24.673 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T12:21:24.673 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:21:24.673 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T12:21:24.728 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T12:21:24.728 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:21:24.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T12:21:24.796 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T12:21:24.796 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:21:24.796 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T12:21:24.850 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T12:21:24.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEnUzcXK12/3jrEazwrLTsCaOh/KPG5n6lufyT+vVN+WB/NtEpldiYeuwhAdywY9e0lJMBrk8vexSY/K03ACDswLc+/aO1zmSWLVWlBzmFupMSZYfVJ3j36s+PF9eLBWVStojqTBLCTK/lyk1+himFgmUOY4FFJGr4BJcDK3NlIejxjyTSWZHAIyztIr4j2P+z3Pfob1geOYE4xlmqe+E6XTFoaCB4LBweiFOcP4TCNDiSszDfjX/xCcnm0tNu+sU18h8wnKt5paJmn96dfHDsWFT244N8MzynjgxC0nAe9wXqDkCcBhdlNfJLIjCo8ffjQyD40gdvv43CJYIvZfQiEAdePCcj+OJh9tNyAc7kb8FeJLDxzu8gfyr+xlxgOblFQef0yJqpTwC7VDupvZHwT0JFhpXERnNOtFqzeCxjiCPAVJnB9qs+YsKTQdN7aOGGQg7/fnK/qMRAkhBzSVgzRGJNYtbwvY6WU2ygPJrIVXtYVP3HUl0n5IkF93Znxs8= ceph-bc017092-e9d8-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T12:21:24.934 INFO:teuthology.orchestra.run.smithi029.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEnUzcXK12/3jrEazwrLTsCaOh/KPG5n6lufyT+vVN+WB/NtEpldiYeuwhAdywY9e0lJMBrk8vexSY/K03ACDswLc+/aO1zmSWLVWlBzmFupMSZYfVJ3j36s+PF9eLBWVStojqTBLCTK/lyk1+himFgmUOY4FFJGr4BJcDK3NlIejxjyTSWZHAIyztIr4j2P+z3Pfob1geOYE4xlmqe+E6XTFoaCB4LBweiFOcP4TCNDiSszDfjX/xCcnm0tNu+sU18h8wnKt5paJmn96dfHDsWFT244N8MzynjgxC0nAe9wXqDkCcBhdlNfJLIjCo8ffjQyD40gdvv43CJYIvZfQiEAdePCcj+OJh9tNyAc7kb8FeJLDxzu8gfyr+xlxgOblFQef0yJqpTwC7VDupvZHwT0JFhpXERnNOtFqzeCxjiCPAVJnB9qs+YsKTQdN7aOGGQg7/fnK/qMRAkhBzSVgzRGJNYtbwvY6WU2ygPJrIVXtYVP3HUl0n5IkF93Znxs8= ceph-bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:21:24.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEnUzcXK12/3jrEazwrLTsCaOh/KPG5n6lufyT+vVN+WB/NtEpldiYeuwhAdywY9e0lJMBrk8vexSY/K03ACDswLc+/aO1zmSWLVWlBzmFupMSZYfVJ3j36s+PF9eLBWVStojqTBLCTK/lyk1+himFgmUOY4FFJGr4BJcDK3NlIejxjyTSWZHAIyztIr4j2P+z3Pfob1geOYE4xlmqe+E6XTFoaCB4LBweiFOcP4TCNDiSszDfjX/xCcnm0tNu+sU18h8wnKt5paJmn96dfHDsWFT244N8MzynjgxC0nAe9wXqDkCcBhdlNfJLIjCo8ffjQyD40gdvv43CJYIvZfQiEAdePCcj+OJh9tNyAc7kb8FeJLDxzu8gfyr+xlxgOblFQef0yJqpTwC7VDupvZHwT0JFhpXERnNOtFqzeCxjiCPAVJnB9qs+YsKTQdN7aOGGQg7/fnK/qMRAkhBzSVgzRGJNYtbwvY6WU2ygPJrIVXtYVP3HUl0n5IkF93Znxs8= ceph-bc017092-e9d8-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T12:21:24.994 INFO:teuthology.orchestra.run.smithi190.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEnUzcXK12/3jrEazwrLTsCaOh/KPG5n6lufyT+vVN+WB/NtEpldiYeuwhAdywY9e0lJMBrk8vexSY/K03ACDswLc+/aO1zmSWLVWlBzmFupMSZYfVJ3j36s+PF9eLBWVStojqTBLCTK/lyk1+himFgmUOY4FFJGr4BJcDK3NlIejxjyTSWZHAIyztIr4j2P+z3Pfob1geOYE4xlmqe+E6XTFoaCB4LBweiFOcP4TCNDiSszDfjX/xCcnm0tNu+sU18h8wnKt5paJmn96dfHDsWFT244N8MzynjgxC0nAe9wXqDkCcBhdlNfJLIjCo8ffjQyD40gdvv43CJYIvZfQiEAdePCcj+OJh9tNyAc7kb8FeJLDxzu8gfyr+xlxgOblFQef0yJqpTwC7VDupvZHwT0JFhpXERnNOtFqzeCxjiCPAVJnB9qs+YsKTQdN7aOGGQg7/fnK/qMRAkhBzSVgzRGJNYtbwvY6WU2ygPJrIVXtYVP3HUl0n5IkF93Znxs8= ceph-bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:21:25.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T12:21:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:24 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1971134857' entity='client.admin' 2024-03-24T12:21:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:21:27.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:27.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:27.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi029", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:21:27.898 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:27.899 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:21:27.899 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:28.593 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T12:21:28.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T12:21:29.149 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:28 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2689831816' entity='client.admin' 2024-03-24T12:21:29.149 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:28 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:21:29.149 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:28 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:21:32.028 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi190 2024-03-24T12:21:32.028 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:21:32.029 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.conf 2024-03-24T12:21:32.046 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:21:32.047 DEBUG:teuthology.orchestra.run.smithi190:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:21:32.104 INFO:tasks.cephadm:Adding host smithi190 to orchestrator... 2024-03-24T12:21:32.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch host add smithi190 2024-03-24T12:21:32.344 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:21:32.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:21:32.589 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:21:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:21:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:21:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-24T12:21:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:21:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:21:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T12:21:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:32.590 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:32 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:35.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:35 smithi029 ceph-mon[29919]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi190", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:21:35.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:35 smithi029 ceph-mon[29919]: Deploying cephadm binary to smithi190 2024-03-24T12:21:36.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:36 smithi029 ceph-mon[29919]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:38.472 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:38 smithi029 ceph-mon[29919]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:38.596 INFO:teuthology.orchestra.run.smithi029.stdout:Added host 'smithi190' with addr '172.21.15.190' 2024-03-24T12:21:39.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T12:21:39.512 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:21:39.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:39 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:39.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:39 smithi029 ceph-mon[29919]: Added host smithi190 2024-03-24T12:21:39.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:39 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:21:39.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:39 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:40.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:40 smithi029 ceph-mon[29919]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:40.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:40 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:41.416 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:21:41.416 INFO:teuthology.orchestra.run.smithi029.stdout:[{"addr": "172.21.15.29", "hostname": "smithi029", "labels": [], "status": ""}, {"addr": "172.21.15.190", "hostname": "smithi190", "labels": [], "status": ""}] 2024-03-24T12:21:42.010 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T12:21:42.010 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T12:21:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:21:42.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:42 smithi029 ceph-mon[29919]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:42.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:42.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:42 smithi029 ceph-mon[29919]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:21:44.172 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2024-03-24T12:21:44.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:44 smithi029 ceph-mon[29919]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:44.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:44 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2095531710' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T12:21:44.730 INFO:tasks.cephadm:Adding mon.a on smithi029 2024-03-24T12:21:44.730 INFO:tasks.cephadm:Adding mon.c on smithi029 2024-03-24T12:21:44.730 INFO:tasks.cephadm:Adding mon.b on smithi190 2024-03-24T12:21:44.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi029:172.21.15.29=a;smithi029:[v2:172.21.15.29:3301,v1:172.21.15.29:6790]=c;smithi190:172.21.15.190=b' 2024-03-24T12:21:44.966 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2095531710' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:21:45.428 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:21:45.429 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:45 smithi029 ceph-mon[29919]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T12:21:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:46 smithi029 ceph-mon[29919]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:46.721 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mon update... 2024-03-24T12:21:47.375 DEBUG:teuthology.orchestra.run.smithi029:mon.c> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T12:21:47.378 DEBUG:teuthology.orchestra.run.smithi190:mon.b> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T12:21:47.381 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T12:21:47.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T12:21:48.141 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:21:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:47 smithi029 ceph-mon[29919]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi029:[v2:172.21.15.29:3301,v1:172.21.15.29:6790]=c;smithi190:172.21.15.190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:21:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:47 smithi029 ceph-mon[29919]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi029:[v2:172.21.15.29:3301,v1:172.21.15.29:6790]=c;smithi190:172.21.15.190=b;count:3 2024-03-24T12:21:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:47 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:47 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:21:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:47 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:21:48.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:47 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:21:48.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:47 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:48.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:47 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:21:48.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:47 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:21:48.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:47 smithi029 ceph-mon[29919]: Deploying daemon mon.b on smithi190 2024-03-24T12:21:49.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:48 smithi029 ceph-mon[29919]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:50 smithi029 ceph-mon[29919]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:51.479 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:21:51.479 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":1,"fsid":"bc017092-e9d8-11ee-95cd-87774f69a715","modified":"2024-03-24T12:20:15.541022Z","created":"2024-03-24T12:20:15.541022Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T12:21:51.481 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 1 2024-03-24T12:21:51.774 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:51 smithi190 systemd[1]: Starting Ceph mon.b for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:21:52.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:51 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/4082753022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:21:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 podman[33710]: 2024-03-24 12:21:52.209723764 +0000 UTC m=+0.114323493 container create 23685832c25178b2296b2b643f0c117d36e6b71d971f5615cf76af52cdd6b1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T12:21:52.323 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 podman[33710]: 2024-03-24 12:21:52.113037353 +0000 UTC m=+0.017637089 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:21:52.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 podman[33710]: 2024-03-24 12:21:52.41219004 +0000 UTC m=+0.316789768 container init 23685832c25178b2296b2b643f0c117d36e6b71d971f5615cf76af52cdd6b1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:21:52.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 podman[33710]: 2024-03-24 12:21:52.417904836 +0000 UTC m=+0.322504573 container start 23685832c25178b2296b2b643f0c117d36e6b71d971f5615cf76af52cdd6b1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.build-date=20240307) 2024-03-24T12:21:52.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T12:21:52.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: pidfile_write: ignore empty --pid-file 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: load: jerasure load: lrc 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: RocksDB version: 7.9.2 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Git sha 0 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: DB SUMMARY 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: DB Session ID: W2F1WJT28IPFC06H4MDZ 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: CURRENT file: CURRENT 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-24T12:21:52.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 511 ; 2024-03-24T12:21:52.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.error_if_exists: 0 2024-03-24T12:21:52.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.create_if_missing: 0 2024-03-24T12:21:52.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.paranoid_checks: 1 2024-03-24T12:21:52.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T12:21:52.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T12:21:52.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T12:21:52.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.env: 0x560144765ae0 2024-03-24T12:21:52.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T12:21:52.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.info_log: 0x5601465bf8c0 2024-03-24T12:21:52.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T12:21:52.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.statistics: (nil) 2024-03-24T12:21:52.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.use_fsync: 0 2024-03-24T12:21:52.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_log_file_size: 0 2024-03-24T12:21:52.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T12:21:52.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T12:21:52.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T12:21:52.890 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.allow_fallocate: 1 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.use_direct_reads: 0 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.db_log_dir: 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.wal_dir: 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T12:21:52.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.write_buffer_manager: 0x560146593400 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T12:21:52.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.unordered_write: 0 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.row_cache: None 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.wal_filter: None 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.two_write_queues: 0 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T12:21:52.893 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.wal_compression: 0 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.atomic_flush: 0 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.log_readahead_size: 0 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T12:21:52.894 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_background_jobs: 2 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_background_compactions: -1 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_subcompactions: 1 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_open_files: -1 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T12:21:52.895 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_background_flushes: -1 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Compression algorithms supported: 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: kZSTD supported: 0 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: kXpressCompression supported: 0 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: kZlibCompression supported: 1 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: kSnappyCompression supported: 1 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: kLZ4Compression supported: 1 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: kBZip2Compression supported: 0 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-24T12:21:52.896 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.merge_operator: 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_filter: None 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5601465bf9c0) 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout: cache_index_and_filter_blocks: 1 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout: pin_top_level_index_and_filter: 1 2024-03-24T12:21:52.897 INFO:journalctl@ceph.mon.b.smithi190.stdout: index_type: 0 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: data_block_index_type: 0 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: index_shortening: 1 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: checksum: 4 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: no_block_cache: 0 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: block_cache: 0x5601465c3090 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: block_cache_name: BinnedLRUCache 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: block_cache_options: 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: capacity : 536870912 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: num_shard_bits : 4 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: strict_capacity_limit : 0 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: high_pri_pool_ratio: 0.000 2024-03-24T12:21:52.898 INFO:journalctl@ceph.mon.b.smithi190.stdout: block_cache_compressed: (nil) 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: persistent_cache: (nil) 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: block_size: 4096 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: block_size_deviation: 10 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: block_restart_interval: 16 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: index_block_restart_interval: 1 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: metadata_block_size: 4096 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: partition_filters: 0 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: use_delta_encoding: 1 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: filter_policy: bloomfilter 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: whole_key_filtering: 1 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: verify_compression: 0 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: read_amp_bytes_per_bit: 0 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: format_version: 5 2024-03-24T12:21:52.899 INFO:journalctl@ceph.mon.b.smithi190.stdout: enable_index_compression: 1 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout: block_align: 0 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout: max_auto_readahead_size: 262144 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout: prepopulate_block_cache: 0 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout: initial_auto_readahead_size: 8192 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compression: NoCompression 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.num_levels: 7 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T12:21:52.900 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T12:21:52.901 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T12:21:52.902 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T12:21:52.903 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.table_properties_collectors: 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.inplace_update_support: 0 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T12:21:52.904 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.bloom_locality: 0 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.max_successive_merges: 0 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.ttl: 2592000 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T12:21:52.905 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.enable_blob_files: false 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.min_blob_size: 0 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 99c7d1c5-39ba-4f0e-9fe3-b6c2e1ef6090 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711282912447605, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T12:21:52.906 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711282912448028, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711282912, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "99c7d1c5-39ba-4f0e-9fe3-b6c2e1ef6090", "db_session_id": "W2F1WJT28IPFC06H4MDZ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711282912448107, "job": 1, "event": "recovery_finished"} 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5601466b4000 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: DB pointer 0x56014669e000 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout: ** DB Stats ** 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T12:21:52.907 INFO:journalctl@ceph.mon.b.smithi190.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: ** Compaction Stats [default] ** 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: ** Compaction Stats [default] ** 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T12:21:52.908 INFO:journalctl@ceph.mon.b.smithi190.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T12:21:52.909 INFO:journalctl@ceph.mon.b.smithi190.stdout: Block cache BinnedLRUCache@0x5601465c3090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout: 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: using public_addr v2:172.21.15.190:0/0 -> [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: starting mon.b rank -1 at public addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] at bind addrs [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(???) e0 preinit fsid bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).mds e1 new map 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout: e1 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout: legacy client fscid: -1 2024-03-24T12:21:52.910 INFO:journalctl@ceph.mon.b.smithi190.stdout: 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout: No filesystems configured 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:52.911 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2095531710' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2095531710' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-24T12:21:52.912 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi029:[v2:172.21.15.29:3301,v1:172.21.15.29:6790]=c;smithi190:172.21.15.190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi029:[v2:172.21.15.29:3301,v1:172.21.15.29:6790]=c;smithi190:172.21.15.190=b;count:3 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:21:52.913 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:21:52.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: Deploying daemon mon.b on smithi190 2024-03-24T12:21:52.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:52.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:52.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/4082753022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:21:52.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 ceph-mon[33748]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T12:21:52.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 bash[33710]: 23685832c25178b2296b2b643f0c117d36e6b71d971f5615cf76af52cdd6b1e5 2024-03-24T12:21:52.914 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:52 smithi190 systemd[1]: Started Ceph mon.b for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:21:53.026 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T12:21:53.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T12:21:53.261 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:21:57.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:21:57.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: mon.a calling monitor election 2024-03-24T12:21:57.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:57.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: mon.b calling monitor election 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: fsmap 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: mgrmap e13: y(active, since 42s) 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: overall HEALTH_OK 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:57.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:57.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:57.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:21:57.888 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:21:57.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:21:57.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: mon.a calling monitor election 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: mon.b calling monitor election 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T12:21:57.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: fsmap 2024-03-24T12:21:57.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T12:21:57.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: mgrmap e13: y(active, since 42s) 2024-03-24T12:21:57.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: overall HEALTH_OK 2024-03-24T12:21:57.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:57.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:57.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:21:57.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:21:57.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:21:58.334 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:21:58.337 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":2,"fsid":"bc017092-e9d8-11ee-95cd-87774f69a715","modified":"2024-03-24T12:21:52.471462Z","created":"2024-03-24T12:20:15.541022Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T12:21:58.338 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 2 2024-03-24T12:21:58.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:58 smithi190 ceph-mon[33748]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:58.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:58 smithi190 ceph-mon[33748]: Deploying daemon mon.c on smithi029 2024-03-24T12:21:58.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:58 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/508022424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:21:58.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:21:58 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:58 smithi029 ceph-mon[29919]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:21:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:58 smithi029 ceph-mon[29919]: Deploying daemon mon.c on smithi029 2024-03-24T12:21:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:58 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/508022424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:21:58.838 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:21:58 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:21:59.976 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T12:21:59.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T12:22:00.224 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:22:00.624 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:00 smithi190 ceph-mon[33748]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:00.828 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:00 smithi029 ceph-mon[29919]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:00.829 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:00 smithi029 systemd[1]: Starting Ceph mon.c for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:22:01.107 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:00 smithi029 podman[41462]: 2024-03-24 12:22:00.73224157 +0000 UTC m=+0.017819429 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:22:01.107 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:00 smithi029 podman[41462]: 2024-03-24 12:22:00.837956749 +0000 UTC m=+0.123534653 container create e0d35b8f460ba5d45f01a1daa188e8226a0f02594d1f400699e1d2a720caa30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-c, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-03-24T12:22:01.107 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 podman[41462]: 2024-03-24 12:22:01.108956301 +0000 UTC m=+0.394534169 container init e0d35b8f460ba5d45f01a1daa188e8226a0f02594d1f400699e1d2a720caa30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:22:01.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 podman[41462]: 2024-03-24 12:22:01.114534964 +0000 UTC m=+0.400112831 container start e0d35b8f460ba5d45f01a1daa188e8226a0f02594d1f400699e1d2a720caa30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-c, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:22:01.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T12:22:01.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: ceph version 19.0.0-1667-gdb0330b1 (db0330b1e4e2470d52b750e251e55a522b4f7d69) squid (dev), process ceph-mon, pid 7 2024-03-24T12:22:01.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: pidfile_write: ignore empty --pid-file 2024-03-24T12:22:01.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: load: jerasure load: lrc 2024-03-24T12:22:01.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: RocksDB version: 7.9.2 2024-03-24T12:22:01.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Git sha 0 2024-03-24T12:22:01.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Compile date 2024-03-16 10:36:11 2024-03-24T12:22:01.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: DB SUMMARY 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: DB Session ID: 55SW9VM3BUM4214UHYE0 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: CURRENT file: CURRENT 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.error_if_exists: 0 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.create_if_missing: 0 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.paranoid_checks: 1 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T12:22:01.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.env: 0x55fd5e16cae0 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.info_log: 0x55fd6080b8c0 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.statistics: (nil) 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.use_fsync: 0 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_log_file_size: 0 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.allow_fallocate: 1 2024-03-24T12:22:01.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.use_direct_reads: 0 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.db_log_dir: 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.wal_dir: 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T12:22:01.424 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.write_buffer_manager: 0x55fd607df400 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.unordered_write: 0 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T12:22:01.425 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.row_cache: None 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.wal_filter: None 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.two_write_queues: 0 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.wal_compression: 0 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.atomic_flush: 0 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T12:22:01.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T12:22:01.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.log_readahead_size: 0 2024-03-24T12:22:01.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T12:22:01.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T12:22:01.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T12:22:01.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T12:22:01.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T12:22:01.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T12:22:01.428 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_background_jobs: 2 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_background_compactions: -1 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_subcompactions: 1 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_open_files: -1 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T12:22:01.429 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_background_flushes: -1 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Compression algorithms supported: 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: kZSTD supported: 0 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: kXpressCompression supported: 0 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: kZlibCompression supported: 1 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: kSnappyCompression supported: 1 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: kLZ4Compression supported: 1 2024-03-24T12:22:01.430 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: kBZip2Compression supported: 0 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.merge_operator: 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_filter: None 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T12:22:01.431 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55fd6080b9c0) 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: cache_index_and_filter_blocks: 1 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: pin_top_level_index_and_filter: 1 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: index_type: 0 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: data_block_index_type: 0 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: index_shortening: 1 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: checksum: 4 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: no_block_cache: 0 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_cache: 0x55fd6080f090 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_cache_name: BinnedLRUCache 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_cache_options: 2024-03-24T12:22:01.432 INFO:journalctl@ceph.mon.c.smithi029.stdout: capacity : 536870912 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: num_shard_bits : 4 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: strict_capacity_limit : 0 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: high_pri_pool_ratio: 0.000 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_cache_compressed: (nil) 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: persistent_cache: (nil) 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_size: 4096 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_size_deviation: 10 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_restart_interval: 16 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: index_block_restart_interval: 1 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: metadata_block_size: 4096 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: partition_filters: 0 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: use_delta_encoding: 1 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: filter_policy: bloomfilter 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: whole_key_filtering: 1 2024-03-24T12:22:01.433 INFO:journalctl@ceph.mon.c.smithi029.stdout: verify_compression: 0 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout: read_amp_bytes_per_bit: 0 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout: format_version: 5 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout: enable_index_compression: 1 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout: block_align: 0 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout: max_auto_readahead_size: 262144 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout: prepopulate_block_cache: 0 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout: initial_auto_readahead_size: 8192 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compression: NoCompression 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T12:22:01.434 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.num_levels: 7 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T12:22:01.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T12:22:01.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T12:22:01.437 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T12:22:01.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.table_properties_collectors: 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.inplace_update_support: 0 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.bloom_locality: 0 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.max_successive_merges: 0 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T12:22:01.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.ttl: 2592000 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.enable_blob_files: false 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.min_blob_size: 0 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T12:22:01.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 79ba2e48-e10c-4f25-a300-46a75c8c682c 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711282921144965, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711282921145377, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711282921, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "79ba2e48-e10c-4f25-a300-46a75c8c682c", "db_session_id": "55SW9VM3BUM4214UHYE0", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711282921145461, "job": 1, "event": "recovery_finished"} 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55fd60900000 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: DB pointer 0x55fd608ea000 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T12:22:01.441 INFO:journalctl@ceph.mon.c.smithi029.stdout: ** DB Stats ** 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T12:22:01.442 INFO:journalctl@ceph.mon.c.smithi029.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: ** Compaction Stats [default] ** 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T12:22:01.443 INFO:journalctl@ceph.mon.c.smithi029.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout: Block cache BinnedLRUCache@0x55fd6080f090#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: using public_addrv [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: starting mon.c rank -1 at public addrs [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] at bind addrs [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(???) e0 preinit fsid bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).mds e1 new map 2024-03-24T12:22:01.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout: e1 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout: legacy client fscid: -1 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout: 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout: No filesystems configured 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T12:22:01.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2095531710' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2095531710' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi190", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:22:01.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi029:172.21.15.29=a;smithi029:[v2:172.21.15.29:3301,v1:172.21.15.29:6790]=c;smithi190:172.21.15.190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: Saving service mon spec with placement smithi029:172.21.15.29=a;smithi029:[v2:172.21.15.29:3301,v1:172.21.15.29:6790]=c;smithi190:172.21.15.190=b;count:3 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:01.447 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: Deploying daemon mon.b on smithi190 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/4082753022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.a calling monitor election 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:22:01.448 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.b calling monitor election 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: monmap e2: 2 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: fsmap 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mgrmap e13: y(active, since 42s) 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: overall HEALTH_OK 2024-03-24T12:22:01.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: Deploying daemon mon.c on smithi029 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/508022424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 ceph-mon[41500]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 bash[41462]: e0d35b8f460ba5d45f01a1daa188e8226a0f02594d1f400699e1d2a720caa30d 2024-03-24T12:22:01.450 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:01 smithi029 systemd[1]: Started Ceph mon.c for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:22:06.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:06.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:22:06.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: mon.a calling monitor election 2024-03-24T12:22:06.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: mon.b calling monitor election 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: mon.c calling monitor election 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T12:22:06.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: fsmap 2024-03-24T12:22:06.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T12:22:06.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: mgrmap e13: y(active, since 51s) 2024-03-24T12:22:06.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: overall HEALTH_OK 2024-03-24T12:22:06.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.638 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:06.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: mon.a calling monitor election 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: mon.b calling monitor election 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: mon.c calling monitor election 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T12:22:06.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: fsmap 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: mgrmap e13: y(active, since 51s) 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: overall HEALTH_OK 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: mon.a calling monitor election 2024-03-24T12:22:06.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: mon.b calling monitor election 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: mon.c calling monitor election 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: monmap e3: 3 mons at {a=[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0],b=[v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0],c=[v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: fsmap 2024-03-24T12:22:06.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T12:22:06.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: mgrmap e13: y(active, since 51s) 2024-03-24T12:22:06.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: overall HEALTH_OK 2024-03-24T12:22:06.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:06.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:07.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:07 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:07.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:07 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:07.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:07 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:07.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:07 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:07.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:07 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:07.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:07 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:22:08.035 INFO:teuthology.orchestra.run.smithi190.stdout: 2024-03-24T12:22:08.035 INFO:teuthology.orchestra.run.smithi190.stdout:{"epoch":3,"fsid":"bc017092-e9d8-11ee-95cd-87774f69a715","modified":"2024-03-24T12:22:01.165611Z","created":"2024-03-24T12:20:15.541022Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3300","nonce":0},{"type":"v1","addr":"172.21.15.29:6789","nonce":0}]},"addr":"172.21.15.29:6789/0","public_addr":"172.21.15.29:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:3300","nonce":0},{"type":"v1","addr":"172.21.15.190:6789","nonce":0}]},"addr":"172.21.15.190:6789/0","public_addr":"172.21.15.190:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:3301","nonce":0},{"type":"v1","addr":"172.21.15.29:6790","nonce":0}]},"addr":"172.21.15.29:6790/0","public_addr":"172.21.15.29:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-24T12:22:08.038 INFO:teuthology.orchestra.run.smithi190.stderr:dumped monmap epoch 3 2024-03-24T12:22:08.679 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/3008107221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.680 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:08.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T12:22:08.681 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:08.705 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T12:22:08.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T12:22:08.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:08.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:08.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:08.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-24T12:22:08.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-24T12:22:08.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/3008107221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:22:08.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:08.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-24T12:22:08.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/3008107221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T12:22:08.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:09.657 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:22:09.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:09 smithi190 ceph-mon[33748]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T12:22:09.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:09 smithi190 ceph-mon[33748]: Reconfiguring daemon mon.a on smithi029 2024-03-24T12:22:09.911 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:09 smithi029 ceph-mon[29919]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T12:22:09.911 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:09 smithi029 ceph-mon[29919]: Reconfiguring daemon mon.a on smithi029 2024-03-24T12:22:09.911 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:09 smithi029 ceph-mon[41500]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T12:22:09.911 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:09 smithi029 ceph-mon[41500]: Reconfiguring daemon mon.a on smithi029 2024-03-24T12:22:11.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[29919]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:11.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:11.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:11.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:11.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:11.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T12:22:11.061 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:11.061 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[41500]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:11.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:11.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:11.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:11.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:11.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T12:22:11.062 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:11.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:10 smithi190 ceph-mon[33748]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:11.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:11.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:11.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:11.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:11.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T12:22:11.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:12.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:11 smithi029 ceph-mon[29919]: Reconfiguring mon.c (monmap changed)... 2024-03-24T12:22:12.036 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:11 smithi029 ceph-mon[29919]: Reconfiguring daemon mon.c on smithi029 2024-03-24T12:22:12.036 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:11 smithi029 ceph-mon[41500]: Reconfiguring mon.c (monmap changed)... 2024-03-24T12:22:12.036 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:11 smithi029 ceph-mon[41500]: Reconfiguring daemon mon.c on smithi029 2024-03-24T12:22:12.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:11 smithi190 ceph-mon[33748]: Reconfiguring mon.c (monmap changed)... 2024-03-24T12:22:12.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:11 smithi190 ceph-mon[33748]: Reconfiguring daemon mon.c on smithi029 2024-03-24T12:22:13.018 INFO:teuthology.orchestra.run.smithi029.stdout:# minimal ceph.conf for bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:22:13.018 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2024-03-24T12:22:13.018 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:22:13.018 INFO:teuthology.orchestra.run.smithi029.stdout: mon_host = [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] [v2:172.21.15.190:3300/0,v1:172.21.15.190:6789/0] [v2:172.21.15.29:3301/0,v1:172.21.15.29:6790/0] 2024-03-24T12:22:13.031 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[41500]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:13.031 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:13.031 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:13.031 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:13.031 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T12:22:13.031 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:13.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[29919]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:13.031 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:13.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:13.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:13.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T12:22:13.032 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:12 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:13.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:12 smithi190 ceph-mon[33748]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:13.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:12 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:13.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:12 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:13.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:12 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T12:22:13.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:12 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T12:22:13.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:12 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:13.836 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T12:22:13.836 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:22:13.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T12:22:13.866 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:22:13.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:22:13.933 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:22:13.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T12:22:13.963 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:22:13.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:22:14.031 INFO:tasks.cephadm:Adding mgr.y on smithi029 2024-03-24T12:22:14.031 INFO:tasks.cephadm:Adding mgr.x on smithi190 2024-03-24T12:22:14.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch apply mgr '2;smithi029=y;smithi190=x' 2024-03-24T12:22:14.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:13 smithi190 ceph-mon[33748]: Reconfiguring mon.b (monmap changed)... 2024-03-24T12:22:14.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:13 smithi190 ceph-mon[33748]: Reconfiguring daemon mon.b on smithi190 2024-03-24T12:22:14.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:13 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/86022372' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:14.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:13 smithi029 ceph-mon[41500]: Reconfiguring mon.b (monmap changed)... 2024-03-24T12:22:14.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:13 smithi029 ceph-mon[41500]: Reconfiguring daemon mon.b on smithi190 2024-03-24T12:22:14.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:13 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/86022372' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:14.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:13 smithi029 ceph-mon[29919]: Reconfiguring mon.b (monmap changed)... 2024-03-24T12:22:14.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:13 smithi029 ceph-mon[29919]: Reconfiguring daemon mon.b on smithi190 2024-03-24T12:22:14.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:13 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/86022372' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:14.405 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:22:15.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:14 smithi190 ceph-mon[33748]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:15.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:14 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:15.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:14 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:15.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:14 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:15.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:14 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:15.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:14 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:15.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:14 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:15.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[29919]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:15.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:15.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:15.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:15.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:15.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:15.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:15.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[41500]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:15.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:15.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:15.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:15.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:15.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:15.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:14 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:16.263 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled mgr update... 2024-03-24T12:22:16.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:16 smithi190 ceph-mon[33748]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:16.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:16.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:16.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:16.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:16.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:16.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T12:22:16.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T12:22:16.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T12:22:16.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:16.948 DEBUG:teuthology.orchestra.run.smithi190:mgr.x> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T12:22:16.993 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T12:22:16.993 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:22:16.993 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T12:22:17.013 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T12:22:17.013 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2024-03-24T12:22:17.070 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T12:22:17.070 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T12:22:17.070 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-24T12:22:17.070 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T12:22:17.071 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T12:22:17.071 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-24 12:22:00.552084738 +0000 2024-03-24T12:22:17.071 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-24 12:15:13.548848502 +0000 2024-03-24T12:22:17.071 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-24 12:15:13.548848502 +0000 2024-03-24T12:22:17.071 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-24 12:15:13.548848502 +0000 2024-03-24T12:22:17.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T12:22:17.137 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[29919]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:17.138 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:17.138 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:17.138 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:17.138 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:17.138 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:17.138 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T12:22:17.138 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T12:22:17.138 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T12:22:17.138 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:17.138 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[41500]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:17.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:17.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:17.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:17.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:17.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:17.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T12:22:17.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T12:22:17.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T12:22:17.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:17.140 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-24T12:22:17.140 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-24T12:22:17.140 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000105251 s, 4.9 MB/s 2024-03-24T12:22:17.141 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T12:22:17.198 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2024-03-24T12:22:17.254 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T12:22:17.254 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T12:22:17.254 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T12:22:17.254 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T12:22:17.254 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T12:22:17.254 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-24 12:22:00.552084738 +0000 2024-03-24T12:22:17.254 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-24 12:15:13.892842648 +0000 2024-03-24T12:22:17.254 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-24 12:15:13.892842648 +0000 2024-03-24T12:22:17.254 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-24 12:15:13.892842648 +0000 2024-03-24T12:22:17.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T12:22:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-24T12:22:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-24T12:22:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 9.7194e-05 s, 5.3 MB/s 2024-03-24T12:22:17.322 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T12:22:17.379 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2024-03-24T12:22:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T12:22:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T12:22:17.435 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T12:22:17.435 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T12:22:17.435 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T12:22:17.435 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-24 12:22:00.553084721 +0000 2024-03-24T12:22:17.435 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-24 12:15:14.223837016 +0000 2024-03-24T12:22:17.435 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-24 12:15:14.223837016 +0000 2024-03-24T12:22:17.435 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-24 12:15:14.223837016 +0000 2024-03-24T12:22:17.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T12:22:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-24T12:22:17.502 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-24T12:22:17.503 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000114528 s, 4.5 MB/s 2024-03-24T12:22:17.503 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T12:22:17.560 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2024-03-24T12:22:17.615 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T12:22:17.616 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T12:22:17.616 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T12:22:17.616 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T12:22:17.616 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T12:22:17.616 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2024-03-24 12:22:00.553084721 +0000 2024-03-24T12:22:17.616 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2024-03-24 12:15:14.555831367 +0000 2024-03-24T12:22:17.616 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2024-03-24 12:15:14.555831367 +0000 2024-03-24T12:22:17.616 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2024-03-24 12:15:14.555831367 +0000 2024-03-24T12:22:17.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T12:22:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2024-03-24T12:22:17.683 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2024-03-24T12:22:17.683 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000111682 s, 4.6 MB/s 2024-03-24T12:22:17.684 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T12:22:17.740 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:22:17.740 DEBUG:teuthology.orchestra.run.smithi190:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T12:22:17.756 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T12:22:17.757 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_1 2024-03-24T12:22:17.812 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T12:22:17.812 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T12:22:17.812 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-24T12:22:17.812 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T12:22:17.812 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T12:22:17.812 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-24 12:21:51.514866735 +0000 2024-03-24T12:22:17.813 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-24 12:15:12.119274748 +0000 2024-03-24T12:22:17.813 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-24 12:15:12.119274748 +0000 2024-03-24T12:22:17.813 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-24 12:15:12.119274748 +0000 2024-03-24T12:22:17.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T12:22:17.880 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-24T12:22:17.880 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-24T12:22:17.881 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00024522 s, 2.1 MB/s 2024-03-24T12:22:17.881 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T12:22:17.937 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_2 2024-03-24T12:22:17.993 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T12:22:17.993 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T12:22:17.993 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-24T12:22:17.993 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T12:22:17.993 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T12:22:17.993 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-24 12:21:51.514866735 +0000 2024-03-24T12:22:17.993 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-24 12:15:12.453265372 +0000 2024-03-24T12:22:17.993 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-24 12:15:12.453265372 +0000 2024-03-24T12:22:17.993 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-24 12:15:12.453265372 +0000 2024-03-24T12:22:17.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T12:22:18.059 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-24T12:22:18.059 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-24T12:22:18.059 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000196474 s, 2.6 MB/s 2024-03-24T12:22:18.060 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T12:22:18.115 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_3 2024-03-24T12:22:18.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:17 smithi190 ceph-mon[33748]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi029=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:22:18.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:17 smithi190 ceph-mon[33748]: Saving service mgr spec with placement smithi029=y;smithi190=x;count:2 2024-03-24T12:22:18.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:17 smithi190 ceph-mon[33748]: Deploying daemon mgr.x on smithi190 2024-03-24T12:22:18.149 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T12:22:18.149 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T12:22:18.149 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-24T12:22:18.149 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T12:22:18.149 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T12:22:18.150 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-24 12:21:51.515866707 +0000 2024-03-24T12:22:18.150 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-24 12:15:12.810255349 +0000 2024-03-24T12:22:18.150 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-24 12:15:12.810255349 +0000 2024-03-24T12:22:18.150 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-24 12:15:12.810255349 +0000 2024-03-24T12:22:18.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T12:22:18.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:17 smithi029 ceph-mon[29919]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi029=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:22:18.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:17 smithi029 ceph-mon[29919]: Saving service mgr spec with placement smithi029=y;smithi190=x;count:2 2024-03-24T12:22:18.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:17 smithi029 ceph-mon[29919]: Deploying daemon mgr.x on smithi190 2024-03-24T12:22:18.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:17 smithi029 ceph-mon[41500]: from='client.24112 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi029=y;smithi190=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:22:18.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:17 smithi029 ceph-mon[41500]: Saving service mgr spec with placement smithi029=y;smithi190=x;count:2 2024-03-24T12:22:18.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:17 smithi029 ceph-mon[41500]: Deploying daemon mgr.x on smithi190 2024-03-24T12:22:18.215 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-24T12:22:18.215 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-24T12:22:18.215 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.000267153 s, 1.9 MB/s 2024-03-24T12:22:18.217 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T12:22:18.272 DEBUG:teuthology.orchestra.run.smithi190:> stat /dev/vg_nvme/lv_4 2024-03-24T12:22:18.328 INFO:teuthology.orchestra.run.smithi190.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T12:22:18.328 INFO:teuthology.orchestra.run.smithi190.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T12:22:18.328 INFO:teuthology.orchestra.run.smithi190.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-24T12:22:18.328 INFO:teuthology.orchestra.run.smithi190.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T12:22:18.328 INFO:teuthology.orchestra.run.smithi190.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T12:22:18.328 INFO:teuthology.orchestra.run.smithi190.stdout:Access: 2024-03-24 12:21:51.515866707 +0000 2024-03-24T12:22:18.328 INFO:teuthology.orchestra.run.smithi190.stdout:Modify: 2024-03-24 12:15:13.167245326 +0000 2024-03-24T12:22:18.328 INFO:teuthology.orchestra.run.smithi190.stdout:Change: 2024-03-24 12:15:13.167245326 +0000 2024-03-24T12:22:18.328 INFO:teuthology.orchestra.run.smithi190.stdout: Birth: 2024-03-24 12:15:13.167245326 +0000 2024-03-24T12:22:18.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T12:22:18.427 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records in 2024-03-24T12:22:18.427 INFO:teuthology.orchestra.run.smithi190.stderr:1+0 records out 2024-03-24T12:22:18.427 INFO:teuthology.orchestra.run.smithi190.stderr:512 bytes copied, 0.00017639 s, 2.9 MB/s 2024-03-24T12:22:18.428 DEBUG:teuthology.orchestra.run.smithi190:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T12:22:18.442 INFO:tasks.cephadm:Deploying osd.0 on smithi029 with /dev/vg_nvme/lv_4... 2024-03-24T12:22:18.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T12:22:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:22:18.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:18 smithi190 ceph-mon[33748]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:19.074 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:18 smithi190 podman[35841]: 2024-03-24 12:22:18.848298111 +0000 UTC m=+0.126478844 container create f6342263b530aba756661fb81e337ec2c0533b984c5cd8875757d54f70a6a3d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T12:22:19.074 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:19 smithi190 podman[35841]: 2024-03-24 12:22:19.075726977 +0000 UTC m=+0.353907709 container init f6342263b530aba756661fb81e337ec2c0533b984c5cd8875757d54f70a6a3d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T12:22:19.077 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:18 smithi029 ceph-mon[29919]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:19.077 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:18 smithi029 ceph-mon[41500]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:19.385 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:19 smithi190 podman[35841]: 2024-03-24 12:22:19.080986062 +0000 UTC m=+0.359166796 container start f6342263b530aba756661fb81e337ec2c0533b984c5cd8875757d54f70a6a3d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T12:22:19.386 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:19 smithi190 ceph-mgr[35879]: -- 172.21.15.190:0/1734661112 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557feea941a0 con 0x557feea6b000 2024-03-24T12:22:19.386 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:19 smithi190 bash[35841]: f6342263b530aba756661fb81e337ec2c0533b984c5cd8875757d54f70a6a3d5 2024-03-24T12:22:19.386 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:19 smithi190 systemd[1]: Started Ceph mgr.x for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:22:20.136 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:19 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:19.637+0000 7f8d6b171200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T12:22:20.136 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:19 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:19.732+0000 7f8d6b171200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T12:22:20.412 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:20.412+0000 7f8d6b171200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T12:22:20.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-mon[33748]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:20.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:20.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[29919]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:20.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:20.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[41500]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:20.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:20.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:20.729 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T12:22:20.729 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T12:22:20.729 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-24T12:22:20.729 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-24T12:22:20.729 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00883356 s, 1.2 GB/s 2024-03-24T12:22:20.729 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-24T12:22:20.757 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:20.506+0000 7f8d6b171200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T12:22:20.757 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:20.659+0000 7f8d6b171200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T12:22:20.757 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:20.757+0000 7f8d6b171200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T12:22:21.112 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:20 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:20.840+0000 7f8d6b171200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T12:22:21.113 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:21.113+0000 7f8d6b171200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T12:22:21.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_4 2024-03-24T12:22:21.385 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:21.223+0000 7f8d6b171200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T12:22:21.386 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:21.309+0000 7f8d6b171200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T12:22:21.666 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:21.403+0000 7f8d6b171200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T12:22:21.666 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:21.580+0000 7f8d6b171200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T12:22:21.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:21.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:21.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:21.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:21.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:21.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[29919]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T12:22:21.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T12:22:21.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T12:22:21.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:21.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[29919]: Reconfiguring daemon mgr.y on smithi029 2024-03-24T12:22:21.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:21.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:21.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:21.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:21.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:21.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[41500]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T12:22:21.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T12:22:21.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T12:22:21.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:21.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:21 smithi029 ceph-mon[41500]: Reconfiguring daemon mgr.y on smithi029 2024-03-24T12:22:22.020 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:22:22.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:22.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:22.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:22.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:22.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:22.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-mon[33748]: Reconfiguring mgr.y (unknown last config time)... 2024-03-24T12:22:22.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T12:22:22.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T12:22:22.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:22.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:21 smithi190 ceph-mon[33748]: Reconfiguring daemon mgr.y on smithi029 2024-03-24T12:22:22.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:22 smithi190 ceph-mon[33748]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:22.886 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:22 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:22.546+0000 7f8d6b171200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T12:22:22.886 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:22 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:22.674+0000 7f8d6b171200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T12:22:22.933 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:22 smithi029 ceph-mon[29919]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:22.934 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:22 smithi029 ceph-mon[41500]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:23.300 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:23.018+0000 7f8d6b171200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T12:22:23.300 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:23.108+0000 7f8d6b171200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T12:22:23.300 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:23.300+0000 7f8d6b171200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:22:23.635 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:23.484+0000 7f8d6b171200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T12:22:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:23.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:23.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:23.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:23.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:23.921 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:23.921 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:23.921 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:23.921 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:23.921 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:23.921 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:23 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:24.054 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:23.766+0000 7f8d6b171200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:22:24.054 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:24 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:24.053+0000 7f8d6b171200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T12:22:24.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:24.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:24.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:24.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:24.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:24.055 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:23 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:24.385 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:24 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:24.285+0000 7f8d6b171200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T12:22:25.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:24 smithi029 ceph-mon[41500]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:25.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:22:25.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:22:25.049 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:25.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:24 smithi029 ceph-mon[29919]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:25.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:22:25.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:22:25.050 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:25.072 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:24 smithi190 ceph-mon[33748]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:25.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:22:25.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:22:25.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:25.344 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:25 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:25.072+0000 7f8d6b171200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T12:22:25.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:25 smithi190 ceph-mon[33748]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:22:25.991 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:25 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:25.668+0000 7f8d6b171200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T12:22:25.991 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:25 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:25.770+0000 7f8d6b171200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T12:22:25.991 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:22:25 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:22:25.990+0000 7f8d6b171200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T12:22:26.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:25 smithi029 ceph-mon[29919]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:22:26.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:25 smithi029 ceph-mon[41500]: from='client.24122 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:22:27.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:26 smithi190 ceph-mon[33748]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:27.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:26 smithi190 ceph-mon[33748]: Standby manager daemon x started 2024-03-24T12:22:27.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:26 smithi190 ceph-mon[33748]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T12:22:27.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:26 smithi190 ceph-mon[33748]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T12:22:27.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:26 smithi190 ceph-mon[33748]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T12:22:27.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:26 smithi190 ceph-mon[33748]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T12:22:27.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[41500]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:27.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[41500]: Standby manager daemon x started 2024-03-24T12:22:27.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[41500]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T12:22:27.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[41500]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T12:22:27.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[41500]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T12:22:27.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[41500]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T12:22:27.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[29919]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:27.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[29919]: Standby manager daemon x started 2024-03-24T12:22:27.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[29919]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T12:22:27.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[29919]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T12:22:27.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[29919]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T12:22:27.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:26 smithi029 ceph-mon[29919]: from='mgr.? 172.21.15.190:0/4028467577' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T12:22:28.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:27 smithi190 ceph-mon[33748]: mgrmap e14: y(active, since 71s), standbys: x 2024-03-24T12:22:28.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:27 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T12:22:28.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:27 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2440785974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b"}]: dispatch 2024-03-24T12:22:28.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:27 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2440785974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b"}]': finished 2024-03-24T12:22:28.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:27 smithi190 ceph-mon[33748]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T12:22:28.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:27 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:28.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:27 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2565361447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:22:28.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[29919]: mgrmap e14: y(active, since 71s), standbys: x 2024-03-24T12:22:28.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2440785974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b"}]: dispatch 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2440785974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b"}]': finished 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[29919]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2565361447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[41500]: mgrmap e14: y(active, since 71s), standbys: x 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2440785974' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b"}]: dispatch 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2440785974' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b"}]': finished 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[41500]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T12:22:28.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:28.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:27 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2565361447' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:22:29.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:28 smithi190 ceph-mon[33748]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:29.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:28 smithi029 ceph-mon[29919]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:29.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:28 smithi029 ceph-mon[41500]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:31.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:31 smithi029 ceph-mon[29919]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:31.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:31 smithi029 ceph-mon[41500]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:31.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:31 smithi190 ceph-mon[33748]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:33.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:33 smithi029 ceph-mon[29919]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:33.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:33 smithi029 ceph-mon[41500]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:33.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:33 smithi190 ceph-mon[33748]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:34.256 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T12:22:34.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:34.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T12:22:34.257 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:34.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T12:22:34.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:35.210 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:35 smithi029 ceph-mon[29919]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:35.210 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:35 smithi029 ceph-mon[29919]: Deploying daemon osd.0 on smithi029 2024-03-24T12:22:35.210 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:35 smithi029 ceph-mon[41500]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:35.211 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:35 smithi029 ceph-mon[41500]: Deploying daemon osd.0 on smithi029 2024-03-24T12:22:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:35 smithi190 ceph-mon[33748]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:35 smithi190 ceph-mon[33748]: Deploying daemon osd.0 on smithi029 2024-03-24T12:22:37.342 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:37 smithi029 ceph-mon[29919]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:37.342 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:37 smithi029 ceph-mon[41500]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:37.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:37 smithi190 ceph-mon[33748]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:39.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:39 smithi190 ceph-mon[33748]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:39 smithi029 ceph-mon[29919]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:39.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:39 smithi029 ceph-mon[41500]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:40.343 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:40 smithi029 ceph-mon[29919]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:40.344 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:40 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:40.344 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:40 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:40.344 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:40 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:40.344 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:40 smithi029 ceph-mon[41500]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:40.344 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:40 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:40.344 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:40 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:40.344 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:40 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:40.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:40 smithi190 ceph-mon[33748]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:40.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:40 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:40.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:40 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:40.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:40 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:42.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:42 smithi190 ceph-mon[33748]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:42.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:42 smithi029 ceph-mon[29919]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:42.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:42 smithi029 ceph-mon[41500]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:43.384 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 0 on host 'smithi029' 2024-03-24T12:22:43.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:43 smithi190 ceph-mon[33748]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T12:22:43.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:43 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:43.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:43 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:43.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:43 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:43.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:43 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:43.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:43 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:43.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[41500]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T12:22:43.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:43.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:43.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:43.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:43.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:43.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[29919]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T12:22:43.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:43.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:43.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:43.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:43.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:43 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:44.134 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T12:22:44.137 INFO:tasks.cephadm:Deploying osd.1 on smithi029 with /dev/vg_nvme/lv_3... 2024-03-24T12:22:44.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T12:22:44.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[41500]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:44.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[41500]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T12:22:44.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[41500]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[41500]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[29919]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[29919]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[29919]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[29919]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:44.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:44.422 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:44.422 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:44.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:44 smithi190 ceph-mon[33748]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:44.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:44 smithi190 ceph-mon[33748]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T12:22:44.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:44 smithi190 ceph-mon[33748]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T12:22:44.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:44 smithi190 ceph-mon[33748]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:22:44.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:44 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:44.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:44 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:44.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:44 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:44.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:44 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:22:44.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[45963]: 2024-03-24T12:22:44.578+0000 7f5d563cc700 -1 osd.0 0 waiting for initial osdmap 2024-03-24T12:22:44.707 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:22:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[45963]: 2024-03-24T12:22:44.580+0000 7f5d4e1b7700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T12:22:44.925 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:22:45.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[29919]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:22:45.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[29919]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T12:22:45.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:45.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:45.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[29919]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' 2024-03-24T12:22:45.612 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:45.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[41500]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:22:45.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[41500]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T12:22:45.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:45.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:45.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[41500]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' 2024-03-24T12:22:45.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:45 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:45.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:45 smithi190 ceph-mon[33748]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:22:45.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:45 smithi190 ceph-mon[33748]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T12:22:45.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:45 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:45.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:45 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:45.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:45 smithi190 ceph-mon[33748]: from='osd.0 v2:172.21.15.29:6801/2514818464' entity='osd.0' 2024-03-24T12:22:45.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:45 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:46.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:22:46.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:22:46.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[29919]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:46.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[29919]: osd.0 v2:172.21.15.29:6801/2514818464 boot 2024-03-24T12:22:46.466 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[29919]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T12:22:46.467 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:46.467 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:22:46.467 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:22:46.467 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[41500]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:46.467 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[41500]: osd.0 v2:172.21.15.29:6801/2514818464 boot 2024-03-24T12:22:46.467 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[41500]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T12:22:46.467 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:46.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:46 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:22:46.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:46 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:22:46.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:46 smithi190 ceph-mon[33748]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T12:22:46.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:46 smithi190 ceph-mon[33748]: osd.0 v2:172.21.15.29:6801/2514818464 boot 2024-03-24T12:22:46.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:46 smithi190 ceph-mon[33748]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T12:22:46.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:22:47.876 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T12:22:47.876 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T12:22:47.876 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-24T12:22:47.876 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-24T12:22:47.876 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00841396 s, 1.2 GB/s 2024-03-24T12:22:47.876 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-24T12:22:48.515 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:48 smithi029 ceph-mon[29919]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:48.515 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:48 smithi029 ceph-mon[29919]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T12:22:48.515 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:48 smithi029 ceph-mon[41500]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:48.515 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:48 smithi029 ceph-mon[41500]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T12:22:48.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_3 2024-03-24T12:22:48.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:48 smithi190 ceph-mon[33748]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:48.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:48 smithi190 ceph-mon[33748]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T12:22:49.024 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:22:50.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[29919]: Detected new or changed devices on smithi029 2024-03-24T12:22:50.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:22:50.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi029 to 16069M 2024-03-24T12:22:50.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:50.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:50.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[41500]: Detected new or changed devices on smithi029 2024-03-24T12:22:50.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:22:50.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi029 to 16069M 2024-03-24T12:22:50.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:50.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:50.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:50 smithi190 ceph-mon[33748]: Detected new or changed devices on smithi029 2024-03-24T12:22:50.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:22:50.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:50 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi029 to 16069M 2024-03-24T12:22:50.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:50.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:50.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:22:50.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:22:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:51 smithi029 ceph-mon[29919]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:51 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:22:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:51 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:22:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:51 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:51.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:51 smithi029 ceph-mon[41500]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:51.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:51 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:22:51.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:51 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:22:51.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:51 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:51 smithi190 ceph-mon[33748]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:51 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:22:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:51 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:22:51.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:51 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:22:52.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:52 smithi029 ceph-mon[29919]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:22:52.321 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:52 smithi029 ceph-mon[41500]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:22:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:52 smithi190 ceph-mon[33748]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:22:53.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:53 smithi029 ceph-mon[41500]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:53.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:53 smithi029 ceph-mon[29919]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:53.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:53 smithi190 ceph-mon[33748]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:54.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:54 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3136993100' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc"}]: dispatch 2024-03-24T12:22:54.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:54 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3136993100' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc"}]': finished 2024-03-24T12:22:54.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:54 smithi190 ceph-mon[33748]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T12:22:54.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:54 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:22:54.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:54 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3136993100' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc"}]: dispatch 2024-03-24T12:22:54.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:54 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3136993100' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc"}]': finished 2024-03-24T12:22:54.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:54 smithi029 ceph-mon[41500]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T12:22:54.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:54 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:22:54.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:54 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3136993100' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc"}]: dispatch 2024-03-24T12:22:54.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:54 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3136993100' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc"}]': finished 2024-03-24T12:22:54.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:54 smithi029 ceph-mon[29919]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T12:22:54.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:54 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:22:55.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:55 smithi190 ceph-mon[33748]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:55.287 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:55 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/41873354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:22:55.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:55 smithi029 ceph-mon[29919]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:55.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:55 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/41873354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:22:55.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:55 smithi029 ceph-mon[41500]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:55.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:55 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/41873354' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:22:57.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:57 smithi029 ceph-mon[29919]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:57.285 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:57 smithi029 ceph-mon[41500]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:57.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:57 smithi190 ceph-mon[33748]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:59.232 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:22:59 smithi029 ceph-mon[29919]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:59.232 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:22:59 smithi029 ceph-mon[41500]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:22:59.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:22:59 smithi190 ceph-mon[33748]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:01.287 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:01 smithi029 ceph-mon[29919]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:01.287 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:01 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T12:23:01.287 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:01 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:01.287 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:01 smithi029 ceph-mon[41500]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:01.287 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T12:23:01.287 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:01 smithi190 ceph-mon[33748]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:01 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T12:23:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:01 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:02.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:02 smithi190 ceph-mon[33748]: Deploying daemon osd.1 on smithi029 2024-03-24T12:23:02.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:02 smithi029 ceph-mon[29919]: Deploying daemon osd.1 on smithi029 2024-03-24T12:23:02.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:02 smithi029 ceph-mon[41500]: Deploying daemon osd.1 on smithi029 2024-03-24T12:23:03.334 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:03 smithi029 ceph-mon[29919]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:03.334 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:03 smithi029 ceph-mon[41500]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:03.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:03 smithi190 ceph-mon[33748]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:05.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:05 smithi190 ceph-mon[33748]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:05.386 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:05 smithi029 ceph-mon[29919]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:05.386 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:05 smithi029 ceph-mon[41500]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:07.201 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:07 smithi029 ceph-mon[41500]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:07.201 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:07 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:07.201 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:07 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:07.201 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:07 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:07.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:07 smithi029 ceph-mon[29919]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:07.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:07 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:07.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:07 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:07.202 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:07 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:07.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:07 smithi190 ceph-mon[33748]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:07.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:07 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:07.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:07 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:07.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:07 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:09.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:09 smithi029 ceph-mon[29919]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:09.269 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:09 smithi029 ceph-mon[41500]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:09.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:09 smithi190 ceph-mon[33748]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:10.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:10 smithi190 ceph-mon[33748]: from='osd.1 v2:172.21.15.29:6805/2923396572' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T12:23:10.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:10 smithi190 ceph-mon[33748]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T12:23:10.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:10.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:10.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:10.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:10.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:10.394 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[41500]: from='osd.1 v2:172.21.15.29:6805/2923396572' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T12:23:10.394 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[41500]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T12:23:10.394 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:10.394 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:10.395 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:10.395 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:10.395 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:10.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[29919]: from='osd.1 v2:172.21.15.29:6805/2923396572' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T12:23:10.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[29919]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T12:23:10.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:10.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:10.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:10.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:10.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:10.435 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 1 on host 'smithi029' 2024-03-24T12:23:11.082 DEBUG:teuthology.orchestra.run.smithi029:osd.1> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T12:23:11.084 INFO:tasks.cephadm:Deploying osd.2 on smithi029 with /dev/vg_nvme/lv_2... 2024-03-24T12:23:11.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T12:23:11.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:11 smithi190 ceph-mon[33748]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:11.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:11 smithi190 ceph-mon[33748]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T12:23:11.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:11 smithi190 ceph-mon[33748]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T12:23:11.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:11.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:11 smithi190 ceph-mon[33748]: from='osd.1 v2:172.21.15.29:6805/2923396572' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:23:11.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:11 smithi190 ceph-mon[33748]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:23:11.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:11.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:11.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:11.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[29919]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:11.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[29919]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T12:23:11.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[29919]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T12:23:11.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:11.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[29919]: from='osd.1 v2:172.21.15.29:6805/2923396572' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:23:11.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[29919]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:23:11.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:11.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:11.405 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:11.405 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[41500]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:11.405 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[41500]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T12:23:11.405 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[41500]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T12:23:11.406 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:11.406 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[41500]: from='osd.1 v2:172.21.15.29:6805/2923396572' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:23:11.406 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[41500]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:23:11.406 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:11.406 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:11.406 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:23:12.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:12 smithi029 ceph-mon[29919]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:23:12.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:12 smithi029 ceph-mon[29919]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T12:23:12.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:12 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:12.235 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:12 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:12.235 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:12 smithi029 ceph-mon[41500]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:23:12.235 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:12 smithi029 ceph-mon[41500]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T12:23:12.235 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:12 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:12.235 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:12 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:12 smithi190 ceph-mon[33748]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:23:12.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:12 smithi190 ceph-mon[33748]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T12:23:12.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:12 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:12.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:12 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:13.378 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:23:13.378 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:23:13.378 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[29919]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:13.378 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:13.378 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[29919]: osd.1 v2:172.21.15.29:6805/2923396572 boot 2024-03-24T12:23:13.378 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[29919]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T12:23:13.378 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:13.378 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:23:13.378 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:23:13.379 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[41500]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:13.379 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:13.379 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[41500]: osd.1 v2:172.21.15.29:6805/2923396572 boot 2024-03-24T12:23:13.379 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[41500]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T12:23:13.379 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:13 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:13.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:13 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:23:13.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:13 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:23:13.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:13 smithi190 ceph-mon[33748]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-24T12:23:13.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:13 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:13.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:13 smithi190 ceph-mon[33748]: osd.1 v2:172.21.15.29:6805/2923396572 boot 2024-03-24T12:23:13.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:13 smithi190 ceph-mon[33748]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T12:23:13.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:13 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:23:14.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:14 smithi029 ceph-mon[29919]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T12:23:14.330 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:14 smithi029 ceph-mon[41500]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T12:23:14.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:14 smithi190 ceph-mon[33748]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T12:23:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T12:23:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T12:23:14.824 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-24T12:23:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-24T12:23:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121314 s, 864 MB/s 2024-03-24T12:23:14.824 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-24T12:23:15.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:15 smithi029 ceph-mon[29919]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:15.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:15 smithi029 ceph-mon[41500]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:15.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:15 smithi190 ceph-mon[33748]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:15.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_2 2024-03-24T12:23:16.044 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:23:17.188 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:17.188 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: Detected new or changed devices on smithi029 2024-03-24T12:23:17.188 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.188 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.188 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:17.188 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:17.188 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi029 to 8034M 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: Detected new or changed devices on smithi029 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi029 to 8034M 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.189 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:17.190 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:17.190 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:17 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:17.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: Detected new or changed devices on smithi029 2024-03-24T12:23:17.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:17.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:17.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi029 to 8034M 2024-03-24T12:23:17.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:17.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:17.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:17.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:17 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:18.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:18 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:23:18.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:18 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:23:18.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:18 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:18.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:18 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:23:18.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:18 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:23:18.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:18 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:18.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:18 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:23:18.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:18 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:23:18.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:18 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:19.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:19 smithi029 ceph-mon[29919]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:19.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:19 smithi029 ceph-mon[29919]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:23:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:19 smithi029 ceph-mon[41500]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:19 smithi029 ceph-mon[41500]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:23:19.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:19 smithi190 ceph-mon[33748]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:19.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:19 smithi190 ceph-mon[33748]: from='client.24176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:23:21.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:21 smithi190 ceph-mon[33748]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:21.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:21 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/956458495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f83505-25ee-4217-b60a-081c36092e92"}]: dispatch 2024-03-24T12:23:21.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:21 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f83505-25ee-4217-b60a-081c36092e92"}]: dispatch 2024-03-24T12:23:21.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:21 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3f83505-25ee-4217-b60a-081c36092e92"}]': finished 2024-03-24T12:23:21.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:21 smithi190 ceph-mon[33748]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T12:23:21.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[29919]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/956458495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f83505-25ee-4217-b60a-081c36092e92"}]: dispatch 2024-03-24T12:23:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f83505-25ee-4217-b60a-081c36092e92"}]: dispatch 2024-03-24T12:23:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3f83505-25ee-4217-b60a-081c36092e92"}]': finished 2024-03-24T12:23:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[29919]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T12:23:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:21.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[41500]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:21.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/956458495' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f83505-25ee-4217-b60a-081c36092e92"}]: dispatch 2024-03-24T12:23:21.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c3f83505-25ee-4217-b60a-081c36092e92"}]: dispatch 2024-03-24T12:23:21.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c3f83505-25ee-4217-b60a-081c36092e92"}]': finished 2024-03-24T12:23:21.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[41500]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T12:23:21.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:22.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:22 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1198049807' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:23:22.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:22 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1198049807' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:23:22.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:22 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1198049807' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:23:23.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:23 smithi190 ceph-mon[33748]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:23 smithi029 ceph-mon[41500]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:23.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:23 smithi029 ceph-mon[29919]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:25.228 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:25 smithi029 ceph-mon[29919]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:25.228 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:25 smithi029 ceph-mon[41500]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:25.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:25 smithi190 ceph-mon[33748]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:27.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:27 smithi190 ceph-mon[33748]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:27.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:27 smithi029 ceph-mon[29919]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:27.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:27 smithi029 ceph-mon[41500]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:28.314 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:28 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T12:23:28.314 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:28 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:28.314 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:28 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T12:23:28.314 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:28 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:28.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:28 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T12:23:28.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:28 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:29.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:29 smithi029 ceph-mon[29919]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:29.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:29 smithi029 ceph-mon[29919]: Deploying daemon osd.2 on smithi029 2024-03-24T12:23:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:29 smithi029 ceph-mon[41500]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:29 smithi029 ceph-mon[41500]: Deploying daemon osd.2 on smithi029 2024-03-24T12:23:29.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:29 smithi190 ceph-mon[33748]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:29.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:29 smithi190 ceph-mon[33748]: Deploying daemon osd.2 on smithi029 2024-03-24T12:23:31.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:31 smithi029 ceph-mon[29919]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:31.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:31 smithi029 ceph-mon[41500]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:31.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:31 smithi190 ceph-mon[33748]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:33.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:33 smithi190 ceph-mon[33748]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:33.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:33 smithi029 ceph-mon[29919]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:33.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:33 smithi029 ceph-mon[41500]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:34.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:34 smithi190 ceph-mon[33748]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:34.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:34.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:34.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:34.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:34 smithi029 ceph-mon[29919]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:34.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:34.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:34.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:34.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:34 smithi029 ceph-mon[41500]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:34.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:34.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:34.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:37.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:36 smithi029 ceph-mon[29919]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:37.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:36 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:37.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:36 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:37.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:36 smithi029 ceph-mon[29919]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T12:23:37.115 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:36 smithi029 ceph-mon[41500]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:37.115 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:36 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:37.115 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:36 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:37.115 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:36 smithi029 ceph-mon[41500]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T12:23:37.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:36 smithi190 ceph-mon[33748]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:37.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:36 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:37.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:36 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:37.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:36 smithi190 ceph-mon[33748]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T12:23:37.172 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 2 on host 'smithi029' 2024-03-24T12:23:37.974 DEBUG:teuthology.orchestra.run.smithi029:osd.2> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T12:23:37.977 INFO:tasks.cephadm:Deploying osd.3 on smithi029 with /dev/vg_nvme/lv_1... 2024-03-24T12:23:37.977 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T12:23:38.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:38.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:38.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[29919]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T12:23:38.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[29919]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T12:23:38.280 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[29919]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:23:38.281 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:38.281 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:38.281 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:38.281 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[41500]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T12:23:38.281 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[41500]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T12:23:38.281 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[41500]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:23:38.281 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:38.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:38 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:38.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:38 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:38.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:38 smithi190 ceph-mon[33748]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T12:23:38.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:38 smithi190 ceph-mon[33748]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T12:23:38.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:38 smithi190 ceph-mon[33748]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:23:38.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:38 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:38.783 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:23:38.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2[58799]: 2024-03-24T12:23:38.655+0000 7f2ff6f99700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T12:23:38.894 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:23:38 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2[58799]: 2024-03-24T12:23:38.658+0000 7f2fee583700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T12:23:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[29919]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[29919]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:23:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[29919]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T12:23:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[29919]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' 2024-03-24T12:23:39.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[41500]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:39.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[41500]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:23:39.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[41500]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T12:23:39.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:39.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:39.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:39 smithi029 ceph-mon[41500]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' 2024-03-24T12:23:39.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:39 smithi190 ceph-mon[33748]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:39.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:39 smithi190 ceph-mon[33748]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:23:39.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:39 smithi190 ceph-mon[33748]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T12:23:39.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:39 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:39.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:39 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:39.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:39 smithi190 ceph-mon[33748]: from='osd.2 v2:172.21.15.29:6809/126032915' entity='osd.2' 2024-03-24T12:23:40.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:23:40.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:23:40.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[29919]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:40.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:40.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[29919]: osd.2 v2:172.21.15.29:6809/126032915 boot 2024-03-24T12:23:40.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[29919]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T12:23:40.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:40.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:23:40.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:23:40.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[41500]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:40.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:40.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[41500]: osd.2 v2:172.21.15.29:6809/126032915 boot 2024-03-24T12:23:40.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[41500]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T12:23:40.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:40 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:40.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:40 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:23:40.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:40 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:23:40.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:40 smithi190 ceph-mon[33748]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-24T12:23:40.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:40 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:40.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:40 smithi190 ceph-mon[33748]: osd.2 v2:172.21.15.29:6809/126032915 boot 2024-03-24T12:23:40.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:40 smithi190 ceph-mon[33748]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T12:23:40.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:40 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:23:42.282 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T12:23:42.282 INFO:teuthology.orchestra.run.smithi029.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T12:23:42.282 INFO:teuthology.orchestra.run.smithi029.stderr: stderr: 10+0 records in 2024-03-24T12:23:42.282 INFO:teuthology.orchestra.run.smithi029.stderr:10+0 records out 2024-03-24T12:23:42.282 INFO:teuthology.orchestra.run.smithi029.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00838732 s, 1.3 GB/s 2024-03-24T12:23:42.282 INFO:teuthology.orchestra.run.smithi029.stderr:--> Zapping successful for: 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: Detected new or changed devices on smithi029 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi029 to 5356M 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:42.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: Detected new or changed devices on smithi029 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi029 to 5356M 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:42.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T12:23:42.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: pgmap v82: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:42.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: Detected new or changed devices on smithi029 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi029 to 5356M 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:42.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:23:42.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:42.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:23:42.637 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:43.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi029:vg_nvme/lv_1 2024-03-24T12:23:43.287 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:23:43.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:43 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T12:23:43.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:43 smithi029 ceph-mon[29919]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T12:23:43.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:43 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:23:43.409 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:43 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T12:23:43.410 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:43 smithi029 ceph-mon[41500]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T12:23:43.410 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:43 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:23:43.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:43 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T12:23:43.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:43 smithi190 ceph-mon[33748]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T12:23:43.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:43 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:23:43.669 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:23:43 smithi029 sudo[63212]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T12:23:43.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:23:43 smithi029 sudo[63212]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:43.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:23:43 smithi029 sudo[63212]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:43.670 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:23:43 smithi029 sudo[63212]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:44.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:23:43 smithi029 sudo[63254]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T12:23:44.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:23:43 smithi029 sudo[63254]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:44.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:23:43 smithi029 sudo[63254]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:44.046 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:23:43 smithi029 sudo[63254]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:44.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:44 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:23:44.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:44 smithi029 ceph-mon[29919]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T12:23:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:44 smithi029 ceph-mon[29919]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:44 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:44.317 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:44 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:23:44.317 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:44 smithi029 ceph-mon[41500]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T12:23:44.317 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:44 smithi029 ceph-mon[41500]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:44.317 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:44 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:44.317 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63263]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T12:23:44.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63263]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:44.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63263]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:44.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63263]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:44.318 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63267]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T12:23:44.579 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63267]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:44.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63267]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:44.580 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63267]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:44.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:44 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:23:44.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:44 smithi190 ceph-mon[33748]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T12:23:44.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:44 smithi190 ceph-mon[33748]: pgmap v85: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:44.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:44 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:44.852 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63290]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T12:23:44.853 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63290]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:44.853 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63290]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:44.853 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63290]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:45.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63314]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T12:23:45.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63314]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:45.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63314]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:45.112 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:23:44 smithi029 sudo[63314]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:45.379 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:45 smithi029 sudo[63344]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T12:23:45.379 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:45 smithi029 sudo[63344]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:45.379 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:45 smithi029 sudo[63344]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:45.379 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:45 smithi029 sudo[63344]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:45.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:45 smithi190 ceph-mon[33748]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T12:23:45.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:45 smithi190 ceph-mon[33748]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T12:23:45.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:45 smithi190 sudo[36587]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T12:23:45.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:45 smithi190 sudo[36587]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:45.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:45 smithi190 sudo[36587]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:45.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:45 smithi029 ceph-mon[41500]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T12:23:45.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:45 smithi029 ceph-mon[41500]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T12:23:45.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:45 smithi029 sudo[63348]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T12:23:45.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:45 smithi029 sudo[63348]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:45.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:45 smithi029 sudo[63348]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:45.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:45 smithi029 sudo[63348]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:45.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:45 smithi029 ceph-mon[29919]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T12:23:45.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:45 smithi029 ceph-mon[29919]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-24T12:23:46.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:45 smithi190 sudo[36587]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:46.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:45 smithi190 sudo[36590]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T12:23:46.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:45 smithi190 sudo[36590]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:46.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:45 smithi190 sudo[36590]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:46.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:45 smithi190 sudo[36590]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:46.288 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 sudo[63492]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T12:23:46.288 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 sudo[63492]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:46.288 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 sudo[63492]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:46.288 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 sudo[63492]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T12:23:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:23:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T12:23:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:23:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:23:46.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:23:46.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:23:46.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 sudo[63514]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T12:23:46.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 sudo[63514]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T12:23:46.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 sudo[63514]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T12:23:46.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 sudo[63514]: pam_unix(sudo:session): session closed for user root 2024-03-24T12:23:46.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T12:23:46.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:46.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:23:46.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T12:23:46.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:23:46.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:23:46.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:23:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T12:23:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: pgmap v87: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi029:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:23:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T12:23:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:23:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:23:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:23:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:23:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T12:23:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:23:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:23:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:23:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:23:46.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:46.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T12:23:46.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:23:46.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:23:46.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:23:46.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:23:47.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:47 smithi029 ceph-mon[29919]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T12:23:47.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:47 smithi029 ceph-mon[29919]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T12:23:47.505 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:47 smithi029 ceph-mon[41500]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T12:23:47.505 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:47 smithi029 ceph-mon[41500]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T12:23:47.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:47 smithi190 ceph-mon[33748]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T12:23:47.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:47 smithi190 ceph-mon[33748]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T12:23:48.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:48 smithi190 ceph-mon[33748]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:48.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:48 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1682365963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96"}]: dispatch 2024-03-24T12:23:48.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:48 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96"}]: dispatch 2024-03-24T12:23:48.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:48 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96"}]': finished 2024-03-24T12:23:48.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:48 smithi190 ceph-mon[33748]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T12:23:48.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:48 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:23:48.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[29919]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1682365963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96"}]: dispatch 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96"}]: dispatch 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96"}]': finished 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[29919]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[41500]: pgmap v88: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1682365963' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96"}]: dispatch 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96"}]: dispatch 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96"}]': finished 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[41500]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T12:23:48.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:48 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:23:49.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:49 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/327685635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:23:49.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:49 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/327685635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:23:49.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:49 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/327685635' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:23:50.818 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:50 smithi029 ceph-mon[29919]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:50.818 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:50 smithi029 ceph-mon[41500]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:50.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:50 smithi190 ceph-mon[33748]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:52.801 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:52 smithi029 ceph-mon[41500]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:52.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:52 smithi029 ceph-mon[29919]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:52.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:52 smithi190 ceph-mon[33748]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:54.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:54 smithi029 ceph-mon[29919]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:54.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:54 smithi029 ceph-mon[41500]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:54.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:54 smithi190 ceph-mon[33748]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:55.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:55 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T12:23:55.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:55 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:55.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:55 smithi029 ceph-mon[29919]: Deploying daemon osd.3 on smithi029 2024-03-24T12:23:55.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:55 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T12:23:55.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:55 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:55.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:55 smithi029 ceph-mon[41500]: Deploying daemon osd.3 on smithi029 2024-03-24T12:23:56.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:55 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T12:23:56.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:55 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:23:56.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:55 smithi190 ceph-mon[33748]: Deploying daemon osd.3 on smithi029 2024-03-24T12:23:57.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:56 smithi029 ceph-mon[29919]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:57.074 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:56 smithi029 ceph-mon[41500]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:57.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:56 smithi190 ceph-mon[33748]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:58.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:23:58 smithi029 ceph-mon[29919]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:58.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:23:58 smithi029 ceph-mon[41500]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:23:59.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:23:58 smithi190 ceph-mon[33748]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:01.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:00 smithi190 ceph-mon[33748]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:00 smithi029 ceph-mon[29919]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:01.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:00 smithi029 ceph-mon[41500]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:01 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:01 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:01 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:02.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:02.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:02.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:01 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:02.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:01 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:02.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:01 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:02.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:01 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:03.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:02 smithi029 ceph-mon[41500]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:03.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:02 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:24:03.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:02 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:24:03.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:02 smithi029 ceph-mon[41500]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T12:24:03.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:02 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:03.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:02 smithi029 ceph-mon[29919]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:03.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:02 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:24:03.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:02 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:24:03.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:02 smithi029 ceph-mon[29919]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T12:24:03.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:02 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:03.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:02 smithi190 ceph-mon[33748]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:03.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:02 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:24:03.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:02 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:24:03.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:02 smithi190 ceph-mon[33748]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T12:24:03.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:02 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:04.845 INFO:teuthology.orchestra.run.smithi029.stdout:Created osd(s) 3 on host 'smithi029' 2024-03-24T12:24:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: from='osd.3 v2:172.21.15.29:6813/1080171824' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T12:24:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T12:24:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: from='osd.3 v2:172.21.15.29:6813/1080171824' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:05.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:05.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: from='osd.3 v2:172.21.15.29:6813/1080171824' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T12:24:05.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T12:24:05.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:05.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:05.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:05.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:05.543 DEBUG:teuthology.orchestra.run.smithi029:osd.3> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T12:24:05.545 INFO:tasks.cephadm:Deploying osd.4 on smithi190 with /dev/vg_nvme/lv_4... 2024-03-24T12:24:05.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T12:24:05.788 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:24:06.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-mon[41500]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T12:24:06.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-mon[41500]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T12:24:06.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:06.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-mon[41500]: from='osd.3 v2:172.21.15.29:6813/1080171824' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:24:06.065 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-mon[41500]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:24:06.066 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3[65432]: 2024-03-24T12:24:05.958+0000 7fc8ffbfe700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T12:24:06.066 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3[65432]: 2024-03-24T12:24:05.961+0000 7fc8fb9f1700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T12:24:06.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-mon[29919]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T12:24:06.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-mon[29919]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T12:24:06.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:06.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-mon[29919]: from='osd.3 v2:172.21.15.29:6813/1080171824' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:24:06.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:05 smithi029 ceph-mon[29919]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:24:06.162 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:05 smithi190 ceph-mon[33748]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T12:24:06.162 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:05 smithi190 ceph-mon[33748]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T12:24:06.162 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:05 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:06.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:05 smithi190 ceph-mon[33748]: from='osd.3 v2:172.21.15.29:6813/1080171824' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:24:06.163 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:05 smithi190 ceph-mon[33748]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:24:07.103 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:06 smithi029 ceph-mon[41500]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:07.103 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:06 smithi029 ceph-mon[41500]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:24:07.103 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:06 smithi029 ceph-mon[41500]: osdmap e26: 4 total, 3 up, 4 in 2024-03-24T12:24:07.103 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:07.103 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:06 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:07.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:06 smithi029 ceph-mon[29919]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:07.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:06 smithi029 ceph-mon[29919]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:24:07.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:06 smithi029 ceph-mon[29919]: osdmap e26: 4 total, 3 up, 4 in 2024-03-24T12:24:07.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:07.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:06 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:07.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:06 smithi190 ceph-mon[33748]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:07.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:06 smithi190 ceph-mon[33748]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]': finished 2024-03-24T12:24:07.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:06 smithi190 ceph-mon[33748]: osdmap e26: 4 total, 3 up, 4 in 2024-03-24T12:24:07.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:07.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:06 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:07.721 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T12:24:07.721 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T12:24:07.721 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-24T12:24:07.721 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-24T12:24:07.721 INFO:teuthology.orchestra.run.smithi190.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011402 s, 920 MB/s 2024-03-24T12:24:07.721 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-24T12:24:08.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:07 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:24:08.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:07 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:24:08.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:07 smithi190 ceph-mon[33748]: osd.3 v2:172.21.15.29:6813/1080171824 boot 2024-03-24T12:24:08.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:07 smithi190 ceph-mon[33748]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T12:24:08.056 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:07 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:08.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:07 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:24:08.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:07 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:24:08.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:07 smithi029 ceph-mon[29919]: osd.3 v2:172.21.15.29:6813/1080171824 boot 2024-03-24T12:24:08.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:07 smithi029 ceph-mon[29919]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T12:24:08.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:07 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:08.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:07 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:24:08.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:07 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:24:08.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:07 smithi029 ceph-mon[41500]: osd.3 v2:172.21.15.29:6813/1080171824 boot 2024-03-24T12:24:08.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:07 smithi029 ceph-mon[41500]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T12:24:08.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:07 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:24:08.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_4 2024-03-24T12:24:08.577 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:24:09.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:09.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T12:24:09.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:09.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:09.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:08 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-24T12:24:09.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T12:24:09.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:09.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:09.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:09.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:09.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:08 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:10.083 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:09 smithi190 ceph-mon[33748]: Detected new or changed devices on smithi029 2024-03-24T12:24:10.084 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:09 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi029 to 4017M 2024-03-24T12:24:10.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:09 smithi029 ceph-mon[29919]: Detected new or changed devices on smithi029 2024-03-24T12:24:10.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:09 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi029 to 4017M 2024-03-24T12:24:10.159 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:09 smithi029 ceph-mon[41500]: Detected new or changed devices on smithi029 2024-03-24T12:24:10.159 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:09 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi029 to 4017M 2024-03-24T12:24:11.227 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:10 smithi190 ceph-mon[33748]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:11.227 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:24:11.227 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:24:11.227 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:11.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:10 smithi029 ceph-mon[41500]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:11.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:24:11.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:24:11.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:11.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:10 smithi029 ceph-mon[29919]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:11.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:24:11.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:24:11.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:12.100 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:11 smithi190 ceph-mon[33748]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:24:12.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:11 smithi029 ceph-mon[29919]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:24:12.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:11 smithi029 ceph-mon[41500]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:24:13.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:12 smithi190 ceph-mon[33748]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:13.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:12 smithi029 ceph-mon[29919]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:13.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:12 smithi029 ceph-mon[41500]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:14.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:13 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/1216658373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e420304-2798-4d11-9688-d0d89482008c"}]: dispatch 2024-03-24T12:24:14.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:13 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e420304-2798-4d11-9688-d0d89482008c"}]: dispatch 2024-03-24T12:24:14.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:13 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e420304-2798-4d11-9688-d0d89482008c"}]': finished 2024-03-24T12:24:14.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:13 smithi190 ceph-mon[33748]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T12:24:14.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:13 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:14.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:13 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/2299967353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:24:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/1216658373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e420304-2798-4d11-9688-d0d89482008c"}]: dispatch 2024-03-24T12:24:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e420304-2798-4d11-9688-d0d89482008c"}]: dispatch 2024-03-24T12:24:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e420304-2798-4d11-9688-d0d89482008c"}]': finished 2024-03-24T12:24:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[29919]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T12:24:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/2299967353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:24:14.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/1216658373' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e420304-2798-4d11-9688-d0d89482008c"}]: dispatch 2024-03-24T12:24:14.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e420304-2798-4d11-9688-d0d89482008c"}]: dispatch 2024-03-24T12:24:14.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e420304-2798-4d11-9688-d0d89482008c"}]': finished 2024-03-24T12:24:14.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[41500]: osdmap e29: 5 total, 4 up, 5 in 2024-03-24T12:24:14.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:14.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:13 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/2299967353' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:24:15.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:14 smithi190 ceph-mon[33748]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:15.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:14 smithi029 ceph-mon[29919]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:15.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:14 smithi029 ceph-mon[41500]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:17.293 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:16 smithi190 ceph-mon[33748]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:17.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:16 smithi029 ceph-mon[29919]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:17.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:16 smithi029 ceph-mon[41500]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:19.264 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:18 smithi190 ceph-mon[33748]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:19.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:18 smithi029 ceph-mon[29919]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:19.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:18 smithi029 ceph-mon[41500]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:20.313 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:19 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T12:24:20.313 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:19 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:19 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T12:24:20.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:19 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:19 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T12:24:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:19 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:21.170 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:21 smithi190 ceph-mon[33748]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:21.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:21 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:24:21.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:21 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:24:21.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:21 smithi190 ceph-mon[33748]: Deploying daemon osd.4 on smithi190 2024-03-24T12:24:21.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:21 smithi190 ceph-mon[33748]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T12:24:21.171 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:20 smithi029 ceph-mon[29919]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[29919]: Deploying daemon osd.4 on smithi190 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[29919]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[41500]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[41500]: Deploying daemon osd.4 on smithi190 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[41500]: osdmap e30: 5 total, 4 up, 5 in 2024-03-24T12:24:21.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:23.211 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:23 smithi190 ceph-mon[33748]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:23.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:23 smithi029 ceph-mon[29919]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:23.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:23 smithi029 ceph-mon[41500]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:25.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:25 smithi190 ceph-mon[33748]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:25.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:25 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:25.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:25 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:25.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:25 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:25.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:25 smithi029 ceph-mon[29919]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:25.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:25 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:25.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:25 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:25.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:25 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:25.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:25 smithi029 ceph-mon[41500]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:25.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:25 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:25.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:25 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:25.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:25 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:27.179 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:27 smithi190 ceph-mon[33748]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:27 smithi029 ceph-mon[41500]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:27.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:27 smithi029 ceph-mon[29919]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:28.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:28 smithi190 ceph-mon[33748]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:28.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:28 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:28.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:28 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:28.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:28 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:28.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:28 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:28.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:28 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:28.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:28 smithi190 ceph-mon[33748]: from='osd.4 v2:172.21.15.190:6800/3690258869' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T12:24:28.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:28 smithi190 ceph-mon[33748]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T12:24:28.784 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 4 on host 'smithi190' 2024-03-24T12:24:28.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[41500]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:28.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:28.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:28.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:28.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:28.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:28.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[41500]: from='osd.4 v2:172.21.15.190:6800/3690258869' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T12:24:28.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[41500]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T12:24:28.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[29919]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:28.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:28.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:28.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:28.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:28.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:28.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[29919]: from='osd.4 v2:172.21.15.190:6800/3690258869' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T12:24:28.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:28 smithi029 ceph-mon[29919]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T12:24:29.524 DEBUG:teuthology.orchestra.run.smithi190:osd.4> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T12:24:29.527 INFO:tasks.cephadm:Deploying osd.5 on smithi190 with /dev/vg_nvme/lv_3... 2024-03-24T12:24:29.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T12:24:29.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:29 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:29.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:29 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:29.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:29 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:29.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:29 smithi190 ceph-mon[33748]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T12:24:29.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:29 smithi190 ceph-mon[33748]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T12:24:29.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:29 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:29.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:29 smithi190 ceph-mon[33748]: from='osd.4 v2:172.21.15.190:6800/3690258869' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:29.824 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:29 smithi190 ceph-mon[33748]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:30.093 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:24:30 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:24:30.023+0000 7f9f53c54700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T12:24:30.093 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:24:30 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:24:30.028+0000 7f9f4fa47700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T12:24:30.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:30.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:30.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:30.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[29919]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T12:24:30.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[29919]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T12:24:30.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:30.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[29919]: from='osd.4 v2:172.21.15.190:6800/3690258869' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:30.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[29919]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:30.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:30.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:30.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:30.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[41500]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T12:24:30.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[41500]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T12:24:30.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:30.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[41500]: from='osd.4 v2:172.21.15.190:6800/3690258869' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:30.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:29 smithi029 ceph-mon[41500]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:30.267 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:24:31.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:31 smithi190 ceph-mon[33748]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:31.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:31 smithi190 ceph-mon[33748]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:24:31.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:31 smithi190 ceph-mon[33748]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T12:24:31.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:31 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:31.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:31 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:31.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:31 smithi029 ceph-mon[29919]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:31.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:31 smithi029 ceph-mon[29919]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:24:31.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:31 smithi029 ceph-mon[29919]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T12:24:31.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:31 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:31.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:31 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:31.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:31 smithi029 ceph-mon[41500]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:31.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:31 smithi029 ceph-mon[41500]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:24:31.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:31 smithi029 ceph-mon[41500]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T12:24:31.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:31 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:31.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:31 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:32.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:32 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:24:32.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:32 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:24:32.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:32 smithi190 ceph-mon[33748]: osd.4 v2:172.21.15.190:6800/3690258869 boot 2024-03-24T12:24:32.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:32 smithi190 ceph-mon[33748]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T12:24:32.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:32 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:32 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:24:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:32 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:24:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:32 smithi029 ceph-mon[29919]: osd.4 v2:172.21.15.190:6800/3690258869 boot 2024-03-24T12:24:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:32 smithi029 ceph-mon[29919]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T12:24:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:32 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:32.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:32 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:24:32.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:32 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:24:32.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:32 smithi029 ceph-mon[41500]: osd.4 v2:172.21.15.190:6800/3690258869 boot 2024-03-24T12:24:32.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:32 smithi029 ceph-mon[41500]: osdmap e33: 5 total, 5 up, 5 in 2024-03-24T12:24:32.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:32 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:24:33.047 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:33 smithi190 ceph-mon[33748]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:33.047 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:33 smithi190 ceph-mon[33748]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T12:24:33.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:33 smithi029 ceph-mon[41500]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:33.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:33 smithi029 ceph-mon[41500]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T12:24:33.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:33 smithi029 ceph-mon[29919]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-24T12:24:33.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:33 smithi029 ceph-mon[29919]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T12:24:33.621 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T12:24:33.622 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T12:24:33.622 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-24T12:24:33.622 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-24T12:24:33.622 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115023 s, 912 MB/s 2024-03-24T12:24:33.622 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-24T12:24:34.183 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:34 smithi190 ceph-mon[33748]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T12:24:34.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:34 smithi190 ceph-mon[33748]: Detected new or changed devices on smithi190 2024-03-24T12:24:34.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:34.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:34 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi190 to 17088M 2024-03-24T12:24:34.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:34.184 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:34.185 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:34 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_3 2024-03-24T12:24:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[29919]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T12:24:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[29919]: Detected new or changed devices on smithi190 2024-03-24T12:24:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi190 to 17088M 2024-03-24T12:24:34.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:34.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:34.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.421 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:24:34.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[41500]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T12:24:34.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[41500]: Detected new or changed devices on smithi190 2024-03-24T12:24:34.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:24:34.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi190 to 17088M 2024-03-24T12:24:34.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:34.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:34.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:34.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:34 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:35.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:35 smithi190 ceph-mon[33748]: pgmap v124: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T12:24:35.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:35 smithi190 ceph-mon[33748]: osdmap e36: 5 total, 5 up, 5 in 2024-03-24T12:24:35.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:35 smithi029 ceph-mon[29919]: pgmap v124: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T12:24:35.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:35 smithi029 ceph-mon[29919]: osdmap e36: 5 total, 5 up, 5 in 2024-03-24T12:24:35.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:35 smithi029 ceph-mon[41500]: pgmap v124: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T12:24:35.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:35 smithi029 ceph-mon[41500]: osdmap e36: 5 total, 5 up, 5 in 2024-03-24T12:24:36.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:36 smithi190 ceph-mon[33748]: osdmap e37: 5 total, 5 up, 5 in 2024-03-24T12:24:36.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:36 smithi029 ceph-mon[29919]: osdmap e37: 5 total, 5 up, 5 in 2024-03-24T12:24:36.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:36 smithi029 ceph-mon[41500]: osdmap e37: 5 total, 5 up, 5 in 2024-03-24T12:24:37.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:37 smithi190 ceph-mon[33748]: pgmap v127: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T12:24:37.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:37 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:24:37.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:37 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:24:37.054 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:37 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:37.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:37 smithi029 ceph-mon[29919]: pgmap v127: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T12:24:37.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:37 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:24:37.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:37 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:24:37.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:37 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:37.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:37 smithi029 ceph-mon[41500]: pgmap v127: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T12:24:37.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:37 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:24:37.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:37 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:24:37.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:37 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:38.082 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:38 smithi190 ceph-mon[33748]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:24:38.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:38 smithi029 ceph-mon[29919]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:24:38.321 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:38 smithi029 ceph-mon[41500]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:24:39.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:39 smithi190 ceph-mon[33748]: pgmap v128: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-24T12:24:39.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:39 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/3189016460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1213431a-652c-4cbf-b32a-36335430c666"}]: dispatch 2024-03-24T12:24:39.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:39 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1213431a-652c-4cbf-b32a-36335430c666"}]: dispatch 2024-03-24T12:24:39.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:39 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1213431a-652c-4cbf-b32a-36335430c666"}]': finished 2024-03-24T12:24:39.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:39 smithi190 ceph-mon[33748]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T12:24:39.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:39 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:39.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[29919]: pgmap v128: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-24T12:24:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/3189016460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1213431a-652c-4cbf-b32a-36335430c666"}]: dispatch 2024-03-24T12:24:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1213431a-652c-4cbf-b32a-36335430c666"}]: dispatch 2024-03-24T12:24:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1213431a-652c-4cbf-b32a-36335430c666"}]': finished 2024-03-24T12:24:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[29919]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T12:24:39.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:39.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[41500]: pgmap v128: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 112 KiB/s, 0 objects/s recovering 2024-03-24T12:24:39.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/3189016460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1213431a-652c-4cbf-b32a-36335430c666"}]: dispatch 2024-03-24T12:24:39.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1213431a-652c-4cbf-b32a-36335430c666"}]: dispatch 2024-03-24T12:24:39.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1213431a-652c-4cbf-b32a-36335430c666"}]': finished 2024-03-24T12:24:39.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[41500]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T12:24:39.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:39 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:40.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:40 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/2040264274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:24:40.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:40 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/2040264274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:24:40.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:40 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/2040264274' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:24:41.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:41 smithi190 ceph-mon[33748]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:41.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:41 smithi029 ceph-mon[29919]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:41.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:41 smithi029 ceph-mon[41500]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:43.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:43 smithi190 ceph-mon[33748]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:43.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:43 smithi029 ceph-mon[29919]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:43.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:43 smithi029 ceph-mon[41500]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:45.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:45 smithi190 ceph-mon[33748]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:45.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:45 smithi029 ceph-mon[29919]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:45.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:45 smithi029 ceph-mon[41500]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T12:24:46.322 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:46.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T12:24:46.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:46.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T12:24:46.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:47.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:47 smithi190 ceph-mon[33748]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:47.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:47 smithi190 ceph-mon[33748]: Deploying daemon osd.5 on smithi190 2024-03-24T12:24:47.348 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:47 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:24:47.349 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:47 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:24:47.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:47 smithi029 ceph-mon[29919]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:47 smithi029 ceph-mon[29919]: Deploying daemon osd.5 on smithi190 2024-03-24T12:24:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:47 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:24:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:47 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:24:47.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:47 smithi029 ceph-mon[41500]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:47.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:47 smithi029 ceph-mon[41500]: Deploying daemon osd.5 on smithi190 2024-03-24T12:24:47.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:47 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:24:47.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:47 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:24:49.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:49 smithi190 ceph-mon[33748]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:49.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:49 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:24:49.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:49 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:24:49.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:49 smithi190 ceph-mon[33748]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T12:24:49.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:49 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:49.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:49 smithi029 ceph-mon[29919]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:49.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:49 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:24:49.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:49 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:24:49.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:49 smithi029 ceph-mon[29919]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T12:24:49.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:49 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:49.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:49 smithi029 ceph-mon[41500]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:49.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:49 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:24:49.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:49 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:24:49.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:49 smithi029 ceph-mon[41500]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T12:24:49.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:49 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:51.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:51 smithi190 ceph-mon[33748]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:51.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:51 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:51.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:51 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:51.328 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:51 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:51.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:51 smithi029 ceph-mon[29919]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:51.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:51 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:51.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:51 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:51.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:51 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:51.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:51 smithi029 ceph-mon[41500]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:51.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:51 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:51.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:51 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:51.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:51 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:53.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:53 smithi190 ceph-mon[33748]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:53.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:53 smithi029 ceph-mon[29919]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:53.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:53 smithi029 ceph-mon[41500]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:54.726 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 5 on host 'smithi190' 2024-03-24T12:24:55.037 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:55.037 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.037 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.037 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:55.037 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:55.037 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: from='osd.5 v2:172.21.15.190:6804/1327220488' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:24:55.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:24:55.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.038 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:54 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:55.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:55.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:55.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:55.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: from='osd.5 v2:172.21.15.190:6804/1327220488' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:24:55.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:24:55.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: from='osd.5 v2:172.21.15.190:6804/1327220488' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:24:55.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:54 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:24:55.371 DEBUG:teuthology.orchestra.run.smithi190:osd.5> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T12:24:55.404 INFO:tasks.cephadm:Deploying osd.6 on smithi190 with /dev/vg_nvme/lv_2... 2024-03-24T12:24:55.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T12:24:55.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:55 smithi190 ceph-mon[33748]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T12:24:55.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:55 smithi190 ceph-mon[33748]: osdmap e40: 6 total, 5 up, 6 in 2024-03-24T12:24:55.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:55 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:55.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:55 smithi190 ceph-mon[33748]: from='osd.5 v2:172.21.15.190:6804/1327220488' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:55.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:55 smithi190 ceph-mon[33748]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:55.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:24:55 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[45195]: 2024-03-24T12:24:55.849+0000 7f1cf9bb4700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T12:24:55.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:24:55 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[45195]: 2024-03-24T12:24:55.855+0000 7f1cf59a7700 -1 osd.5 41 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T12:24:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:55 smithi029 ceph-mon[29919]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T12:24:56.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:55 smithi029 ceph-mon[29919]: osdmap e40: 6 total, 5 up, 6 in 2024-03-24T12:24:56.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:55 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:56.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:55 smithi029 ceph-mon[29919]: from='osd.5 v2:172.21.15.190:6804/1327220488' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:56.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:55 smithi029 ceph-mon[29919]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:55 smithi029 ceph-mon[41500]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T12:24:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:55 smithi029 ceph-mon[41500]: osdmap e40: 6 total, 5 up, 6 in 2024-03-24T12:24:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:55 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:55 smithi029 ceph-mon[41500]: from='osd.5 v2:172.21.15.190:6804/1327220488' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:55 smithi029 ceph-mon[41500]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:24:56.191 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:24:57.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:56 smithi190 ceph-mon[33748]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:57.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:56 smithi190 ceph-mon[33748]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:24:57.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:56 smithi190 ceph-mon[33748]: osdmap e41: 6 total, 5 up, 6 in 2024-03-24T12:24:57.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:56 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:57.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:56 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:56 smithi029 ceph-mon[29919]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:57.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:56 smithi029 ceph-mon[29919]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:24:57.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:56 smithi029 ceph-mon[29919]: osdmap e41: 6 total, 5 up, 6 in 2024-03-24T12:24:57.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:56 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:57.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:56 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:57.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:56 smithi029 ceph-mon[41500]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-24T12:24:57.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:56 smithi029 ceph-mon[41500]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:24:57.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:56 smithi029 ceph-mon[41500]: osdmap e41: 6 total, 5 up, 6 in 2024-03-24T12:24:57.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:56 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:57.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:56 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:58.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:57 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:24:58.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:57 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:24:58.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:57 smithi190 ceph-mon[33748]: osd.5 v2:172.21.15.190:6804/1327220488 boot 2024-03-24T12:24:58.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:57 smithi190 ceph-mon[33748]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T12:24:58.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:57 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:58.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:57 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:24:58.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:57 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:24:58.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:57 smithi029 ceph-mon[29919]: osd.5 v2:172.21.15.190:6804/1327220488 boot 2024-03-24T12:24:58.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:57 smithi029 ceph-mon[29919]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T12:24:58.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:57 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:58.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:57 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:24:58.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:57 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:24:58.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:57 smithi029 ceph-mon[41500]: osd.5 v2:172.21.15.190:6804/1327220488 boot 2024-03-24T12:24:58.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:57 smithi029 ceph-mon[41500]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T12:24:58.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:57 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:24:58.957 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T12:24:58.957 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T12:24:58.957 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-24T12:24:58.958 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-24T12:24:58.958 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0121485 s, 863 MB/s 2024-03-24T12:24:58.958 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-24T12:24:59.014 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:58 smithi190 ceph-mon[33748]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:24:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:58 smithi190 ceph-mon[33748]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T12:24:59.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:24:58 smithi190 ceph-mon[33748]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T12:24:59.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:58 smithi029 ceph-mon[29919]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:24:59.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:58 smithi029 ceph-mon[29919]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T12:24:59.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:24:58 smithi029 ceph-mon[29919]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T12:24:59.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:58 smithi029 ceph-mon[41500]: pgmap v143: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:24:59.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:58 smithi029 ceph-mon[41500]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T12:24:59.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:24:58 smithi029 ceph-mon[41500]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T12:24:59.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_2 2024-03-24T12:25:00.020 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:25:01.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: pgmap v146: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:01.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T12:25:01.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: Detected new or changed devices on smithi190 2024-03-24T12:25:01.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:01.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:01.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi190 to 8544M 2024-03-24T12:25:01.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:01.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:01.137 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:00 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: pgmap v146: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: Detected new or changed devices on smithi190 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi190 to 8544M 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: pgmap v146: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: Detected new or changed devices on smithi190 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi190 to 8544M 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:00 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:02.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:01 smithi190 ceph-mon[33748]: osdmap e46: 6 total, 6 up, 6 in 2024-03-24T12:25:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:01 smithi029 ceph-mon[29919]: osdmap e46: 6 total, 6 up, 6 in 2024-03-24T12:25:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:01 smithi029 ceph-mon[41500]: osdmap e46: 6 total, 6 up, 6 in 2024-03-24T12:25:03.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:02 smithi190 ceph-mon[33748]: pgmap v149: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:03.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:02 smithi190 ceph-mon[33748]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:25:03.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:02 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:25:03.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:02 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:25:03.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:02 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:03.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:02 smithi029 ceph-mon[29919]: pgmap v149: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:03.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:02 smithi029 ceph-mon[29919]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:25:03.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:02 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:25:03.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:02 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:25:03.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:02 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:03.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:02 smithi029 ceph-mon[41500]: pgmap v149: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:03.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:02 smithi029 ceph-mon[41500]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:25:03.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:02 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:25:03.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:02 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:25:03.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:02 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:05.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:04 smithi190 ceph-mon[33748]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-24T12:25:05.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:04 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/16333679' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c"}]: dispatch 2024-03-24T12:25:05.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:04 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c"}]: dispatch 2024-03-24T12:25:05.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:04 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c"}]': finished 2024-03-24T12:25:05.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:04 smithi190 ceph-mon[33748]: osdmap e47: 7 total, 6 up, 7 in 2024-03-24T12:25:05.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:05.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[29919]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-24T12:25:05.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/16333679' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c"}]: dispatch 2024-03-24T12:25:05.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c"}]: dispatch 2024-03-24T12:25:05.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c"}]': finished 2024-03-24T12:25:05.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[29919]: osdmap e47: 7 total, 6 up, 7 in 2024-03-24T12:25:05.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:05.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[41500]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-03-24T12:25:05.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/16333679' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c"}]: dispatch 2024-03-24T12:25:05.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c"}]: dispatch 2024-03-24T12:25:05.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c"}]': finished 2024-03-24T12:25:05.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[41500]: osdmap e47: 7 total, 6 up, 7 in 2024-03-24T12:25:05.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:06.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:05 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/3936391027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:25:06.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:05 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/3936391027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:25:06.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:05 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/3936391027' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:25:07.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:06 smithi190 ceph-mon[33748]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T12:25:07.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:06 smithi029 ceph-mon[29919]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T12:25:07.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:06 smithi029 ceph-mon[41500]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T12:25:09.040 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:08 smithi190 ceph-mon[33748]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T12:25:09.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:08 smithi029 ceph-mon[29919]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T12:25:09.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:08 smithi029 ceph-mon[41500]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-24T12:25:11.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:10 smithi190 ceph-mon[33748]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-24T12:25:11.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:10 smithi029 ceph-mon[29919]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-24T12:25:11.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:10 smithi029 ceph-mon[41500]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-24T12:25:12.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T12:25:12.095 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:12.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T12:25:12.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:12.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T12:25:12.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:13.024 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:12 smithi190 ceph-mon[33748]: Deploying daemon osd.6 on smithi190 2024-03-24T12:25:13.025 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:12 smithi190 ceph-mon[33748]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T12:25:13.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:12 smithi029 ceph-mon[29919]: Deploying daemon osd.6 on smithi190 2024-03-24T12:25:13.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:12 smithi029 ceph-mon[29919]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T12:25:13.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:12 smithi029 ceph-mon[41500]: Deploying daemon osd.6 on smithi190 2024-03-24T12:25:13.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:12 smithi029 ceph-mon[41500]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T12:25:14.975 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:14 smithi190 ceph-mon[33748]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:15.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:14 smithi029 ceph-mon[29919]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:15.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:14 smithi029 ceph-mon[41500]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:16.085 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:15 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:25:16.086 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:15 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:25:16.086 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:15 smithi190 ceph-mon[33748]: osdmap e48: 7 total, 6 up, 7 in 2024-03-24T12:25:16.086 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:15 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:16.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:15 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:25:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:15 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:25:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:15 smithi029 ceph-mon[29919]: osdmap e48: 7 total, 6 up, 7 in 2024-03-24T12:25:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:16.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:15 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:25:16.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:15 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:25:16.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:15 smithi029 ceph-mon[41500]: osdmap e48: 7 total, 6 up, 7 in 2024-03-24T12:25:16.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:15 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:17.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:16 smithi190 ceph-mon[33748]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:17.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:17.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:17.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:17.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:16 smithi029 ceph-mon[29919]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:17.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:17.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:17.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:16 smithi029 ceph-mon[41500]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:19.128 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:18 smithi190 ceph-mon[33748]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:19.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:18 smithi029 ceph-mon[29919]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:18 smithi029 ceph-mon[41500]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:20.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:19 smithi190 ceph-mon[33748]: from='osd.6 v2:172.21.15.190:6808/1295064976' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T12:25:20.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:19 smithi190 ceph-mon[33748]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T12:25:20.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:19 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:20.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:19 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:20.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:19 smithi029 ceph-mon[29919]: from='osd.6 v2:172.21.15.190:6808/1295064976' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T12:25:20.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:19 smithi029 ceph-mon[29919]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T12:25:20.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:19 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:20.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:19 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:20.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:19 smithi029 ceph-mon[41500]: from='osd.6 v2:172.21.15.190:6808/1295064976' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T12:25:20.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:19 smithi029 ceph-mon[41500]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T12:25:20.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:19 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:20.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:19 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:20.254 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 6 on host 'smithi190' 2024-03-24T12:25:20.920 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:20 smithi190 ceph-mon[33748]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:20.920 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:20 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:25:20.920 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:20 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:25:20.920 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:20 smithi190 ceph-mon[33748]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T12:25:20.920 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:20 smithi190 ceph-mon[33748]: from='osd.6 v2:172.21.15.190:6808/1295064976' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:20.920 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:20 smithi190 ceph-mon[33748]: osdmap e49: 7 total, 6 up, 7 in 2024-03-24T12:25:20.920 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:20.920 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:20 smithi190 ceph-mon[33748]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:20.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:20.921 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:20.922 DEBUG:teuthology.orchestra.run.smithi190:osd.6> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T12:25:20.924 INFO:tasks.cephadm:Deploying osd.7 on smithi190 with /dev/vg_nvme/lv_1... 2024-03-24T12:25:20.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T12:25:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[41500]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:25:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[41500]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[41500]: from='osd.6 v2:172.21.15.190:6808/1295064976' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[41500]: osdmap e49: 7 total, 6 up, 7 in 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[41500]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[29919]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:25:21.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[29919]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T12:25:21.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[29919]: from='osd.6 v2:172.21.15.190:6808/1295064976' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:21.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[29919]: osdmap e49: 7 total, 6 up, 7 in 2024-03-24T12:25:21.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:21.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[29919]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:21.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:21.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:21.595 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:25:21.635 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:25:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:25:21.325+0000 7f2bdc6b4700 -1 osd.6 0 waiting for initial osdmap 2024-03-24T12:25:21.635 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:25:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:25:21.332+0000 7f2bd449f700 -1 osd.6 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T12:25:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:21 smithi190 ceph-mon[33748]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:25:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:21 smithi190 ceph-mon[33748]: osdmap e50: 7 total, 6 up, 7 in 2024-03-24T12:25:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:21 smithi190 ceph-mon[33748]: from='osd.6 ' entity='osd.6' 2024-03-24T12:25:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:21 smithi029 ceph-mon[29919]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:25:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:21 smithi029 ceph-mon[29919]: osdmap e50: 7 total, 6 up, 7 in 2024-03-24T12:25:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:22.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:21 smithi029 ceph-mon[29919]: from='osd.6 ' entity='osd.6' 2024-03-24T12:25:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:21 smithi029 ceph-mon[41500]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:25:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:21 smithi029 ceph-mon[41500]: osdmap e50: 7 total, 6 up, 7 in 2024-03-24T12:25:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:21 smithi029 ceph-mon[41500]: from='osd.6 ' entity='osd.6' 2024-03-24T12:25:23.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:22 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:25:23.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:22 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:25:23.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:22 smithi190 ceph-mon[33748]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:23.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:22 smithi190 ceph-mon[33748]: osd.6 v2:172.21.15.190:6808/1295064976 boot 2024-03-24T12:25:23.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:22 smithi190 ceph-mon[33748]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T12:25:23.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:22 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:23.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:22 smithi190 ceph-mon[33748]: osdmap e52: 7 total, 7 up, 7 in 2024-03-24T12:25:23.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:25:23.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:25:23.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[29919]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:23.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[29919]: osd.6 v2:172.21.15.190:6808/1295064976 boot 2024-03-24T12:25:23.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[29919]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T12:25:23.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:23.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[29919]: osdmap e52: 7 total, 7 up, 7 in 2024-03-24T12:25:23.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:25:23.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:25:23.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[41500]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-24T12:25:23.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[41500]: osd.6 v2:172.21.15.190:6808/1295064976 boot 2024-03-24T12:25:23.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[41500]: osdmap e51: 7 total, 7 up, 7 in 2024-03-24T12:25:23.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:25:23.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:22 smithi029 ceph-mon[41500]: osdmap e52: 7 total, 7 up, 7 in 2024-03-24T12:25:24.714 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T12:25:24.715 INFO:teuthology.orchestra.run.smithi190.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T12:25:24.715 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10+0 records in 2024-03-24T12:25:24.715 INFO:teuthology.orchestra.run.smithi190.stderr:10+0 records out 2024-03-24T12:25:24.715 INFO:teuthology.orchestra.run.smithi190.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125038 s, 839 MB/s 2024-03-24T12:25:24.715 INFO:teuthology.orchestra.run.smithi190.stderr:--> Zapping successful for: 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: osdmap e53: 7 total, 7 up, 7 in 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[29919]: osdmap e54: 7 total, 7 up, 7 in 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: osdmap e53: 7 total, 7 up, 7 in 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:25.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:25.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:25.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:25.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:25.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:25.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:25.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:25.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:24 smithi029 ceph-mon[41500]: osdmap e54: 7 total, 7 up, 7 in 2024-03-24T12:25:25.360 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:25.360 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: osdmap e53: 7 total, 7 up, 7 in 2024-03-24T12:25:25.360 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:25.361 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:24 smithi190 ceph-mon[33748]: osdmap e54: 7 total, 7 up, 7 in 2024-03-24T12:25:25.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi190:vg_nvme/lv_1 2024-03-24T12:25:25.628 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:25:26.002 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:25 smithi190 ceph-mon[33748]: Detected new or changed devices on smithi190 2024-03-24T12:25:26.003 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:25 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi190 to 5696M 2024-03-24T12:25:26.003 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:25 smithi190 ceph-mon[33748]: osdmap e55: 7 total, 7 up, 7 in 2024-03-24T12:25:26.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:25 smithi029 ceph-mon[29919]: Detected new or changed devices on smithi190 2024-03-24T12:25:26.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:25 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi190 to 5696M 2024-03-24T12:25:26.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:25 smithi029 ceph-mon[29919]: osdmap e55: 7 total, 7 up, 7 in 2024-03-24T12:25:26.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:25 smithi029 ceph-mon[41500]: Detected new or changed devices on smithi190 2024-03-24T12:25:26.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:25 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi190 to 5696M 2024-03-24T12:25:26.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:25 smithi029 ceph-mon[41500]: osdmap e55: 7 total, 7 up, 7 in 2024-03-24T12:25:27.041 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:26 smithi190 ceph-mon[33748]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:27.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:26 smithi029 ceph-mon[29919]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:27.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:26 smithi029 ceph-mon[41500]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:28.153 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:27 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:25:28.153 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:27 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:25:28.153 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:27 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:28.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:25:28.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:25:28.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:27 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:27 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T12:25:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:27 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T12:25:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:27 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:29.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:28 smithi190 ceph-mon[33748]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-24T12:25:29.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:28 smithi190 ceph-mon[33748]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:25:29.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:28 smithi029 ceph-mon[29919]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-24T12:25:29.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:28 smithi029 ceph-mon[29919]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:25:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:28 smithi029 ceph-mon[41500]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 134 KiB/s, 0 objects/s recovering 2024-03-24T12:25:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:28 smithi029 ceph-mon[41500]: from='client.24298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi190:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:25:31.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[29919]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T12:25:31.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/764574278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda"}]: dispatch 2024-03-24T12:25:31.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda"}]: dispatch 2024-03-24T12:25:31.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[29919]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda"}]': finished 2024-03-24T12:25:31.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[29919]: osdmap e56: 8 total, 7 up, 8 in 2024-03-24T12:25:31.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:31.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[41500]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T12:25:31.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/764574278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda"}]: dispatch 2024-03-24T12:25:31.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda"}]: dispatch 2024-03-24T12:25:31.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[41500]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda"}]': finished 2024-03-24T12:25:31.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[41500]: osdmap e56: 8 total, 7 up, 8 in 2024-03-24T12:25:31.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:30 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:31.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:30 smithi190 ceph-mon[33748]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T12:25:31.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:30 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/764574278' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda"}]: dispatch 2024-03-24T12:25:31.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:30 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda"}]: dispatch 2024-03-24T12:25:31.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:30 smithi190 ceph-mon[33748]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda"}]': finished 2024-03-24T12:25:31.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:30 smithi190 ceph-mon[33748]: osdmap e56: 8 total, 7 up, 8 in 2024-03-24T12:25:31.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:30 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:32.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:31 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/4260174475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:25:32.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:31 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/4260174475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:25:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:31 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/4260174475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T12:25:33.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:32 smithi029 ceph-mon[29919]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-24T12:25:33.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:32 smithi029 ceph-mon[41500]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-24T12:25:33.272 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:32 smithi190 ceph-mon[33748]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-03-24T12:25:35.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:34 smithi029 ceph-mon[29919]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T12:25:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:34 smithi029 ceph-mon[41500]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T12:25:35.350 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:34 smithi190 ceph-mon[33748]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T12:25:36.965 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:36 smithi190 ceph-mon[33748]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-24T12:25:37.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:36 smithi029 ceph-mon[29919]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-24T12:25:37.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:36 smithi029 ceph-mon[41500]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-03-24T12:25:38.316 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:37 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:25:38.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:37 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:25:38.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:37 smithi190 ceph-mon[33748]: osdmap e57: 8 total, 7 up, 8 in 2024-03-24T12:25:38.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:37 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:38.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:37 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T12:25:38.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:37 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:38.317 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:37 smithi190 ceph-mon[33748]: Deploying daemon osd.7 on smithi190 2024-03-24T12:25:38.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:25:38.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:25:38.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[29919]: osdmap e57: 8 total, 7 up, 8 in 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[29919]: Deploying daemon osd.7 on smithi190 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[41500]: osdmap e57: 8 total, 7 up, 8 in 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:37 smithi029 ceph-mon[41500]: Deploying daemon osd.7 on smithi190 2024-03-24T12:25:39.200 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:38 smithi190 ceph-mon[33748]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:39.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:38 smithi029 ceph-mon[29919]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:39.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:38 smithi029 ceph-mon[41500]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:40 smithi190 ceph-mon[33748]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:41.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:40 smithi029 ceph-mon[29919]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:41.451 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:40 smithi029 ceph-mon[41500]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:43.332 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:42 smithi190 ceph-mon[33748]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:43.332 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:43.332 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:43.332 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:42 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:43.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:42 smithi029 ceph-mon[29919]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:43.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:43.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:43.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:42 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:43.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:42 smithi029 ceph-mon[41500]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:43.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:43.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:43.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:42 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:45.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:44 smithi190 ceph-mon[33748]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:45.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:44 smithi029 ceph-mon[29919]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:45.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:44 smithi029 ceph-mon[41500]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:46.483 INFO:teuthology.orchestra.run.smithi190.stdout:Created osd(s) 7 on host 'smithi190' 2024-03-24T12:25:46.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:46.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:46.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:46.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:46.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:46.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:46.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: from='osd.7 v2:172.21.15.190:6812/110963949' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T12:25:46.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T12:25:46.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:46.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:46.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: from='osd.7 v2:172.21.15.190:6812/110963949' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T12:25:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T12:25:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:46 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:47.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:47.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:47.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:47.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:47.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:47.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:47.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: from='osd.7 v2:172.21.15.190:6812/110963949' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T12:25:47.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T12:25:47.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:47.126 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:47.127 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:46 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:47.152 DEBUG:teuthology.orchestra.run.smithi190:osd.7> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T12:25:47.155 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-24T12:25:47.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T12:25:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:25:47.819 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:47 smithi029 ceph-mon[29919]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T12:25:47.819 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:47 smithi029 ceph-mon[29919]: from='osd.7 v2:172.21.15.190:6812/110963949' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:47.819 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:47 smithi029 ceph-mon[29919]: osdmap e58: 8 total, 7 up, 8 in 2024-03-24T12:25:47.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:47 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:47.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:47 smithi029 ceph-mon[29919]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:47.820 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:47 smithi029 ceph-mon[41500]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T12:25:47.820 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:47 smithi029 ceph-mon[41500]: from='osd.7 v2:172.21.15.190:6812/110963949' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:47.820 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:47 smithi029 ceph-mon[41500]: osdmap e58: 8 total, 7 up, 8 in 2024-03-24T12:25:47.820 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:47 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:47.820 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:47 smithi029 ceph-mon[41500]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:48.041 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:47 smithi190 ceph-mon[33748]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T12:25:48.041 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:47 smithi190 ceph-mon[33748]: from='osd.7 v2:172.21.15.190:6812/110963949' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:48.041 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:47 smithi190 ceph-mon[33748]: osdmap e58: 8 total, 7 up, 8 in 2024-03-24T12:25:48.042 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:47 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:48.042 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:47 smithi190 ceph-mon[33748]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:25:48.042 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:25:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:25:47.681+0000 7fa364a2a700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T12:25:48.042 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:25:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:25:47.689+0000 7fa36081d700 -1 osd.7 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T12:25:48.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:48 smithi190 ceph-mon[33748]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:48.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:48 smithi190 ceph-mon[33748]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:25:48.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:48 smithi190 ceph-mon[33748]: osdmap e59: 8 total, 7 up, 8 in 2024-03-24T12:25:48.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:48 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:48.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:48 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:49.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:48 smithi029 ceph-mon[29919]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:49.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:48 smithi029 ceph-mon[29919]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:25:49.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:48 smithi029 ceph-mon[29919]: osdmap e59: 8 total, 7 up, 8 in 2024-03-24T12:25:49.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:48 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:49.040 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:48 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:49.040 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:48 smithi029 ceph-mon[41500]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail 2024-03-24T12:25:49.040 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:48 smithi029 ceph-mon[41500]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]': finished 2024-03-24T12:25:49.040 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:48 smithi029 ceph-mon[41500]: osdmap e59: 8 total, 7 up, 8 in 2024-03-24T12:25:49.040 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:48 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:49.040 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:48 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:49.473 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:25:49.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:25:49.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:25:49.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[29919]: osd.7 v2:172.21.15.190:6812/110963949 boot 2024-03-24T12:25:49.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[29919]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T12:25:49.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:49.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3309314355' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T12:25:49.845 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[29919]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T12:25:49.846 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:25:49.846 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:25:49.846 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[41500]: osd.7 v2:172.21.15.190:6812/110963949 boot 2024-03-24T12:25:49.846 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[41500]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T12:25:49.846 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:49.846 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3309314355' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T12:25:49.846 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:49 smithi029 ceph-mon[41500]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T12:25:49.939 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:49 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:25:49.939 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:49 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:25:49.939 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:49 smithi190 ceph-mon[33748]: osd.7 v2:172.21.15.190:6812/110963949 boot 2024-03-24T12:25:49.939 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:49 smithi190 ceph-mon[33748]: osdmap e60: 8 total, 8 up, 8 in 2024-03-24T12:25:49.939 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:49 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:25:49.939 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:49 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3309314355' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T12:25:49.939 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:49 smithi190 ceph-mon[33748]: osdmap e61: 8 total, 8 up, 8 in 2024-03-24T12:25:50.292 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":60,"num_osds":8,"num_up_osds":8,"osd_up_since":1711283148,"num_in_osds":8,"osd_in_since":1711283130,"num_remapped_pgs":1} 2024-03-24T12:25:50.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T12:25:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:25:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:25:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:25:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:25:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: Detected new or changed devices on smithi190 2024-03-24T12:25:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi190 to 4272M 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[29919]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:25:51.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: Detected new or changed devices on smithi190 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi190 to 4272M 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:50 smithi029 ceph-mon[41500]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T12:25:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:25:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:25:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:25:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: Detected new or changed devices on smithi190 2024-03-24T12:25:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:25:51.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi190 to 4272M 2024-03-24T12:25:51.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:25:51.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:25:51.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:51.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:50 smithi190 ceph-mon[33748]: osdmap e62: 8 total, 8 up, 8 in 2024-03-24T12:25:52.496 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:25:52.496 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":63,"fsid":"bc017092-e9d8-11ee-95cd-87774f69a715","created":"2024-03-24T12:20:18.712998+0000","modified":"2024-03-24T12:25:51.680549+0000","last_up_change":"2024-03-24T12:25:48.676565+0000","last_in_change":"2024-03-24T12:25:30.369664+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T12:23:41.491597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0aa1c49b-7887-4f74-b03c-08cc0adec90b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6801","nonce":2514818464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2514818464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2514818464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6803","nonce":2514818464}]},"public_addr":"172.21.15.29:6801/2514818464","cluster_addr":"172.21.15.29:6802/2514818464","heartbeat_back_addr":"172.21.15.29:6804/2514818464","heartbeat_front_addr":"172.21.15.29:6803/2514818464","state":["exists","up"]},{"osd":1,"uuid":"99d8ef8f-eb59-4882-ab51-b398e6b53bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":36,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":2923396572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2923396572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2923396572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6807","nonce":2923396572}]},"public_addr":"172.21.15.29:6805/2923396572","cluster_addr":"172.21.15.29:6806/2923396572","heartbeat_back_addr":"172.21.15.29:6808/2923396572","heartbeat_front_addr":"172.21.15.29:6807/2923396572","state":["exists","up"]},{"osd":2,"uuid":"c3f83505-25ee-4217-b60a-081c36092e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":126032915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":126032915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":126032915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":126032915}]},"public_addr":"172.21.15.29:6809/126032915","cluster_addr":"172.21.15.29:6810/126032915","heartbeat_back_addr":"172.21.15.29:6812/126032915","heartbeat_front_addr":"172.21.15.29:6811/126032915","state":["exists","up"]},{"osd":3,"uuid":"f5e479a8-459a-4315-a808-711a7b84bb96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1080171824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1080171824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1080171824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":1080171824}]},"public_addr":"172.21.15.29:6813/1080171824","cluster_addr":"172.21.15.29:6814/1080171824","heartbeat_back_addr":"172.21.15.29:6816/1080171824","heartbeat_front_addr":"172.21.15.29:6815/1080171824","state":["exists","up"]},{"osd":4,"uuid":"5e420304-2798-4d11-9688-d0d89482008c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3690258869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":3690258869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":3690258869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3690258869}]},"public_addr":"172.21.15.190:6800/3690258869","cluster_addr":"172.21.15.190:6801/3690258869","heartbeat_back_addr":"172.21.15.190:6803/3690258869","heartbeat_front_addr":"172.21.15.190:6802/3690258869","state":["exists","up"]},{"osd":5,"uuid":"1213431a-652c-4cbf-b32a-36335430c666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1327220488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":1327220488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":1327220488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1327220488}]},"public_addr":"172.21.15.190:6804/1327220488","cluster_addr":"172.21.15.190:6805/1327220488","heartbeat_back_addr":"172.21.15.190:6807/1327220488","heartbeat_front_addr":"172.21.15.190:6806/1327220488","state":["exists","up"]},{"osd":6,"uuid":"0712e511-0257-44f3-a82f-ee09ab957c0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1295064976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":1295064976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":1295064976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1295064976}]},"public_addr":"172.21.15.190:6808/1295064976","cluster_addr":"172.21.15.190:6809/1295064976","heartbeat_back_addr":"172.21.15.190:6811/1295064976","heartbeat_front_addr":"172.21.15.190:6810/1295064976","state":["exists","up"]},{"osd":7,"uuid":"7e36bde4-14d0-4e7e-9a82-2bb97859dbda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":110963949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":110963949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":110963949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":110963949}]},"public_addr":"172.21.15.190:6812/110963949","cluster_addr":"172.21.15.190:6813/110963949","heartbeat_back_addr":"172.21.15.190:6815/110963949","heartbeat_front_addr":"172.21.15.190:6814/110963949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:14.593793+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:36.630023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:24:47.542754+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:19.880932+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:49.675888+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:24:55.056047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:20.545588+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:46.876664+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/4130171883":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/2929430696":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/327626505":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/3715448582":"2024-03-25T12:20:46.948976+0000","172.21.15.29:6800/2109454792":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/1631806715":"2024-03-25T12:20:46.948976+0000","172.21.15.29:6800/852212806":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/1823252077":"2024-03-25T12:21:15.151901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T12:25:52.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:52 smithi029 ceph-mon[29919]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:25:52.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:52 smithi029 ceph-mon[29919]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T12:25:52.825 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:52 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3781821949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:25:52.825 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:52 smithi029 ceph-mon[41500]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:25:52.825 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:52 smithi029 ceph-mon[41500]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T12:25:52.825 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:52 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3781821949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:25:53.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:52 smithi190 ceph-mon[33748]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:25:53.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:52 smithi190 ceph-mon[33748]: osdmap e63: 8 total, 8 up, 8 in 2024-03-24T12:25:53.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:52 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3781821949' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:25:53.331 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T12:23:41.491597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T12:25:53.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T12:25:53.579 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:25:53.976 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:53 smithi029 ceph-mon[29919]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T12:25:53.976 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:53 smithi029 ceph-mon[41500]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T12:25:54.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:53 smithi190 ceph-mon[33748]: osdmap e64: 8 total, 8 up, 8 in 2024-03-24T12:25:54.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:54 smithi029 ceph-mon[29919]: pgmap v193: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T12:25:54.870 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:54 smithi029 ceph-mon[41500]: pgmap v193: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T12:25:55.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:54 smithi190 ceph-mon[33748]: pgmap v193: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T12:25:55.645 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2024-03-24T12:25:56.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:55 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:25:56.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:55 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:25:56.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:55 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1084595018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T12:25:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:55 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:25:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:55 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:25:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:55 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1084595018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T12:25:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:55 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:25:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:55 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:25:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:55 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1084595018' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T12:25:56.306 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi029 2024-03-24T12:25:56.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi029=foo.a' 2024-03-24T12:25:56.560 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:25:56.992 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:56 smithi190 ceph-mon[33748]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-24T12:25:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:56 smithi029 ceph-mon[29919]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-24T12:25:57.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:56 smithi029 ceph-mon[41500]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-24T12:25:58.477 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled rgw.foo.a update... 2024-03-24T12:25:59.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:58 smithi190 ceph-mon[33748]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-24T12:25:59.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:58 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:25:59.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:58 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:25:59.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:58 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:59.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:58 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:59.164 DEBUG:teuthology.orchestra.run.smithi029:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@rgw.foo.a.service 2024-03-24T12:25:59.166 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi190 2024-03-24T12:25:59.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-24T12:25:59.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:58 smithi029 ceph-mon[29919]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-24T12:25:59.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:58 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:25:59.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:58 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:25:59.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:58 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:59.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:58 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:59.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:58 smithi029 ceph-mon[41500]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-24T12:25:59.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:58 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:25:59.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:58 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:25:59.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:58 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:25:59.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:58 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:25:59.402 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi029=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:00.014 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[41500]: Deploying daemon rgw.foo.a on smithi029 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi029=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:00.015 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:25:59 smithi029 ceph-mon[29919]: Deploying daemon rgw.foo.a on smithi029 2024-03-24T12:26:00.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: from='client.24337 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi029=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:00.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-03-24T12:26:00.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: osdmap e65: 8 total, 8 up, 8 in 2024-03-24T12:26:00.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:00.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:26:00.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:00.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-24T12:26:00.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-24T12:26:00.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:00.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:00.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:25:59 smithi190 ceph-mon[33748]: Deploying daemon rgw.foo.a on smithi029 2024-03-24T12:26:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:01 smithi190 ceph-mon[33748]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:01.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:01 smithi029 ceph-mon[29919]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:01.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:01 smithi029 ceph-mon[41500]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:01.420 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:26:01 smithi029 systemd[1]: Starting Ceph rgw.foo.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:26:01.816 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:26:01 smithi029 podman[70557]: 2024-03-24 12:26:01.567955476 +0000 UTC m=+0.114461117 container create e3dc4a015866345f50ac4ad0fa6c83782b04f4ada54dc023a46b6b9d996c8953 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-rgw-foo-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T12:26:01.816 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:26:01 smithi029 podman[70557]: 2024-03-24 12:26:01.471800215 +0000 UTC m=+0.018305866 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:26:01.816 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:26:01 smithi029 podman[70557]: 2024-03-24 12:26:01.813321009 +0000 UTC m=+0.359826647 container init e3dc4a015866345f50ac4ad0fa6c83782b04f4ada54dc023a46b6b9d996c8953 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-rgw-foo-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-db0330b) 2024-03-24T12:26:01.816 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:26:01 smithi029 podman[70557]: 2024-03-24 12:26:01.818526219 +0000 UTC m=+0.365031857 container start e3dc4a015866345f50ac4ad0fa6c83782b04f4ada54dc023a46b6b9d996c8953 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-rgw-foo-a, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:26:02.146 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:26:01 smithi029 bash[70557]: e3dc4a015866345f50ac4ad0fa6c83782b04f4ada54dc023a46b6b9d996c8953 2024-03-24T12:26:02.146 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:26:01 smithi029 systemd[1]: Started Ceph rgw.foo.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:26:02.146 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/421454269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T12:26:02.146 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T12:26:02.146 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.157 INFO:teuthology.orchestra.run.smithi190.stderr:pool 'datapool' created 2024-03-24T12:26:02.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/421454269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T12:26:02.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T12:26:02.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:02 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.514 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:02 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/421454269' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T12:26:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:02 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-24T12:26:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:02 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:02 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:02 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:02 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.515 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:02 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:02.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- rbd pool init datapool 2024-03-24T12:26:03.067 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:26:03.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[29919]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:03.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[29919]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-03-24T12:26:03.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T12:26:03.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[29919]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T12:26:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3179604327' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T12:26:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[29919]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T12:26:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:03.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[41500]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:03.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[41500]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-03-24T12:26:03.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T12:26:03.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[41500]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T12:26:03.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3179604327' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T12:26:03.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[41500]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T12:26:03.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:03 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:03.459 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:03 smithi190 ceph-mon[33748]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:03.460 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:03 smithi190 ceph-mon[33748]: Saving service rgw.foo.a spec with placement smithi029=foo.a;count:1 2024-03-24T12:26:03.460 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:03 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-24T12:26:03.460 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:03 smithi190 ceph-mon[33748]: osdmap e66: 8 total, 8 up, 8 in 2024-03-24T12:26:03.460 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:03 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3179604327' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T12:26:03.460 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:03 smithi190 ceph-mon[33748]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-24T12:26:03.460 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:03 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:04.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:04 smithi190 ceph-mon[33748]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T12:26:04.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:04 smithi190 ceph-mon[33748]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T12:26:04.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:04.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:04.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:04.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:04 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:26:04.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[29919]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T12:26:04.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[29919]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T12:26:04.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:04.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:04.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:04.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:26:04.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[41500]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-24T12:26:04.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[41500]: osdmap e67: 8 total, 8 up, 8 in 2024-03-24T12:26:04.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:04.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:04.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:04.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:04 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: pgmap v201: 36 pgs: 9 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: Checking dashboard <-> RGW credentials 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/1274157617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T12:26:05.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:05 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:05.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: pgmap v201: 36 pgs: 9 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:05.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: Checking dashboard <-> RGW credentials 2024-03-24T12:26:05.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:26:05.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T12:26:05.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/1274157617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T12:26:05.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T12:26:05.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:05.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: pgmap v201: 36 pgs: 9 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:05.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: Checking dashboard <-> RGW credentials 2024-03-24T12:26:05.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:26:05.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: osdmap e68: 8 total, 8 up, 8 in 2024-03-24T12:26:05.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-24T12:26:05.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/1274157617' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T12:26:05.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-24T12:26:05.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:05 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:06.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:06 smithi190 ceph-mon[33748]: pgmap v203: 68 pgs: 8 active+clean, 9 creating+peering, 51 unknown; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 341 B/s wr, 0 op/s 2024-03-24T12:26:06.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:06 smithi190 ceph-mon[33748]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T12:26:06.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:06 smithi190 ceph-mon[33748]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T12:26:06.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:06 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T12:26:06.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:06 smithi190 ceph-mon[33748]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T12:26:06.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:06 smithi029 ceph-mon[29919]: pgmap v203: 68 pgs: 8 active+clean, 9 creating+peering, 51 unknown; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 341 B/s wr, 0 op/s 2024-03-24T12:26:06.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:06 smithi029 ceph-mon[29919]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T12:26:06.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:06 smithi029 ceph-mon[29919]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T12:26:06.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:06 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T12:26:06.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:06 smithi029 ceph-mon[29919]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T12:26:06.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:06 smithi029 ceph-mon[41500]: pgmap v203: 68 pgs: 8 active+clean, 9 creating+peering, 51 unknown; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail; 341 B/s rd, 341 B/s wr, 0 op/s 2024-03-24T12:26:06.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:06 smithi029 ceph-mon[41500]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T12:26:06.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:06 smithi029 ceph-mon[41500]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-24T12:26:06.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:06 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-24T12:26:06.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:06 smithi029 ceph-mon[41500]: osdmap e69: 8 total, 8 up, 8 in 2024-03-24T12:26:07.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:07 smithi190 ceph-mon[33748]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T12:26:07.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:07 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:07 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:07 smithi190 ceph-mon[33748]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:07 smithi190 ceph-mon[33748]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:07 smithi190 ceph-mon[33748]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T12:26:07.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:07 smithi190 ceph-mon[33748]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T12:26:07.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:07 smithi190 ceph-mon[33748]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T12:26:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[29919]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T12:26:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[29919]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[29919]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[29919]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T12:26:07.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[29919]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T12:26:07.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[29919]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T12:26:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[41500]: osdmap e70: 8 total, 8 up, 8 in 2024-03-24T12:26:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[41500]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[41500]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-24T12:26:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[41500]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T12:26:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[41500]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-24T12:26:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:07 smithi029 ceph-mon[41500]: osdmap e71: 8 total, 8 up, 8 in 2024-03-24T12:26:07.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.190 --placement '1;smithi190=iscsi.a' 2024-03-24T12:26:08.238 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:26:08.334 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:08 smithi190 ceph-mon[33748]: pgmap v206: 100 pgs: 47 active+clean, 21 creating+peering, 32 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-24T12:26:08.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:08 smithi029 ceph-mon[29919]: pgmap v206: 100 pgs: 47 active+clean, 21 creating+peering, 32 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-24T12:26:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:08 smithi029 ceph-mon[41500]: pgmap v206: 100 pgs: 47 active+clean, 21 creating+peering, 32 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-24T12:26:09.590 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T12:26:09.590 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.590 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T12:26:09.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T12:26:09.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T12:26:09.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:09.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:09.591 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:09 smithi190 ceph-mon[33748]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:09.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T12:26:09.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[29919]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: osdmap e72: 8 total, 8 up, 8 in 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-24T12:26:09.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T12:26:09.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-24T12:26:09.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: osdmap e73: 8 total, 8 up, 8 in 2024-03-24T12:26:09.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3808533613' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:09.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4003367372' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:09.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:09 smithi029 ceph-mon[41500]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:10.057 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled iscsi.datapool update... 2024-03-24T12:26:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:10 smithi190 ceph-mon[33748]: pgmap v209: 132 pgs: 80 active+clean, 18 creating+peering, 34 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T12:26:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:10 smithi190 ceph-mon[33748]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:10.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:10 smithi190 ceph-mon[33748]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.190", "placement": "1;smithi190=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:10.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:10 smithi190 ceph-mon[33748]: Saving service iscsi.datapool spec with placement smithi190=iscsi.a;count:1 2024-03-24T12:26:10.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:10 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:10.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:10 smithi190 ceph-mon[33748]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T12:26:10.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:10 smithi190 ceph-mon[33748]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T12:26:10.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:10 smithi190 ceph-mon[33748]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T12:26:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[29919]: pgmap v209: 132 pgs: 80 active+clean, 18 creating+peering, 34 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T12:26:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[29919]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[29919]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.190", "placement": "1;smithi190=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[29919]: Saving service iscsi.datapool spec with placement smithi190=iscsi.a;count:1 2024-03-24T12:26:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[29919]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T12:26:10.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[29919]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T12:26:10.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[29919]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T12:26:10.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[41500]: pgmap v209: 132 pgs: 80 active+clean, 18 creating+peering, 34 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-24T12:26:10.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[41500]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-24T12:26:10.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[41500]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.190", "placement": "1;smithi190=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:10.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[41500]: Saving service iscsi.datapool spec with placement smithi190=iscsi.a;count:1 2024-03-24T12:26:10.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:10.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[41500]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T12:26:10.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[41500]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-24T12:26:10.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:10 smithi029 ceph-mon[41500]: osdmap e74: 8 total, 8 up, 8 in 2024-03-24T12:26:10.707 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-24T12:26:10.708 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:26:10.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-24T12:26:10.738 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:26:10.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-24T12:26:10.766 DEBUG:teuthology.orchestra.run.smithi190:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@iscsi.iscsi.a.service 2024-03-24T12:26:10.809 INFO:tasks.cephadm:Adding prometheus.a on smithi190 2024-03-24T12:26:10.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch apply prometheus '1;smithi190=a' 2024-03-24T12:26:11.093 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:26:11.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:11.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:11.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:11.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T12:26:11.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T12:26:11.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:11 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:11.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:11 smithi190 ceph-mon[33748]: Deploying daemon iscsi.iscsi.a on smithi190 2024-03-24T12:26:11.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:11.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:11.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:11.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T12:26:11.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T12:26:11.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:11.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[29919]: Deploying daemon iscsi.iscsi.a on smithi190 2024-03-24T12:26:11.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:11.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:11.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:11.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-24T12:26:11.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-24T12:26:11.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:11.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:11 smithi029 ceph-mon[41500]: Deploying daemon iscsi.iscsi.a on smithi190 2024-03-24T12:26:12.816 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:12 smithi190 ceph-mon[33748]: pgmap v212: 132 pgs: 87 active+clean, 18 creating+peering, 27 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-24T12:26:12.816 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:12 smithi190 ceph-mon[33748]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:26:12.817 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:12 smithi190 ceph-mon[33748]: Cluster is now healthy 2024-03-24T12:26:12.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:12 smithi029 ceph-mon[29919]: pgmap v212: 132 pgs: 87 active+clean, 18 creating+peering, 27 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-24T12:26:12.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:12 smithi029 ceph-mon[29919]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:26:12.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:12 smithi029 ceph-mon[29919]: Cluster is now healthy 2024-03-24T12:26:12.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:12 smithi029 ceph-mon[41500]: pgmap v212: 132 pgs: 87 active+clean, 18 creating+peering, 27 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 767 B/s wr, 5 op/s 2024-03-24T12:26:12.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:12 smithi029 ceph-mon[41500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:26:12.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:12 smithi029 ceph-mon[41500]: Cluster is now healthy 2024-03-24T12:26:14.276 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled prometheus update... 2024-03-24T12:26:14.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:14 smithi190 ceph-mon[33748]: pgmap v213: 132 pgs: 126 active+clean, 6 creating+peering; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 6.3 KiB/s wr, 186 op/s 2024-03-24T12:26:14.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:14 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:14.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:14 smithi029 ceph-mon[29919]: pgmap v213: 132 pgs: 126 active+clean, 6 creating+peering; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 6.3 KiB/s wr, 186 op/s 2024-03-24T12:26:14.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:14 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:14.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:14 smithi029 ceph-mon[41500]: pgmap v213: 132 pgs: 126 active+clean, 6 creating+peering; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 6.3 KiB/s wr, 186 op/s 2024-03-24T12:26:14.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:14 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:14.926 DEBUG:teuthology.orchestra.run.smithi190:prometheus.a> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@prometheus.a.service 2024-03-24T12:26:14.929 INFO:tasks.cephadm:Adding node-exporter.a on smithi029 2024-03-24T12:26:14.929 INFO:tasks.cephadm:Adding node-exporter.b on smithi190 2024-03-24T12:26:14.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch apply node-exporter '2;smithi029=a;smithi190=b' 2024-03-24T12:26:15.200 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:15 smithi190 podman[62204]: 2024-03-24 12:26:15.100102024 +0000 UTC m=+1.018909684 container start d64804989f550cbafeb03a7184d7f6801a7169e011b27acaab5bcbec3bbc81c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a, ceph=True, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:26:15.200 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:15 smithi190 bash[62204]: d64804989f550cbafeb03a7184d7f6801a7169e011b27acaab5bcbec3bbc81c2 2024-03-24T12:26:15.308 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:26:15.569 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:15 smithi190 systemd[1]: Started Ceph iscsi.iscsi.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:26:15.569 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug Started the configuration object watcher 2024-03-24T12:26:15.569 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug Checking for config object changes every 1s 2024-03-24T12:26:15.569 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug Processing osd blocklist entries for this node 2024-03-24T12:26:15.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug Reading the configuration object to update local LIO configuration 2024-03-24T12:26:15.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug Configuration does not have an entry for this host(smithi190.front.sepia.ceph.com) - nothing to define to LIO 2024-03-24T12:26:15.929 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-24T12:26:15.929 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:15.929 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: Saving service prometheus spec with placement smithi190=a;count:1 2024-03-24T12:26:15.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T12:26:15.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-24T12:26:15.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-24T12:26:15.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-24T12:26:15.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:15.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:15.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:15.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:15.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:15.930 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:15 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/2296368452' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T12:26:16.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:16.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: Saving service prometheus spec with placement smithi190=a;count:1 2024-03-24T12:26:16.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T12:26:16.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/2296368452' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: Saving service prometheus spec with placement smithi190=a;count:1 2024-03-24T12:26:16.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: osdmap e75: 8 total, 8 up, 8 in 2024-03-24T12:26:16.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-24T12:26:16.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-24T12:26:16.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2024-03-24T12:26:16.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:16.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:16.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:16.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:16.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:16.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:15 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/2296368452' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T12:26:17.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:16 smithi190 ceph-mon[33748]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 6.0 KiB/s wr, 182 op/s 2024-03-24T12:26:17.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:16 smithi190 ceph-mon[33748]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T12:26:17.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:16 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:26:17.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:16 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:26:17.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-24T12:26:17.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-24T12:26:17.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:16 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-24T12:26:17.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:16 smithi190 ceph-mon[33748]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T12:26:17.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[29919]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 6.0 KiB/s wr, 182 op/s 2024-03-24T12:26:17.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[29919]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T12:26:17.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:26:17.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:26:17.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[29919]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[41500]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 6.0 KiB/s wr, 182 op/s 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[41500]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2024-03-24T12:26:17.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:16 smithi029 ceph-mon[41500]: osdmap e76: 8 total, 8 up, 8 in 2024-03-24T12:26:17.991 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:17 smithi190 ceph-mon[33748]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T12:26:18.068 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled node-exporter update... 2024-03-24T12:26:18.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:17 smithi029 ceph-mon[29919]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T12:26:18.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:17 smithi029 ceph-mon[41500]: osdmap e77: 8 total, 8 up, 8 in 2024-03-24T12:26:18.771 DEBUG:teuthology.orchestra.run.smithi029:node-exporter.a> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@node-exporter.a.service 2024-03-24T12:26:18.775 DEBUG:teuthology.orchestra.run.smithi190:node-exporter.b> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@node-exporter.b.service 2024-03-24T12:26:18.777 INFO:tasks.cephadm:Adding alertmanager.a on smithi029 2024-03-24T12:26:18.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch apply alertmanager '1;smithi029=a' 2024-03-24T12:26:19.072 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:26:19.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:18 smithi190 ceph-mon[33748]: pgmap v218: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.8 KiB/s wr, 185 op/s 2024-03-24T12:26:19.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:18 smithi190 ceph-mon[33748]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-24T12:26:19.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:18 smithi190 ceph-mon[33748]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi029=a;smithi190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:19.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:18 smithi190 ceph-mon[33748]: Saving service node-exporter spec with placement smithi029=a;smithi190=b;count:2 2024-03-24T12:26:19.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:18 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:19.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:18 smithi029 ceph-mon[29919]: pgmap v218: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.8 KiB/s wr, 185 op/s 2024-03-24T12:26:19.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:18 smithi029 ceph-mon[29919]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-24T12:26:19.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:18 smithi029 ceph-mon[29919]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi029=a;smithi190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:19.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:18 smithi029 ceph-mon[29919]: Saving service node-exporter spec with placement smithi029=a;smithi190=b;count:2 2024-03-24T12:26:19.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:18 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:18 smithi029 ceph-mon[41500]: pgmap v218: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 76 KiB/s rd, 5.8 KiB/s wr, 185 op/s 2024-03-24T12:26:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:18 smithi029 ceph-mon[41500]: mgrmap e16: y(active, since 5m), standbys: x 2024-03-24T12:26:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:18 smithi029 ceph-mon[41500]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi029=a;smithi190=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:18 smithi029 ceph-mon[41500]: Saving service node-exporter spec with placement smithi029=a;smithi190=b;count:2 2024-03-24T12:26:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:18 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:20 smithi190 ceph-mon[33748]: pgmap v219: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-24T12:26:21.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:20 smithi190 ceph-mon[33748]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T12:26:21.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:20 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:20 smithi029 ceph-mon[41500]: pgmap v219: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-24T12:26:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:20 smithi029 ceph-mon[41500]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T12:26:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:20 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:20 smithi029 ceph-mon[29919]: pgmap v219: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 3.3 KiB/s rd, 341 B/s wr, 8 op/s 2024-03-24T12:26:21.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:20 smithi029 ceph-mon[29919]: osdmap e78: 8 total, 8 up, 8 in 2024-03-24T12:26:21.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:20 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.609 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled alertmanager update... 2024-03-24T12:26:21.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:21 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:26:21.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:21 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:26:21.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:21.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:26:21.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:21.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:21 smithi190 ceph-mon[33748]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T12:26:21.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:21 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:26:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:26:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:22.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:26:22.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:22.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[29919]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T12:26:22.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:26:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:26:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:26:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:26:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[41500]: osdmap e79: 8 total, 8 up, 8 in 2024-03-24T12:26:22.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:21 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:22.331 DEBUG:teuthology.orchestra.run.smithi029:alertmanager.a> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@alertmanager.a.service 2024-03-24T12:26:22.334 INFO:tasks.cephadm:Adding grafana.a on smithi190 2024-03-24T12:26:22.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph orch apply grafana '1;smithi190=a' 2024-03-24T12:26:22.580 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:26:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:23 smithi190 ceph-mon[33748]: pgmap v221: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 7 op/s; 18 B/s, 1 objects/s recovering 2024-03-24T12:26:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:23 smithi190 ceph-mon[33748]: Deploying daemon prometheus.a on smithi190 2024-03-24T12:26:23.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:23 smithi190 ceph-mon[33748]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi029=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:23.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:23 smithi190 ceph-mon[33748]: Saving service alertmanager spec with placement smithi029=a;count:1 2024-03-24T12:26:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:23 smithi029 ceph-mon[41500]: pgmap v221: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 7 op/s; 18 B/s, 1 objects/s recovering 2024-03-24T12:26:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:23 smithi029 ceph-mon[41500]: Deploying daemon prometheus.a on smithi190 2024-03-24T12:26:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:23 smithi029 ceph-mon[41500]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi029=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:23 smithi029 ceph-mon[41500]: Saving service alertmanager spec with placement smithi029=a;count:1 2024-03-24T12:26:23.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:23 smithi029 ceph-mon[29919]: pgmap v221: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 170 B/s wr, 7 op/s; 18 B/s, 1 objects/s recovering 2024-03-24T12:26:23.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:23 smithi029 ceph-mon[29919]: Deploying daemon prometheus.a on smithi190 2024-03-24T12:26:23.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:23 smithi029 ceph-mon[29919]: from='client.24448 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi029=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:23.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:23 smithi029 ceph-mon[29919]: Saving service alertmanager spec with placement smithi029=a;count:1 2024-03-24T12:26:25.124 INFO:teuthology.orchestra.run.smithi190.stdout:Scheduled grafana update... 2024-03-24T12:26:25.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:25 smithi190 ceph-mon[33748]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 161 B/s wr, 2 op/s; 52 B/s, 2 objects/s recovering 2024-03-24T12:26:25.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:25 smithi190 ceph-mon[33748]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T12:26:25.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:25 smithi029 ceph-mon[41500]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 161 B/s wr, 2 op/s; 52 B/s, 2 objects/s recovering 2024-03-24T12:26:25.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:25 smithi029 ceph-mon[41500]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T12:26:25.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:25 smithi029 ceph-mon[29919]: pgmap v223: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 645 B/s rd, 161 B/s wr, 2 op/s; 52 B/s, 2 objects/s recovering 2024-03-24T12:26:25.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:25 smithi029 ceph-mon[29919]: osdmap e80: 8 total, 8 up, 8 in 2024-03-24T12:26:25.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:25 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:26:26.048 DEBUG:teuthology.orchestra.run.smithi190:grafana.a> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@grafana.a.service 2024-03-24T12:26:26.050 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T12:26:26.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T12:26:26.309 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:26.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:26 smithi190 ceph-mon[33748]: from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:26.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:26 smithi190 ceph-mon[33748]: Saving service grafana spec with placement smithi190=a;count:1 2024-03-24T12:26:26.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:26 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:26.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:26 smithi029 ceph-mon[29919]: from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:26.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:26 smithi029 ceph-mon[29919]: Saving service grafana spec with placement smithi190=a;count:1 2024-03-24T12:26:26.415 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:26 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:26.415 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:26 smithi029 ceph-mon[41500]: from='client.24454 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi190=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T12:26:26.415 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:26 smithi029 ceph-mon[41500]: Saving service grafana spec with placement smithi190=a;count:1 2024-03-24T12:26:26.415 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:26 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:27.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:27 smithi190 ceph-mon[33748]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54 B/s, 2 objects/s recovering 2024-03-24T12:26:27.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:27 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:27.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:27 smithi029 ceph-mon[29919]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54 B/s, 2 objects/s recovering 2024-03-24T12:26:27.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:27 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:27 smithi029 ceph-mon[41500]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54 B/s, 2 objects/s recovering 2024-03-24T12:26:27.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:27 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:28.423 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2024-03-24T12:26:28.423 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQD0GwBmXv4wGRAAR8iFuS7Wxf8wHHAVghKWFQ== 2024-03-24T12:26:29.217 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2024-03-24T12:26:29.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T12:26:29.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T12:26:29.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T12:26:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:29 smithi190 ceph-mon[33748]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 695 B/s rd, 0 op/s; 44 B/s, 2 objects/s recovering 2024-03-24T12:26:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:29 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2168903015' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:29 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:29 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T12:26:29.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:29 smithi029 ceph-mon[29919]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 695 B/s rd, 0 op/s; 44 B/s, 2 objects/s recovering 2024-03-24T12:26:29.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:29 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2168903015' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:29.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:29 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:29.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:29 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T12:26:29.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:29 smithi029 ceph-mon[41500]: pgmap v226: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 695 B/s rd, 0 op/s; 44 B/s, 2 objects/s recovering 2024-03-24T12:26:29.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:29 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2168903015' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:29.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:29 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:29.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:29 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T12:26:29.890 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:26:30.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:30 smithi190 ceph-mon[33748]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-24T12:26:30.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:30 smithi190 ceph-mon[33748]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T12:26:31.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:30 smithi029 ceph-mon[29919]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-24T12:26:31.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:30 smithi029 ceph-mon[29919]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T12:26:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:30 smithi029 ceph-mon[41500]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 0 objects/s recovering 2024-03-24T12:26:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:30 smithi029 ceph-mon[41500]: osdmap e81: 8 total, 8 up, 8 in 2024-03-24T12:26:33.063 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:32 smithi190 ceph-mon[33748]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:26:33.064 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:32 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:26:33.064 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:32 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:26:33.064 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:32 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:26:33.064 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:32 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:26:33.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:32 smithi029 ceph-mon[41500]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:26:33.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:32 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:26:33.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:32 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:26:33.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:32 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:26:33.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:32 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:26:33.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:32 smithi029 ceph-mon[29919]: pgmap v229: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:26:33.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:32 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:26:33.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:32 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:26:33.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:32 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:26:33.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:32 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:26:33.387 INFO:teuthology.orchestra.run.smithi190.stdout:[client.1] 2024-03-24T12:26:33.387 INFO:teuthology.orchestra.run.smithi190.stdout: key = AQD5GwBmr9v/FhAA80T4yKWo2O2WzcnGnGAg/g== 2024-03-24T12:26:34.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:33 smithi190 ceph-mon[33748]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T12:26:34.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:33 smithi190 ceph-mon[33748]: from='client.? 172.21.15.190:0/4195627531' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:34.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:33 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:34.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:33 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T12:26:34.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:33 smithi029 ceph-mon[41500]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T12:26:34.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:33 smithi029 ceph-mon[41500]: from='client.? 172.21.15.190:0/4195627531' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:34.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:33 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:34.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:33 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T12:26:34.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:33 smithi029 ceph-mon[29919]: osdmap e82: 8 total, 8 up, 8 in 2024-03-24T12:26:34.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:33 smithi029 ceph-mon[29919]: from='client.? 172.21.15.190:0/4195627531' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:34.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:33 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T12:26:34.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:33 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T12:26:34.549 DEBUG:teuthology.orchestra.run.smithi190:> set -ex 2024-03-24T12:26:34.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T12:26:34.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T12:26:34.597 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T12:26:34.597 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T12:26:34.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T12:26:34.849 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:34.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:34 smithi029 ceph-mon[29919]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:26:34.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:34 smithi029 ceph-mon[41500]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:26:35.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:34 smithi190 ceph-mon[33748]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:26:35.634 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:35 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:26:37.006 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:26:37.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:36 smithi190 ceph-mon[33748]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:26:37.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:36 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:37.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:36 smithi029 ceph-mon[29919]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:26:37.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:36 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:37.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:36 smithi029 ceph-mon[41500]: pgmap v232: 132 pgs: 132 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:26:37.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:36 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:37.426 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 podman[64293]: 2024-03-24 12:26:37.392258968 +0000 UTC m=+0.130408437 container create 3c089fa4a41da5e73cc096a5228d811a557319979d32f7adf54dfc5834524240 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:26:37.692 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 podman[64293]: 2024-03-24 12:26:37.644610274 +0000 UTC m=+0.382759710 container init 3c089fa4a41da5e73cc096a5228d811a557319979d32f7adf54dfc5834524240 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:26:37.692 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 podman[64293]: 2024-03-24 12:26:37.649903948 +0000 UTC m=+0.388053376 container start 3c089fa4a41da5e73cc096a5228d811a557319979d32f7adf54dfc5834524240 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:26:37.692 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.673Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T12:26:37.692 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.673Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.673Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi190 (none))" 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.673Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.673Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.674Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.674Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.675Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.675Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.677Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.677Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.335µs 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.677Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.677Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-24T12:26:37.693 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.678Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=25.992µs wal_replay_duration=212.527µs wbl_replay_duration=132ns total_replay_duration=264.172µs 2024-03-24T12:26:37.694 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.678Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T12:26:37.694 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.678Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T12:26:37.694 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.678Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T12:26:37.694 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.694Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=15.861968ms db_storage=737ns remote_storage=1.051µs web_handler=416ns query_engine=521ns scrape=197.783µs scrape_sd=61.428µs notify=689ns notify_sd=1.22µs rules=15.414824ms tracing=5.144µs 2024-03-24T12:26:37.694 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.694Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T12:26:37.694 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:26:37.694Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T12:26:37.786 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":2612078930}]},"active_addr":"172.21.15.29:6800/2612078930","active_change":"2024-03-24T12:21:15.152023+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24118,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.29:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3697529912}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":4113609697}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2633022206}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":861146182}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2365028933}]}]} 2024-03-24T12:26:37.790 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T12:26:37.790 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T12:26:37.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T12:26:38.046 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:38.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1606076258' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T12:26:38.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:38.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:38.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:38.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:37 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T12:26:38.135 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 bash[64293]: 3c089fa4a41da5e73cc096a5228d811a557319979d32f7adf54dfc5834524240 2024-03-24T12:26:38.135 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:26:37 smithi190 systemd[1]: Started Ceph prometheus.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:26:38.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:37 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1606076258' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T12:26:38.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:37 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:38.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:37 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:38.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:37 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:38.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:37 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T12:26:38.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:37 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1606076258' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T12:26:38.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:37 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:38.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:37 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:38.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:37 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' 2024-03-24T12:26:38.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:37 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T12:26:39.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:38 smithi190 ceph-mon[33748]: pgmap v233: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:26:39.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:38 smithi190 ceph-mon[33748]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T12:26:39.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:38 smithi190 ceph-mon[33748]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T12:26:39.135 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: ignoring --setuser ceph since I am not root 2024-03-24T12:26:39.136 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: ignoring --setgroup ceph since I am not root 2024-03-24T12:26:39.136 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:38 smithi190 ceph-mgr[35879]: -- 172.21.15.190:0/2162161105 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5644d1e881a0 con 0x5644d1e5f000 2024-03-24T12:26:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:38 smithi029 ceph-mon[29919]: pgmap v233: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:26:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:38 smithi029 ceph-mon[29919]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T12:26:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:38 smithi029 ceph-mon[29919]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T12:26:39.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:38 smithi029 ceph-mon[41500]: pgmap v233: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:26:39.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:38 smithi029 ceph-mon[41500]: from='mgr.14152 172.21.15.29:0/187854056' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T12:26:39.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:38 smithi029 ceph-mon[41500]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-24T12:26:39.170 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:38 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ignoring --setuser ceph since I am not root 2024-03-24T12:26:39.170 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:38 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ignoring --setgroup ceph since I am not root 2024-03-24T12:26:39.669 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:39 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:39.465+0000 7f40315cc200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T12:26:39.669 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:39 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:39.561+0000 7f40315cc200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T12:26:39.884 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:39 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:39.437+0000 7f90d6713200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T12:26:39.884 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:39 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:39.533+0000 7f90d6713200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T12:26:40.249 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:26:40.250 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":82,"fsid":"bc017092-e9d8-11ee-95cd-87774f69a715","created":"2024-03-24T12:20:18.712998+0000","modified":"2024-03-24T12:26:32.857662+0000","last_up_change":"2024-03-24T12:25:48.676565+0000","last_in_change":"2024-03-24T12:25:30.369664+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T12:23:41.491597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-24T12:26:01.465080+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-24T12:26:02.100021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T12:26:03.473862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T12:26:05.573828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T12:26:07.645083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0aa1c49b-7887-4f74-b03c-08cc0adec90b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6801","nonce":2514818464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2514818464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2514818464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6803","nonce":2514818464}]},"public_addr":"172.21.15.29:6801/2514818464","cluster_addr":"172.21.15.29:6802/2514818464","heartbeat_back_addr":"172.21.15.29:6804/2514818464","heartbeat_front_addr":"172.21.15.29:6803/2514818464","state":["exists","up"]},{"osd":1,"uuid":"99d8ef8f-eb59-4882-ab51-b398e6b53bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":2923396572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2923396572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2923396572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6807","nonce":2923396572}]},"public_addr":"172.21.15.29:6805/2923396572","cluster_addr":"172.21.15.29:6806/2923396572","heartbeat_back_addr":"172.21.15.29:6808/2923396572","heartbeat_front_addr":"172.21.15.29:6807/2923396572","state":["exists","up"]},{"osd":2,"uuid":"c3f83505-25ee-4217-b60a-081c36092e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":126032915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":126032915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":126032915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":126032915}]},"public_addr":"172.21.15.29:6809/126032915","cluster_addr":"172.21.15.29:6810/126032915","heartbeat_back_addr":"172.21.15.29:6812/126032915","heartbeat_front_addr":"172.21.15.29:6811/126032915","state":["exists","up"]},{"osd":3,"uuid":"f5e479a8-459a-4315-a808-711a7b84bb96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1080171824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1080171824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1080171824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":1080171824}]},"public_addr":"172.21.15.29:6813/1080171824","cluster_addr":"172.21.15.29:6814/1080171824","heartbeat_back_addr":"172.21.15.29:6816/1080171824","heartbeat_front_addr":"172.21.15.29:6815/1080171824","state":["exists","up"]},{"osd":4,"uuid":"5e420304-2798-4d11-9688-d0d89482008c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3690258869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":3690258869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":3690258869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3690258869}]},"public_addr":"172.21.15.190:6800/3690258869","cluster_addr":"172.21.15.190:6801/3690258869","heartbeat_back_addr":"172.21.15.190:6803/3690258869","heartbeat_front_addr":"172.21.15.190:6802/3690258869","state":["exists","up"]},{"osd":5,"uuid":"1213431a-652c-4cbf-b32a-36335430c666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1327220488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":1327220488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":1327220488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1327220488}]},"public_addr":"172.21.15.190:6804/1327220488","cluster_addr":"172.21.15.190:6805/1327220488","heartbeat_back_addr":"172.21.15.190:6807/1327220488","heartbeat_front_addr":"172.21.15.190:6806/1327220488","state":["exists","up"]},{"osd":6,"uuid":"0712e511-0257-44f3-a82f-ee09ab957c0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1295064976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":1295064976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":1295064976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1295064976}]},"public_addr":"172.21.15.190:6808/1295064976","cluster_addr":"172.21.15.190:6809/1295064976","heartbeat_back_addr":"172.21.15.190:6811/1295064976","heartbeat_front_addr":"172.21.15.190:6810/1295064976","state":["exists","up"]},{"osd":7,"uuid":"7e36bde4-14d0-4e7e-9a82-2bb97859dbda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":110963949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":110963949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":110963949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":110963949}]},"public_addr":"172.21.15.190:6812/110963949","cluster_addr":"172.21.15.190:6813/110963949","heartbeat_back_addr":"172.21.15.190:6815/110963949","heartbeat_front_addr":"172.21.15.190:6814/110963949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:26:32.019647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:36.630023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:57.919767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:26:20.595213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:49.675888+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:26:15.790889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:26:32.516737+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:46.876664+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/4130171883":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/2929430696":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/327626505":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/3715448582":"2024-03-25T12:20:46.948976+0000","172.21.15.29:6800/2109454792":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/1631806715":"2024-03-25T12:20:46.948976+0000","172.21.15.29:6800/852212806":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/1823252077":"2024-03-25T12:21:15.151901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T12:26:40.500 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:40 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/816567745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:26:40.501 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:40.170+0000 7f90d6713200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T12:26:40.501 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:40.262+0000 7f90d6713200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T12:26:40.501 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:40.405+0000 7f90d6713200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T12:26:40.501 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:40.502+0000 7f90d6713200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T12:26:40.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:40 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/816567745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:26:40.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:40 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/816567745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:26:40.557 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:40 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:40.316+0000 7f40315cc200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T12:26:40.557 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:40 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:40.412+0000 7f40315cc200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T12:26:40.558 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:40 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:40.559+0000 7f40315cc200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T12:26:40.847 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:40.585+0000 7f90d6713200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T12:26:40.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:40 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:40.659+0000 7f40315cc200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T12:26:40.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:40 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:40.745+0000 7f40315cc200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T12:26:41.070 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T12:26:41.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T12:26:41.134 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:40.849+0000 7f90d6713200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T12:26:41.134 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:40.958+0000 7f90d6713200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T12:26:41.134 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:41 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:41.043+0000 7f90d6713200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T12:26:41.324 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:41.337 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:41.020+0000 7f40315cc200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T12:26:41.337 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:41.136+0000 7f40315cc200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T12:26:41.337 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:41.227+0000 7f40315cc200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T12:26:41.589 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:41.338+0000 7f40315cc200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T12:26:41.589 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:41.521+0000 7f40315cc200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T12:26:41.634 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:41 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:41.138+0000 7f90d6713200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T12:26:41.634 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:41 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:41.313+0000 7f90d6713200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T12:26:42.634 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:42.266+0000 7f90d6713200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T12:26:42.634 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:42.394+0000 7f90d6713200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T12:26:42.821 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:42 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:42.520+0000 7f40315cc200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T12:26:42.821 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:42 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:42.654+0000 7f40315cc200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T12:26:43.007 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:42.726+0000 7f90d6713200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T12:26:43.008 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:42.815+0000 7f90d6713200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T12:26:43.096 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:43.006+0000 7f40315cc200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T12:26:43.097 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:43.098+0000 7f40315cc200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T12:26:43.384 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:43.009+0000 7f90d6713200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:26:43.384 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:43.197+0000 7f90d6713200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T12:26:43.419 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:43.309+0000 7f40315cc200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:26:43.430 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:26:43.431 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":82,"fsid":"bc017092-e9d8-11ee-95cd-87774f69a715","created":"2024-03-24T12:20:18.712998+0000","modified":"2024-03-24T12:26:32.857662+0000","last_up_change":"2024-03-24T12:25:48.676565+0000","last_in_change":"2024-03-24T12:25:30.369664+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T12:23:41.491597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-24T12:26:01.465080+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-24T12:26:02.100021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T12:26:03.473862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T12:26:05.573828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T12:26:07.645083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0aa1c49b-7887-4f74-b03c-08cc0adec90b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6801","nonce":2514818464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":2514818464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":2514818464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6803","nonce":2514818464}]},"public_addr":"172.21.15.29:6801/2514818464","cluster_addr":"172.21.15.29:6802/2514818464","heartbeat_back_addr":"172.21.15.29:6804/2514818464","heartbeat_front_addr":"172.21.15.29:6803/2514818464","state":["exists","up"]},{"osd":1,"uuid":"99d8ef8f-eb59-4882-ab51-b398e6b53bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":2923396572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2923396572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2923396572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6807","nonce":2923396572}]},"public_addr":"172.21.15.29:6805/2923396572","cluster_addr":"172.21.15.29:6806/2923396572","heartbeat_back_addr":"172.21.15.29:6808/2923396572","heartbeat_front_addr":"172.21.15.29:6807/2923396572","state":["exists","up"]},{"osd":2,"uuid":"c3f83505-25ee-4217-b60a-081c36092e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":126032915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":126032915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":126032915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":126032915}]},"public_addr":"172.21.15.29:6809/126032915","cluster_addr":"172.21.15.29:6810/126032915","heartbeat_back_addr":"172.21.15.29:6812/126032915","heartbeat_front_addr":"172.21.15.29:6811/126032915","state":["exists","up"]},{"osd":3,"uuid":"f5e479a8-459a-4315-a808-711a7b84bb96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1080171824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1080171824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1080171824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":1080171824}]},"public_addr":"172.21.15.29:6813/1080171824","cluster_addr":"172.21.15.29:6814/1080171824","heartbeat_back_addr":"172.21.15.29:6816/1080171824","heartbeat_front_addr":"172.21.15.29:6815/1080171824","state":["exists","up"]},{"osd":4,"uuid":"5e420304-2798-4d11-9688-d0d89482008c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3690258869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":3690258869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":3690258869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3690258869}]},"public_addr":"172.21.15.190:6800/3690258869","cluster_addr":"172.21.15.190:6801/3690258869","heartbeat_back_addr":"172.21.15.190:6803/3690258869","heartbeat_front_addr":"172.21.15.190:6802/3690258869","state":["exists","up"]},{"osd":5,"uuid":"1213431a-652c-4cbf-b32a-36335430c666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":1327220488}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":1327220488}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":1327220488}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":1327220488}]},"public_addr":"172.21.15.190:6804/1327220488","cluster_addr":"172.21.15.190:6805/1327220488","heartbeat_back_addr":"172.21.15.190:6807/1327220488","heartbeat_front_addr":"172.21.15.190:6806/1327220488","state":["exists","up"]},{"osd":6,"uuid":"0712e511-0257-44f3-a82f-ee09ab957c0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1295064976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":1295064976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":1295064976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1295064976}]},"public_addr":"172.21.15.190:6808/1295064976","cluster_addr":"172.21.15.190:6809/1295064976","heartbeat_back_addr":"172.21.15.190:6811/1295064976","heartbeat_front_addr":"172.21.15.190:6810/1295064976","state":["exists","up"]},{"osd":7,"uuid":"7e36bde4-14d0-4e7e-9a82-2bb97859dbda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":110963949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":110963949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":110963949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":110963949}]},"public_addr":"172.21.15.190:6812/110963949","cluster_addr":"172.21.15.190:6813/110963949","heartbeat_back_addr":"172.21.15.190:6815/110963949","heartbeat_front_addr":"172.21.15.190:6814/110963949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:26:32.019647+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:36.630023+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:57.919767+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:26:20.595213+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:49.675888+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:26:15.790889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:26:32.516737+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:25:46.876664+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.12","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/4130171883":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/2929430696":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/327626505":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/3715448582":"2024-03-25T12:20:46.948976+0000","172.21.15.29:6800/2109454792":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/1631806715":"2024-03-25T12:20:46.948976+0000","172.21.15.29:6800/852212806":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/1823252077":"2024-03-25T12:21:15.151901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T12:26:43.737 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:43 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1160313748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:26:43.737 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:43.479+0000 7f90d6713200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:26:43.793 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:43 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1160313748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:26:43.793 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:43 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1160313748' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:26:43.794 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:43.503+0000 7f40315cc200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T12:26:44.058 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:43.795+0000 7f40315cc200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T12:26:44.058 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:44.060+0000 7f40315cc200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T12:26:44.134 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:43.739+0000 7f90d6713200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T12:26:44.134 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:43.974+0000 7f90d6713200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T12:26:44.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T12:26:44.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T12:26:44.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T12:26:44.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T12:26:44.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T12:26:44.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T12:26:44.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T12:26:44.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T12:26:44.346 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:44.304+0000 7f40315cc200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T12:26:44.883 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:44.889 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:44.890 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:44.890 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:44.891 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:44.895 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:26:45.134 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:44.763+0000 7f90d6713200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T12:26:45.634 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:45.332+0000 7f90d6713200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T12:26:45.634 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:45.430+0000 7f90d6713200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T12:26:45.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:45.408+0000 7f40315cc200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T12:26:45.967 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[41500]: Standby manager daemon x restarted 2024-03-24T12:26:45.967 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[41500]: Standby manager daemon x started 2024-03-24T12:26:45.967 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[41500]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T12:26:45.967 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[41500]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T12:26:45.967 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[41500]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T12:26:45.967 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[41500]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T12:26:45.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[29919]: Standby manager daemon x restarted 2024-03-24T12:26:45.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[29919]: Standby manager daemon x started 2024-03-24T12:26:45.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[29919]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T12:26:45.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[29919]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T12:26:45.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[29919]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T12:26:45.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:45 smithi029 ceph-mon[29919]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T12:26:46.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-mon[33748]: Standby manager daemon x restarted 2024-03-24T12:26:46.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-mon[33748]: Standby manager daemon x started 2024-03-24T12:26:46.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-mon[33748]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-24T12:26:46.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-mon[33748]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T12:26:46.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-mon[33748]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-24T12:26:46.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-mon[33748]: from='mgr.? 172.21.15.190:0/2392894008' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T12:26:46.135 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:45.645+0000 7f90d6713200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T12:26:46.135 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: [24/Mar/2024:12:26:45] ENGINE Bus STARTING 2024-03-24T12:26:46.135 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: CherryPy Checker: 2024-03-24T12:26:46.135 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: The Application mounted at '' has an empty config. 2024-03-24T12:26:46.135 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: 2024-03-24T12:26:46.135 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: [24/Mar/2024:12:26:45] ENGINE Serving on http://:::9283 2024-03-24T12:26:46.135 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:26:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x[35856]: [24/Mar/2024:12:26:45] ENGINE Bus STARTED 2024-03-24T12:26:46.228 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:46.019+0000 7f40315cc200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T12:26:46.228 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:46.121+0000 7f40315cc200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T12:26:46.522 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:46.351+0000 7f40315cc200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T12:26:46.523 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:26:46] ENGINE Bus STARTING 2024-03-24T12:26:46.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: Active manager daemon y restarted 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: Activating manager daemon y 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: mgrmap e19: y(active, starting, since 0.00880285s), standbys: x 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:26:46.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: Manager daemon y is now available 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:46.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:26:46.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T12:26:46.887 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T12:26:46.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: CherryPy Checker: 2024-03-24T12:26:46.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: The Application mounted at '' has an empty config. 2024-03-24T12:26:46.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:26:46.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:26:46] ENGINE Serving on http://:::9283 2024-03-24T12:26:46.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:26:46] ENGINE Bus STARTED 2024-03-24T12:26:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-24T12:26:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: Active manager daemon y restarted 2024-03-24T12:26:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: Activating manager daemon y 2024-03-24T12:26:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T12:26:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: mgrmap e19: y(active, starting, since 0.00880285s), standbys: x 2024-03-24T12:26:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:26:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:26:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:26:46.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T12:26:46.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: Manager daemon y is now available 2024-03-24T12:26:46.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:46.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:46.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:46.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:26:46.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T12:26:46.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T12:26:46.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-24T12:26:46.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: Active manager daemon y restarted 2024-03-24T12:26:46.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: Activating manager daemon y 2024-03-24T12:26:46.922 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: osdmap e83: 8 total, 8 up, 8 in 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: mgrmap e19: y(active, starting, since 0.00880285s), standbys: x 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T12:26:46.923 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: Manager daemon y is now available 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-24T12:26:46.924 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-24T12:26:47.634 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:26:47.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:26:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:26:48.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:48 smithi190 ceph-mon[33748]: mgrmap e20: y(active, since 1.01086s), standbys: x 2024-03-24T12:26:48.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:48 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:48.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:48 smithi190 ceph-mon[33748]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:48.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:48 smithi029 ceph-mon[29919]: mgrmap e20: y(active, since 1.01086s), standbys: x 2024-03-24T12:26:48.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:48 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:48.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:48 smithi029 ceph-mon[29919]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:48.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:48 smithi029 ceph-mon[41500]: mgrmap e20: y(active, since 1.01086s), standbys: x 2024-03-24T12:26:48.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:48 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:48.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:48 smithi029 ceph-mon[41500]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:49.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:49 smithi190 ceph-mon[33748]: [24/Mar/2024:12:26:47] ENGINE Bus STARTING 2024-03-24T12:26:49.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:49 smithi190 ceph-mon[33748]: [24/Mar/2024:12:26:48] ENGINE Serving on https://172.21.15.29:7150 2024-03-24T12:26:49.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:49 smithi190 ceph-mon[33748]: [24/Mar/2024:12:26:48] ENGINE Serving on http://172.21.15.29:8765 2024-03-24T12:26:49.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:49 smithi190 ceph-mon[33748]: [24/Mar/2024:12:26:48] ENGINE Bus STARTED 2024-03-24T12:26:49.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:49 smithi190 ceph-mon[33748]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:49.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:49 smithi029 ceph-mon[29919]: [24/Mar/2024:12:26:47] ENGINE Bus STARTING 2024-03-24T12:26:49.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:49 smithi029 ceph-mon[29919]: [24/Mar/2024:12:26:48] ENGINE Serving on https://172.21.15.29:7150 2024-03-24T12:26:49.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:49 smithi029 ceph-mon[29919]: [24/Mar/2024:12:26:48] ENGINE Serving on http://172.21.15.29:8765 2024-03-24T12:26:49.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:49 smithi029 ceph-mon[29919]: [24/Mar/2024:12:26:48] ENGINE Bus STARTED 2024-03-24T12:26:49.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:49 smithi029 ceph-mon[29919]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:49.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:49 smithi029 ceph-mon[41500]: [24/Mar/2024:12:26:47] ENGINE Bus STARTING 2024-03-24T12:26:49.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:49 smithi029 ceph-mon[41500]: [24/Mar/2024:12:26:48] ENGINE Serving on https://172.21.15.29:7150 2024-03-24T12:26:49.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:49 smithi029 ceph-mon[41500]: [24/Mar/2024:12:26:48] ENGINE Serving on http://172.21.15.29:8765 2024-03-24T12:26:49.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:49 smithi029 ceph-mon[41500]: [24/Mar/2024:12:26:48] ENGINE Bus STARTED 2024-03-24T12:26:49.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:49 smithi029 ceph-mon[41500]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:50.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:50 smithi190 ceph-mon[33748]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-24T12:26:50.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:50 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:50.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:50 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:50.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:50 smithi190 ceph-mon[33748]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T12:26:50.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:50 smithi029 ceph-mon[29919]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-24T12:26:50.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:50 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:50.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:50 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:50.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:50 smithi029 ceph-mon[29919]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T12:26:50.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:50 smithi029 ceph-mon[41500]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-24T12:26:50.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:50 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:50.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:50 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:50.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:50 smithi029 ceph-mon[41500]: osdmap e84: 8 total, 8 up, 8 in 2024-03-24T12:26:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[41500]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:52.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:52.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:52.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi190 to 3760M 2024-03-24T12:26:52.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:52.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[41500]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-24T12:26:52.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[29919]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:52.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:52.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:52.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi190 to 3760M 2024-03-24T12:26:52.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:52.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:51 smithi029 ceph-mon[29919]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-24T12:26:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:51 smithi190 ceph-mon[33748]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:26:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:51 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi190 to 3760M 2024-03-24T12:26:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:26:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:51 smithi190 ceph-mon[33748]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-24T12:26:54.165 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:53 smithi029 ceph-mon[41500]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:54.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:53 smithi029 ceph-mon[29919]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:53 smithi190 ceph-mon[33748]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail 2024-03-24T12:26:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:55 smithi029 ceph-mon[29919]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T12:26:56.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:55 smithi029 ceph-mon[41500]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T12:26:56.186 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:55 smithi190 ceph-mon[33748]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T12:26:56.646 INFO:teuthology.orchestra.run.smithi029.stdout:180388626457 2024-03-24T12:26:56.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T12:26:57.433 INFO:teuthology.orchestra.run.smithi029.stdout:257698037774 2024-03-24T12:26:57.434 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T12:26:57.634 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:26:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:26:57.662 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:26:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:26:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:26:57.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:57 smithi029 ceph-mon[29919]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T12:26:57.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:57 smithi029 ceph-mon[41500]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T12:26:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:57 smithi190 ceph-mon[33748]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-24T12:26:58.580 INFO:teuthology.orchestra.run.smithi029.stdout:55834574895 2024-03-24T12:26:58.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T12:26:59.121 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:58 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:59.121 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:58 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:26:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:00.031 INFO:teuthology.orchestra.run.smithi029.stdout:141733920798 2024-03-24T12:27:00.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T12:27:00.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:26:59 smithi029 ceph-mon[41500]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T12:27:00.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:26:59 smithi029 ceph-mon[29919]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T12:27:00.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:26:59 smithi190 ceph-mon[33748]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-24T12:27:01.103 INFO:teuthology.orchestra.run.smithi029.stdout:34359738421 2024-03-24T12:27:01.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T12:27:01.787 INFO:teuthology.orchestra.run.smithi029.stdout:77309411370 2024-03-24T12:27:01.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T12:27:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:01 smithi029 ceph-mon[41500]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T12:27:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:01 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:01 smithi029 ceph-mon[29919]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T12:27:02.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:01 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:02.291 INFO:teuthology.orchestra.run.smithi029.stdout:219043332117 2024-03-24T12:27:02.292 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T12:27:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:01 smithi190 ceph-mon[33748]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-24T12:27:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:01 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:03.352 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:03.353 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:03.359 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:03.361 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:03.606 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:04.028 INFO:teuthology.orchestra.run.smithi029.stdout:115964117028 2024-03-24T12:27:04.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T12:27:04.049 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:04.049 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:04.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:03 smithi029 ceph-mon[29919]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T12:27:04.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:03 smithi029 ceph-mon[41500]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T12:27:04.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:03 smithi190 ceph-mon[33748]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T12:27:06.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:05 smithi029 ceph-mon[29919]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T12:27:06.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:05 smithi029 ceph-mon[41500]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T12:27:06.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:05 smithi190 ceph-mon[33748]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T12:27:07.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[41500]: 4.7 deep-scrub starts 2024-03-24T12:27:07.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[41500]: 4.7 deep-scrub ok 2024-03-24T12:27:07.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[41500]: 3.11 scrub starts 2024-03-24T12:27:07.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[41500]: 3.11 scrub ok 2024-03-24T12:27:07.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[41500]: 3.10 scrub starts 2024-03-24T12:27:07.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[41500]: 3.10 scrub ok 2024-03-24T12:27:07.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[29919]: 4.7 deep-scrub starts 2024-03-24T12:27:07.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[29919]: 4.7 deep-scrub ok 2024-03-24T12:27:07.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[29919]: 3.11 scrub starts 2024-03-24T12:27:07.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[29919]: 3.11 scrub ok 2024-03-24T12:27:07.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[29919]: 3.10 scrub starts 2024-03-24T12:27:07.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:06 smithi029 ceph-mon[29919]: 3.10 scrub ok 2024-03-24T12:27:07.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:06 smithi190 ceph-mon[33748]: 4.7 deep-scrub starts 2024-03-24T12:27:07.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:06 smithi190 ceph-mon[33748]: 4.7 deep-scrub ok 2024-03-24T12:27:07.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:06 smithi190 ceph-mon[33748]: 3.11 scrub starts 2024-03-24T12:27:07.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:06 smithi190 ceph-mon[33748]: 3.11 scrub ok 2024-03-24T12:27:07.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:06 smithi190 ceph-mon[33748]: 3.10 scrub starts 2024-03-24T12:27:07.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:06 smithi190 ceph-mon[33748]: 3.10 scrub ok 2024-03-24T12:27:07.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:27:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:27:07.915 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:27:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:27:08.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[29919]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T12:27:08.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:27:08.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:27:08.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[29919]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T12:27:08.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[29919]: 3.12 scrub starts 2024-03-24T12:27:08.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[29919]: 3.12 scrub ok 2024-03-24T12:27:08.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:27:08.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:27:08.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[41500]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T12:27:08.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:27:08.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:27:08.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[41500]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T12:27:08.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[41500]: 3.12 scrub starts 2024-03-24T12:27:08.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[41500]: 3.12 scrub ok 2024-03-24T12:27:08.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:27:08.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:07 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:27:08.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:07 smithi190 ceph-mon[33748]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T12:27:08.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:07 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:27:08.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:07 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:27:08.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:07 smithi190 ceph-mon[33748]: osdmap e85: 8 total, 8 up, 8 in 2024-03-24T12:27:08.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:07 smithi190 ceph-mon[33748]: 3.12 scrub starts 2024-03-24T12:27:08.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:07 smithi190 ceph-mon[33748]: 3.12 scrub ok 2024-03-24T12:27:08.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:07 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:27:08.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:07 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:27:09.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:09.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:08 smithi190 ceph-mon[33748]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T12:27:09.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:08 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:27:09.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:08 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:27:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:08 smithi029 ceph-mon[29919]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T12:27:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:08 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:27:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:08 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:27:09.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:09.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:08 smithi029 ceph-mon[41500]: osdmap e86: 8 total, 8 up, 8 in 2024-03-24T12:27:09.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:08 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:27:09.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:08 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:27:10.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:09 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:27:10.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:09 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:27:10.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:09 smithi029 ceph-mon[41500]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:10.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:09 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:27:10.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:09 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:27:10.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:09 smithi029 ceph-mon[29919]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:10.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:09 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:27:10.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:09 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:27:10.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:09 smithi190 ceph-mon[33748]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:11.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:10 smithi029 ceph-mon[29919]: 3.14 scrub starts 2024-03-24T12:27:11.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:10 smithi029 ceph-mon[29919]: 3.14 scrub ok 2024-03-24T12:27:11.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:10 smithi029 ceph-mon[29919]: 3.c deep-scrub starts 2024-03-24T12:27:11.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:10 smithi029 ceph-mon[29919]: 3.c deep-scrub ok 2024-03-24T12:27:11.104 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:10 smithi029 ceph-mon[41500]: 3.14 scrub starts 2024-03-24T12:27:11.104 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:10 smithi029 ceph-mon[41500]: 3.14 scrub ok 2024-03-24T12:27:11.105 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:10 smithi029 ceph-mon[41500]: 3.c deep-scrub starts 2024-03-24T12:27:11.105 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:10 smithi029 ceph-mon[41500]: 3.c deep-scrub ok 2024-03-24T12:27:11.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:10 smithi190 ceph-mon[33748]: 3.14 scrub starts 2024-03-24T12:27:11.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:10 smithi190 ceph-mon[33748]: 3.14 scrub ok 2024-03-24T12:27:11.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:10 smithi190 ceph-mon[33748]: 3.c deep-scrub starts 2024-03-24T12:27:11.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:10 smithi190 ceph-mon[33748]: 3.c deep-scrub ok 2024-03-24T12:27:11.430 INFO:teuthology.orchestra.run.smithi029.stdout:141733920801 2024-03-24T12:27:12.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:27:12.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:27:12.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: 3.1c deep-scrub starts 2024-03-24T12:27:12.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: 3.1c deep-scrub ok 2024-03-24T12:27:12.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:12.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: 5.0 scrub starts 2024-03-24T12:27:12.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: 5.0 scrub ok 2024-03-24T12:27:12.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:27:12.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:27:12.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T12:27:12.073 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:27:12.073 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:27:12.073 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3327660733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T12:27:12.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:27:12.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:27:12.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: 3.1c deep-scrub starts 2024-03-24T12:27:12.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: 3.1c deep-scrub ok 2024-03-24T12:27:12.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:12.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: 5.0 scrub starts 2024-03-24T12:27:12.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: 5.0 scrub ok 2024-03-24T12:27:12.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:27:12.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:27:12.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T12:27:12.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:27:12.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:27:12.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:11 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3327660733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T12:27:12.174 INFO:teuthology.orchestra.run.smithi029.stdout:55834574898 2024-03-24T12:27:12.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: 3.1c deep-scrub starts 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: 3.1c deep-scrub ok 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: 5.0 scrub starts 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: 5.0 scrub ok 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: osdmap e87: 8 total, 8 up, 8 in 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:27:12.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:11 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3327660733' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T12:27:13.124 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:12 smithi029 ceph-mon[41500]: 4.0 scrub starts 2024-03-24T12:27:13.124 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:12 smithi029 ceph-mon[41500]: 4.0 scrub ok 2024-03-24T12:27:13.124 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:12 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2047642153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T12:27:13.124 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:12 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:27:13.124 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:12 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:27:13.125 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:12 smithi029 ceph-mon[29919]: 4.0 scrub starts 2024-03-24T12:27:13.125 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:12 smithi029 ceph-mon[29919]: 4.0 scrub ok 2024-03-24T12:27:13.125 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:12 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2047642153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T12:27:13.125 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:12 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:27:13.125 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:12 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:27:13.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:12 smithi190 ceph-mon[33748]: 4.0 scrub starts 2024-03-24T12:27:13.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:12 smithi190 ceph-mon[33748]: 4.0 scrub ok 2024-03-24T12:27:13.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:12 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2047642153' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T12:27:13.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:12 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:27:13.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:12 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:27:13.867 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920798 got 141733920801 for osd.4 2024-03-24T12:27:13.867 DEBUG:teuthology.parallel:result is None 2024-03-24T12:27:14.100 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[41500]: 6.7 scrub starts 2024-03-24T12:27:14.100 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[41500]: 6.7 scrub ok 2024-03-24T12:27:14.100 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[41500]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T12:27:14.100 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[41500]: 3.18 deep-scrub starts 2024-03-24T12:27:14.100 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[41500]: 3.18 deep-scrub ok 2024-03-24T12:27:14.100 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:27:14.100 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:27:14.100 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:14.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[29919]: 6.7 scrub starts 2024-03-24T12:27:14.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[29919]: 6.7 scrub ok 2024-03-24T12:27:14.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[29919]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T12:27:14.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[29919]: 3.18 deep-scrub starts 2024-03-24T12:27:14.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[29919]: 3.18 deep-scrub ok 2024-03-24T12:27:14.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:27:14.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:27:14.101 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:13 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:14.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:13 smithi190 ceph-mon[33748]: 6.7 scrub starts 2024-03-24T12:27:14.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:13 smithi190 ceph-mon[33748]: 6.7 scrub ok 2024-03-24T12:27:14.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:13 smithi190 ceph-mon[33748]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T12:27:14.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:13 smithi190 ceph-mon[33748]: 3.18 deep-scrub starts 2024-03-24T12:27:14.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:13 smithi190 ceph-mon[33748]: 3.18 deep-scrub ok 2024-03-24T12:27:14.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:13 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:27:14.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:13 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:27:14.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:13 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:14.533 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574895 got 55834574898 for osd.1 2024-03-24T12:27:14.533 DEBUG:teuthology.parallel:result is None 2024-03-24T12:27:15.198 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:15.290 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:14 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:15.290 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:14 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:27:15.290 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:14 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:27:15.290 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:14 smithi029 ceph-mon[41500]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:27:15.290 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:14 smithi029 ceph-mon[41500]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T12:27:15.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:14 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:15.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:14 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:27:15.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:14 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:27:15.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:14 smithi029 ceph-mon[29919]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:27:15.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:14 smithi029 ceph-mon[29919]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T12:27:15.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:14 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:15.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:14 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:27:15.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:14 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:27:15.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:14 smithi190 ceph-mon[33748]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:27:15.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:14 smithi190 ceph-mon[33748]: osdmap e88: 8 total, 8 up, 8 in 2024-03-24T12:27:15.439 INFO:teuthology.orchestra.run.smithi029.stdout:34359738424 2024-03-24T12:27:16.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:16.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi029 to 3761M 2024-03-24T12:27:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:27:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:27:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-24T12:27:16.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-24T12:27:16.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3406677087' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T12:27:16.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: 4.10 scrub starts 2024-03-24T12:27:16.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: 4.10 scrub ok 2024-03-24T12:27:16.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:27:16.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:27:16.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:27:16.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:16 smithi190 ceph-mon[33748]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:27:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi029 to 3761M 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3406677087' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: 4.10 scrub starts 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: 4.10 scrub ok 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:27:16.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[41500]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi029 to 3761M 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: Updating smithi029:/etc/ceph/ceph.conf 2024-03-24T12:27:16.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: Updating smithi190:/etc/ceph/ceph.conf 2024-03-24T12:27:16.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3406677087' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T12:27:16.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: 4.10 scrub starts 2024-03-24T12:27:16.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: 4.10 scrub ok 2024-03-24T12:27:16.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:27:16.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T12:27:16.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: Updating smithi190:/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:27:16.672 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:16 smithi029 ceph-mon[29919]: Updating smithi029:/etc/ceph/ceph.client.admin.keyring 2024-03-24T12:27:16.821 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738421 got 34359738424 for osd.0 2024-03-24T12:27:16.822 DEBUG:teuthology.parallel:result is None 2024-03-24T12:27:16.900 INFO:teuthology.orchestra.run.smithi029.stdout:257698037778 2024-03-24T12:27:17.474 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:27:17.474 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:27:17.474 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: 4.15 deep-scrub starts 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: 4.15 deep-scrub ok 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: Deploying daemon alertmanager.a on smithi029 2024-03-24T12:27:17.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:17.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2489186615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T12:27:17.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:27:17.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: 4.11 scrub ok 2024-03-24T12:27:17.476 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:17.476 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:27:17.476 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: 4.15 deep-scrub starts 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: 4.15 deep-scrub ok 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.477 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: Deploying daemon alertmanager.a on smithi029 2024-03-24T12:27:17.478 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:17.478 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2489186615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T12:27:17.478 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:27:17.478 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: 4.11 scrub ok 2024-03-24T12:27:17.478 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:17.556 INFO:teuthology.orchestra.run.smithi029.stdout:180388626461 2024-03-24T12:27:17.634 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: Updating smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: Updating smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: 4.15 deep-scrub starts 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: 4.15 deep-scrub ok 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:17.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: Deploying daemon alertmanager.a on smithi029 2024-03-24T12:27:17.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:17.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2489186615' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T12:27:17.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:27:17.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: 4.11 scrub ok 2024-03-24T12:27:17.636 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:17.801 INFO:teuthology.orchestra.run.smithi029.stdout:77309411373 2024-03-24T12:27:17.801 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:27:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:27:18.367 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037774 got 257698037778 for osd.7 2024-03-24T12:27:18.367 DEBUG:teuthology.parallel:result is None 2024-03-24T12:27:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:18 smithi190 ceph-mon[33748]: 4.13 deep-scrub starts 2024-03-24T12:27:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:18 smithi190 ceph-mon[33748]: 4.13 deep-scrub ok 2024-03-24T12:27:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:18 smithi190 ceph-mon[33748]: 4.8 scrub starts 2024-03-24T12:27:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:18 smithi190 ceph-mon[33748]: 4.8 scrub ok 2024-03-24T12:27:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:18 smithi190 ceph-mon[33748]: 5.1a deep-scrub starts 2024-03-24T12:27:18.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:18 smithi190 ceph-mon[33748]: 5.1a deep-scrub ok 2024-03-24T12:27:18.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:18.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:18 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1477488759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T12:27:18.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:18 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4247864965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T12:27:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[29919]: 4.13 deep-scrub starts 2024-03-24T12:27:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[29919]: 4.13 deep-scrub ok 2024-03-24T12:27:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[29919]: 4.8 scrub starts 2024-03-24T12:27:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[29919]: 4.8 scrub ok 2024-03-24T12:27:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[29919]: 5.1a deep-scrub starts 2024-03-24T12:27:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[29919]: 5.1a deep-scrub ok 2024-03-24T12:27:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1477488759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T12:27:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4247864965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T12:27:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[41500]: 4.13 deep-scrub starts 2024-03-24T12:27:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[41500]: 4.13 deep-scrub ok 2024-03-24T12:27:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[41500]: 4.8 scrub starts 2024-03-24T12:27:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[41500]: 4.8 scrub ok 2024-03-24T12:27:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[41500]: 5.1a deep-scrub starts 2024-03-24T12:27:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[41500]: 5.1a deep-scrub ok 2024-03-24T12:27:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1477488759' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T12:27:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:18 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4247864965' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T12:27:19.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:19 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:27:19.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:19 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:27:19.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:19 smithi190 ceph-mon[33748]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:19.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:19 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:27:19.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:19 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:27:19.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:19 smithi029 ceph-mon[41500]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:19.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:19 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:27:19.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:19 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:27:19.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:19 smithi029 ceph-mon[29919]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:20.376 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626457 got 180388626461 for osd.5 2024-03-24T12:27:20.377 DEBUG:teuthology.parallel:result is None 2024-03-24T12:27:21.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:20 smithi190 ceph-mon[33748]: 5.6 scrub starts 2024-03-24T12:27:21.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:20 smithi190 ceph-mon[33748]: 5.6 scrub ok 2024-03-24T12:27:21.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:20 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:27:21.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:20 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:27:21.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:20 smithi029 ceph-mon[29919]: 5.6 scrub starts 2024-03-24T12:27:21.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:20 smithi029 ceph-mon[29919]: 5.6 scrub ok 2024-03-24T12:27:21.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:20 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:27:21.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:20 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:27:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:20 smithi029 ceph-mon[41500]: 5.6 scrub starts 2024-03-24T12:27:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:20 smithi029 ceph-mon[41500]: 5.6 scrub ok 2024-03-24T12:27:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:20 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:27:21.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:20 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:27:22.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:21 smithi190 ceph-mon[33748]: 5.e scrub starts 2024-03-24T12:27:22.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:21 smithi190 ceph-mon[33748]: 5.e scrub ok 2024-03-24T12:27:22.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:21 smithi190 ceph-mon[33748]: 3.1d scrub starts 2024-03-24T12:27:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:21 smithi190 ceph-mon[33748]: 3.1d scrub ok 2024-03-24T12:27:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:21 smithi190 ceph-mon[33748]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:27:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:21 smithi190 ceph-mon[33748]: 5.13 deep-scrub starts 2024-03-24T12:27:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:21 smithi190 ceph-mon[33748]: 5.13 deep-scrub ok 2024-03-24T12:27:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:21 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:27:22.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:21 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:27:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[29919]: 5.e scrub starts 2024-03-24T12:27:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[29919]: 5.e scrub ok 2024-03-24T12:27:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[29919]: 3.1d scrub starts 2024-03-24T12:27:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[29919]: 3.1d scrub ok 2024-03-24T12:27:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[29919]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:27:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[29919]: 5.13 deep-scrub starts 2024-03-24T12:27:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[29919]: 5.13 deep-scrub ok 2024-03-24T12:27:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:27:22.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:27:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[41500]: 5.e scrub starts 2024-03-24T12:27:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[41500]: 5.e scrub ok 2024-03-24T12:27:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[41500]: 3.1d scrub starts 2024-03-24T12:27:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[41500]: 3.1d scrub ok 2024-03-24T12:27:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[41500]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:27:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[41500]: 5.13 deep-scrub starts 2024-03-24T12:27:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[41500]: 5.13 deep-scrub ok 2024-03-24T12:27:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:27:22.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:21 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:27:22.593 INFO:teuthology.orchestra.run.smithi029.stdout:219043332121 2024-03-24T12:27:23.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: 6.19 deep-scrub starts 2024-03-24T12:27:23.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: 6.19 deep-scrub ok 2024-03-24T12:27:23.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: 4.14 scrub starts 2024-03-24T12:27:23.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: 4.14 scrub ok 2024-03-24T12:27:23.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:27:23.134 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: 5.d scrub starts 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: 5.d scrub ok 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1395130634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: 6.19 deep-scrub starts 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: 6.19 deep-scrub ok 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: 4.14 scrub starts 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: 4.14 scrub ok 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:27:23.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:27:23.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: 5.d scrub starts 2024-03-24T12:27:23.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: 5.d scrub ok 2024-03-24T12:27:23.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:27:23.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:27:23.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T12:27:23.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:22 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1395130634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T12:27:23.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: 6.19 deep-scrub starts 2024-03-24T12:27:23.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: 6.19 deep-scrub ok 2024-03-24T12:27:23.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: 4.14 scrub starts 2024-03-24T12:27:23.136 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: 4.14 scrub ok 2024-03-24T12:27:23.137 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:27:23.137 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:27:23.137 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: 5.d scrub starts 2024-03-24T12:27:23.137 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: 5.d scrub ok 2024-03-24T12:27:23.137 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:27:23.137 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:27:23.137 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: osdmap e89: 8 total, 8 up, 8 in 2024-03-24T12:27:23.137 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:22 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1395130634' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T12:27:23.781 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411370 got 77309411373 for osd.2 2024-03-24T12:27:23.782 DEBUG:teuthology.parallel:result is None 2024-03-24T12:27:23.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:27:23.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:27:23.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[41500]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:23.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:27:23.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:27:23.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[41500]: 6.18 deep-scrub starts 2024-03-24T12:27:23.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[41500]: 6.18 deep-scrub ok 2024-03-24T12:27:23.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:27:23.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:27:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:27:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:27:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[29919]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:27:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:27:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[29919]: 6.18 deep-scrub starts 2024-03-24T12:27:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[29919]: 6.18 deep-scrub ok 2024-03-24T12:27:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:27:23.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:23 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:27:23.979 INFO:teuthology.orchestra.run.smithi029.stdout:115964117032 2024-03-24T12:27:24.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:23 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:27:24.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:23 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:27:24.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:23 smithi190 ceph-mon[33748]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:24.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:23 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:27:24.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:23 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:27:24.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:23 smithi190 ceph-mon[33748]: 6.18 deep-scrub starts 2024-03-24T12:27:24.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:23 smithi190 ceph-mon[33748]: 6.18 deep-scrub ok 2024-03-24T12:27:24.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:23 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:27:24.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:23 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:27:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:27:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:27:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[41500]: 5.7 scrub starts 2024-03-24T12:27:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[41500]: 5.7 scrub ok 2024-03-24T12:27:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[41500]: 5.2 deep-scrub starts 2024-03-24T12:27:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[41500]: 5.2 deep-scrub ok 2024-03-24T12:27:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[41500]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3201075144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[41500]: 4.2 scrub starts 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[41500]: 4.2 scrub ok 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[29919]: 5.7 scrub starts 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[29919]: 5.7 scrub ok 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[29919]: 5.2 deep-scrub starts 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[29919]: 5.2 deep-scrub ok 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[29919]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T12:27:25.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3201075144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T12:27:25.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[29919]: 4.2 scrub starts 2024-03-24T12:27:25.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:24 smithi029 ceph-mon[29919]: 4.2 scrub ok 2024-03-24T12:27:25.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:24 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:27:25.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:24 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:27:25.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:24 smithi190 ceph-mon[33748]: 5.7 scrub starts 2024-03-24T12:27:25.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:24 smithi190 ceph-mon[33748]: 5.7 scrub ok 2024-03-24T12:27:25.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:24 smithi190 ceph-mon[33748]: 5.2 deep-scrub starts 2024-03-24T12:27:25.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:24 smithi190 ceph-mon[33748]: 5.2 deep-scrub ok 2024-03-24T12:27:25.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:24 smithi190 ceph-mon[33748]: osdmap e90: 8 total, 8 up, 8 in 2024-03-24T12:27:25.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:24 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3201075144' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T12:27:25.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:24 smithi190 ceph-mon[33748]: 4.2 scrub starts 2024-03-24T12:27:25.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:24 smithi190 ceph-mon[33748]: 4.2 scrub ok 2024-03-24T12:27:25.400 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332117 got 219043332121 for osd.6 2024-03-24T12:27:25.400 DEBUG:teuthology.parallel:result is None 2024-03-24T12:27:25.939 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:27:25.939 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:27:25.939 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[41500]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[41500]: 6.6 deep-scrub starts 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[41500]: 6.6 deep-scrub ok 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[41500]: 6.4 scrub starts 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[41500]: 6.4 scrub ok 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[29919]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[29919]: 6.6 deep-scrub starts 2024-03-24T12:27:25.940 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[29919]: 6.6 deep-scrub ok 2024-03-24T12:27:25.941 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:27:25.941 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:27:25.941 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[29919]: 6.4 scrub starts 2024-03-24T12:27:25.941 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:25 smithi029 ceph-mon[29919]: 6.4 scrub ok 2024-03-24T12:27:25.972 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117028 got 115964117032 for osd.3 2024-03-24T12:27:25.973 DEBUG:teuthology.parallel:result is None 2024-03-24T12:27:25.973 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T12:27:25.973 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:27:26.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:25 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:27:26.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:25 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:27:26.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:25 smithi190 ceph-mon[33748]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:26.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:25 smithi190 ceph-mon[33748]: 6.6 deep-scrub starts 2024-03-24T12:27:26.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:25 smithi190 ceph-mon[33748]: 6.6 deep-scrub ok 2024-03-24T12:27:26.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:25 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:27:26.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:25 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:27:26.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:25 smithi190 ceph-mon[33748]: 6.4 scrub starts 2024-03-24T12:27:26.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:25 smithi190 ceph-mon[33748]: 6.4 scrub ok 2024-03-24T12:27:27.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:26 smithi029 ceph-mon[29919]: 6.1e scrub starts 2024-03-24T12:27:27.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:26 smithi029 ceph-mon[29919]: 6.1e scrub ok 2024-03-24T12:27:27.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:26 smithi029 ceph-mon[29919]: 4.19 deep-scrub starts 2024-03-24T12:27:27.084 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:26 smithi029 ceph-mon[29919]: 4.19 deep-scrub ok 2024-03-24T12:27:27.084 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:26 smithi029 ceph-mon[41500]: 6.1e scrub starts 2024-03-24T12:27:27.084 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:26 smithi029 ceph-mon[41500]: 6.1e scrub ok 2024-03-24T12:27:27.084 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:26 smithi029 ceph-mon[41500]: 4.19 deep-scrub starts 2024-03-24T12:27:27.084 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:26 smithi029 ceph-mon[41500]: 4.19 deep-scrub ok 2024-03-24T12:27:27.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:26 smithi190 ceph-mon[33748]: 6.1e scrub starts 2024-03-24T12:27:27.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:26 smithi190 ceph-mon[33748]: 6.1e scrub ok 2024-03-24T12:27:27.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:26 smithi190 ceph-mon[33748]: 4.19 deep-scrub starts 2024-03-24T12:27:27.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:26 smithi190 ceph-mon[33748]: 4.19 deep-scrub ok 2024-03-24T12:27:27.565 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:27.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:27:27.900 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:27:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:27:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: 3.5 scrub starts 2024-03-24T12:27:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: 3.5 scrub ok 2024-03-24T12:27:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: 6.11 deep-scrub starts 2024-03-24T12:27:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: 6.11 deep-scrub ok 2024-03-24T12:27:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: 3.1 scrub starts 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: 3.1 scrub ok 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: 3.5 scrub starts 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: 3.5 scrub ok 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: 6.11 deep-scrub starts 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: 6.11 deep-scrub ok 2024-03-24T12:27:28.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:27:28.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:27:28.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: 3.1 scrub starts 2024-03-24T12:27:28.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: 3.1 scrub ok 2024-03-24T12:27:28.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:27:28.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:27 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:27:28.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: 3.5 scrub starts 2024-03-24T12:27:28.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: 3.5 scrub ok 2024-03-24T12:27:28.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:28.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: 6.11 deep-scrub starts 2024-03-24T12:27:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: 6.11 deep-scrub ok 2024-03-24T12:27:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:27:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:27:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: 3.1 scrub starts 2024-03-24T12:27:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: 3.1 scrub ok 2024-03-24T12:27:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:27:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:27 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:27:29.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:29.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[29919]: 6.14 deep-scrub starts 2024-03-24T12:27:29.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[29919]: 6.14 deep-scrub ok 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[29919]: 3.1b deep-scrub starts 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[29919]: 3.1b deep-scrub ok 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[41500]: 6.14 deep-scrub starts 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[41500]: 6.14 deep-scrub ok 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[41500]: 3.1b deep-scrub starts 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[41500]: 3.1b deep-scrub ok 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:27:29.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:28 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:27:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:28 smithi190 ceph-mon[33748]: 6.14 deep-scrub starts 2024-03-24T12:27:29.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:28 smithi190 ceph-mon[33748]: 6.14 deep-scrub ok 2024-03-24T12:27:29.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:28 smithi190 ceph-mon[33748]: 3.1b deep-scrub starts 2024-03-24T12:27:29.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:28 smithi190 ceph-mon[33748]: 3.1b deep-scrub ok 2024-03-24T12:27:29.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:28 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:27:29.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:28 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:27:30.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:29 smithi029 ceph-mon[29919]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:30.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:29 smithi029 ceph-mon[29919]: 3.2 scrub starts 2024-03-24T12:27:30.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:29 smithi029 ceph-mon[29919]: 3.2 scrub ok 2024-03-24T12:27:30.054 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:29 smithi029 ceph-mon[29919]: 5.12 scrub starts 2024-03-24T12:27:30.055 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:29 smithi029 ceph-mon[29919]: 5.12 scrub ok 2024-03-24T12:27:30.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:29 smithi029 ceph-mon[41500]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:30.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:29 smithi029 ceph-mon[41500]: 3.2 scrub starts 2024-03-24T12:27:30.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:29 smithi029 ceph-mon[41500]: 3.2 scrub ok 2024-03-24T12:27:30.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:29 smithi029 ceph-mon[41500]: 5.12 scrub starts 2024-03-24T12:27:30.055 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:29 smithi029 ceph-mon[41500]: 5.12 scrub ok 2024-03-24T12:27:30.120 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:27:30.122 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:27:30.341 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:30 smithi029 podman[79785]: 2024-03-24 12:27:30.152197351 +0000 UTC m=+0.211644724 container create 28b4d8a961099913ae6effbb628288a788d351cbdaa1c7ad2a330f79001f6c30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:27:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:29 smithi190 ceph-mon[33748]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:29 smithi190 ceph-mon[33748]: 3.2 scrub starts 2024-03-24T12:27:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:29 smithi190 ceph-mon[33748]: 3.2 scrub ok 2024-03-24T12:27:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:29 smithi190 ceph-mon[33748]: 5.12 scrub starts 2024-03-24T12:27:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:29 smithi190 ceph-mon[33748]: 5.12 scrub ok 2024-03-24T12:27:30.944 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:30 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:27:30.944 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:30 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:27:30.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:30 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:27:30.944 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:30 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:27:30.973 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-24T12:27:28.388807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1011,"num_read_kb":857,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":622,"ondisk_log_size":622,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236896,"kb_used_data":8252,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724320,"statfs":{"total":767960285184,"available":767717703680,"internally_reserved":0,"allocated":8450048,"data_stored":4661866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001641"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887551+0000","last_change":"2024-03-24T12:26:09.350829+0000","last_active":"2024-03-24T12:27:23.887551+0000","last_peered":"2024-03-24T12:27:23.887551+0000","last_clean":"2024-03-24T12:27:23.887551+0000","last_became_active":"2024-03-24T12:26:09.350521+0000","last_became_peered":"2024-03-24T12:26:09.350521+0000","last_unstale":"2024-03-24T12:27:23.887551+0000","last_undegraded":"2024-03-24T12:27:23.887551+0000","last_fullsized":"2024-03-24T12:27:23.887551+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.019266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.136629+0000","last_change":"2024-03-24T12:26:07.335814+0000","last_active":"2024-03-24T12:27:24.136629+0000","last_peered":"2024-03-24T12:27:24.136629+0000","last_clean":"2024-03-24T12:27:24.136629+0000","last_became_active":"2024-03-24T12:26:07.335517+0000","last_became_peered":"2024-03-24T12:26:07.335517+0000","last_unstale":"2024-03-24T12:27:24.136629+0000","last_undegraded":"2024-03-24T12:27:24.136629+0000","last_fullsized":"2024-03-24T12:27:24.136629+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:34.813926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889332+0000","last_change":"2024-03-24T12:26:03.171110+0000","last_active":"2024-03-24T12:27:23.889332+0000","last_peered":"2024-03-24T12:27:23.889332+0000","last_clean":"2024-03-24T12:27:23.889332+0000","last_became_active":"2024-03-24T12:26:03.170822+0000","last_became_peered":"2024-03-24T12:26:03.170822+0000","last_unstale":"2024-03-24T12:27:23.889332+0000","last_undegraded":"2024-03-24T12:27:23.889332+0000","last_fullsized":"2024-03-24T12:27:23.889332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:31.894081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.141423+0000","last_change":"2024-03-24T12:27:24.141489+0000","last_active":"2024-03-24T12:27:24.141423+0000","last_peered":"2024-03-24T12:27:24.141423+0000","last_clean":"2024-03-24T12:27:24.141423+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:27:24.141423+0000","last_undegraded":"2024-03-24T12:27:24.141423+0000","last_fullsized":"2024-03-24T12:27:24.141423+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:24.141325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.141325+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.796727+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887179+0000","last_change":"2024-03-24T12:26:09.646438+0000","last_active":"2024-03-24T12:27:23.887179+0000","last_peered":"2024-03-24T12:27:23.887179+0000","last_clean":"2024-03-24T12:27:23.887179+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:27:23.887179+0000","last_undegraded":"2024-03-24T12:27:23.887179+0000","last_fullsized":"2024-03-24T12:27:23.887179+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:25.350589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887385+0000","last_change":"2024-03-24T12:26:07.335524+0000","last_active":"2024-03-24T12:27:23.887385+0000","last_peered":"2024-03-24T12:27:23.887385+0000","last_clean":"2024-03-24T12:27:23.887385+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:27:23.887385+0000","last_undegraded":"2024-03-24T12:27:23.887385+0000","last_fullsized":"2024-03-24T12:27:23.887385+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888488+0000","last_change":"2024-03-24T12:26:03.164752+0000","last_active":"2024-03-24T12:27:23.888488+0000","last_peered":"2024-03-24T12:27:23.888488+0000","last_clean":"2024-03-24T12:27:23.888488+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:27:23.888488+0000","last_undegraded":"2024-03-24T12:27:23.888488+0000","last_fullsized":"2024-03-24T12:27:23.888488+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.612593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"90'22","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:25.396603+0000","last_change":"2024-03-24T12:27:25.396689+0000","last_active":"2024-03-24T12:27:25.396603+0000","last_peered":"2024-03-24T12:27:25.396603+0000","last_clean":"2024-03-24T12:27:25.396603+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:27:25.396603+0000","last_undegraded":"2024-03-24T12:27:25.396603+0000","last_fullsized":"2024-03-24T12:27:25.396603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.396514+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.161411+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886854+0000","last_change":"2024-03-24T12:27:23.289755+0000","last_active":"2024-03-24T12:27:23.886854+0000","last_peered":"2024-03-24T12:27:23.886854+0000","last_clean":"2024-03-24T12:27:23.886854+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:27:23.886854+0000","last_undegraded":"2024-03-24T12:27:23.886854+0000","last_fullsized":"2024-03-24T12:27:23.886854+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.289666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.684236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372511+0000","last_change":"2024-03-24T12:26:07.331880+0000","last_active":"2024-03-24T12:27:24.372511+0000","last_peered":"2024-03-24T12:27:24.372511+0000","last_clean":"2024-03-24T12:27:24.372511+0000","last_became_active":"2024-03-24T12:26:07.331562+0000","last_became_peered":"2024-03-24T12:26:07.331562+0000","last_unstale":"2024-03-24T12:27:24.372511+0000","last_undegraded":"2024-03-24T12:27:24.372511+0000","last_fullsized":"2024-03-24T12:27:24.372511+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372688+0000","last_change":"2024-03-24T12:27:20.255011+0000","last_active":"2024-03-24T12:27:24.372688+0000","last_peered":"2024-03-24T12:27:24.372688+0000","last_clean":"2024-03-24T12:27:24.372688+0000","last_became_active":"2024-03-24T12:26:03.161544+0000","last_became_peered":"2024-03-24T12:26:03.161544+0000","last_unstale":"2024-03-24T12:27:24.372688+0000","last_undegraded":"2024-03-24T12:27:24.372688+0000","last_fullsized":"2024-03-24T12:27:24.372688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:20.254898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.254898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.151491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887903+0000","last_change":"2024-03-24T12:26:05.318208+0000","last_active":"2024-03-24T12:27:23.887903+0000","last_peered":"2024-03-24T12:27:23.887903+0000","last_clean":"2024-03-24T12:27:23.887903+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:27:23.887903+0000","last_undegraded":"2024-03-24T12:27:23.887903+0000","last_fullsized":"2024-03-24T12:27:23.887903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.433393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372248+0000","last_change":"2024-03-24T12:27:21.260071+0000","last_active":"2024-03-24T12:27:24.372248+0000","last_peered":"2024-03-24T12:27:24.372248+0000","last_clean":"2024-03-24T12:27:24.372248+0000","last_became_active":"2024-03-24T12:26:09.350721+0000","last_became_peered":"2024-03-24T12:26:09.350721+0000","last_unstale":"2024-03-24T12:27:24.372248+0000","last_undegraded":"2024-03-24T12:27:24.372248+0000","last_fullsized":"2024-03-24T12:27:24.372248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.259962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.801612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437152+0000","last_change":"2024-03-24T12:27:17.365794+0000","last_active":"2024-03-24T12:27:24.437152+0000","last_peered":"2024-03-24T12:27:24.437152+0000","last_clean":"2024-03-24T12:27:24.437152+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:27:24.437152+0000","last_undegraded":"2024-03-24T12:27:24.437152+0000","last_fullsized":"2024-03-24T12:27:24.437152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.365696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.873148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372612+0000","last_change":"2024-03-24T12:27:10.251781+0000","last_active":"2024-03-24T12:27:24.372612+0000","last_peered":"2024-03-24T12:27:24.372612+0000","last_clean":"2024-03-24T12:27:24.372612+0000","last_became_active":"2024-03-24T12:26:03.159997+0000","last_became_peered":"2024-03-24T12:26:03.159997+0000","last_unstale":"2024-03-24T12:27:24.372612+0000","last_undegraded":"2024-03-24T12:27:24.372612+0000","last_fullsized":"2024-03-24T12:27:24.372612+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.251686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.902316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887583+0000","last_change":"2024-03-24T12:26:05.322947+0000","last_active":"2024-03-24T12:27:23.887583+0000","last_peered":"2024-03-24T12:27:23.887583+0000","last_clean":"2024-03-24T12:27:23.887583+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:27:23.887583+0000","last_undegraded":"2024-03-24T12:27:23.887583+0000","last_fullsized":"2024-03-24T12:27:23.887583+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:25.164535+0000","last_change":"2024-03-24T12:27:25.164561+0000","last_active":"2024-03-24T12:27:25.164535+0000","last_peered":"2024-03-24T12:27:25.164535+0000","last_clean":"2024-03-24T12:27:25.164535+0000","last_became_active":"2024-03-24T12:26:09.643671+0000","last_became_peered":"2024-03-24T12:26:09.643671+0000","last_unstale":"2024-03-24T12:27:25.164535+0000","last_undegraded":"2024-03-24T12:27:25.164535+0000","last_fullsized":"2024-03-24T12:27:25.164535+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:25.164460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.164460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.284324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.137048+0000","last_change":"2024-03-24T12:26:07.330308+0000","last_active":"2024-03-24T12:27:24.137048+0000","last_peered":"2024-03-24T12:27:24.137048+0000","last_clean":"2024-03-24T12:27:24.137048+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:27:24.137048+0000","last_undegraded":"2024-03-24T12:27:24.137048+0000","last_fullsized":"2024-03-24T12:27:24.137048+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889180+0000","last_change":"2024-03-24T12:26:03.166440+0000","last_active":"2024-03-24T12:27:23.889180+0000","last_peered":"2024-03-24T12:27:23.889180+0000","last_clean":"2024-03-24T12:27:23.889180+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:27:23.889180+0000","last_undegraded":"2024-03-24T12:27:23.889180+0000","last_fullsized":"2024-03-24T12:27:23.889180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478184+0000","last_change":"2024-03-24T12:26:05.322833+0000","last_active":"2024-03-24T12:27:24.478184+0000","last_peered":"2024-03-24T12:27:24.478184+0000","last_clean":"2024-03-24T12:27:24.478184+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:27:24.478184+0000","last_undegraded":"2024-03-24T12:27:24.478184+0000","last_fullsized":"2024-03-24T12:27:24.478184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.619352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888275+0000","last_change":"2024-03-24T12:26:09.644272+0000","last_active":"2024-03-24T12:27:23.888275+0000","last_peered":"2024-03-24T12:27:23.888275+0000","last_clean":"2024-03-24T12:27:23.888275+0000","last_became_active":"2024-03-24T12:26:09.643738+0000","last_became_peered":"2024-03-24T12:26:09.643738+0000","last_unstale":"2024-03-24T12:27:23.888275+0000","last_undegraded":"2024-03-24T12:27:23.888275+0000","last_fullsized":"2024-03-24T12:27:23.888275+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887737+0000","last_change":"2024-03-24T12:26:07.333844+0000","last_active":"2024-03-24T12:27:23.887737+0000","last_peered":"2024-03-24T12:27:23.887737+0000","last_clean":"2024-03-24T12:27:23.887737+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:27:23.887737+0000","last_undegraded":"2024-03-24T12:27:23.887737+0000","last_fullsized":"2024-03-24T12:27:23.887737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887942+0000","last_change":"2024-03-24T12:26:03.160687+0000","last_active":"2024-03-24T12:27:23.887942+0000","last_peered":"2024-03-24T12:27:23.887942+0000","last_clean":"2024-03-24T12:27:23.887942+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:27:23.887942+0000","last_undegraded":"2024-03-24T12:27:23.887942+0000","last_fullsized":"2024-03-24T12:27:23.887942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.827198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887683+0000","last_change":"2024-03-24T12:26:05.321693+0000","last_active":"2024-03-24T12:27:23.887683+0000","last_peered":"2024-03-24T12:27:23.887683+0000","last_clean":"2024-03-24T12:27:23.887683+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:27:23.887683+0000","last_undegraded":"2024-03-24T12:27:23.887683+0000","last_fullsized":"2024-03-24T12:27:23.887683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437511+0000","last_change":"2024-03-24T12:26:09.344099+0000","last_active":"2024-03-24T12:27:24.437511+0000","last_peered":"2024-03-24T12:27:24.437511+0000","last_clean":"2024-03-24T12:27:24.437511+0000","last_became_active":"2024-03-24T12:26:09.343786+0000","last_became_peered":"2024-03-24T12:26:09.343786+0000","last_unstale":"2024-03-24T12:27:24.437511+0000","last_undegraded":"2024-03-24T12:27:24.437511+0000","last_fullsized":"2024-03-24T12:27:24.437511+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.139873+0000","last_change":"2024-03-24T12:26:07.335552+0000","last_active":"2024-03-24T12:27:24.139873+0000","last_peered":"2024-03-24T12:27:24.139873+0000","last_clean":"2024-03-24T12:27:24.139873+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:27:24.139873+0000","last_undegraded":"2024-03-24T12:27:24.139873+0000","last_fullsized":"2024-03-24T12:27:24.139873+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:35.440267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.136707+0000","last_change":"2024-03-24T12:26:03.164759+0000","last_active":"2024-03-24T12:27:24.136707+0000","last_peered":"2024-03-24T12:27:24.136707+0000","last_clean":"2024-03-24T12:27:24.136707+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:27:24.136707+0000","last_undegraded":"2024-03-24T12:27:24.136707+0000","last_fullsized":"2024-03-24T12:27:24.136707+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888827+0000","last_change":"2024-03-24T12:27:11.269339+0000","last_active":"2024-03-24T12:27:23.888827+0000","last_peered":"2024-03-24T12:27:23.888827+0000","last_clean":"2024-03-24T12:27:23.888827+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:27:23.888827+0000","last_undegraded":"2024-03-24T12:27:23.888827+0000","last_fullsized":"2024-03-24T12:27:23.888827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:11.269181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:11.269181+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:34.970841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.137202+0000","last_change":"2024-03-24T12:26:09.348873+0000","last_active":"2024-03-24T12:27:24.137202+0000","last_peered":"2024-03-24T12:27:24.137202+0000","last_clean":"2024-03-24T12:27:24.137202+0000","last_became_active":"2024-03-24T12:26:09.348355+0000","last_became_peered":"2024-03-24T12:26:09.348355+0000","last_unstale":"2024-03-24T12:27:24.137202+0000","last_undegraded":"2024-03-24T12:27:24.137202+0000","last_fullsized":"2024-03-24T12:27:24.137202+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:33.087921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889095+0000","last_change":"2024-03-24T12:26:07.331932+0000","last_active":"2024-03-24T12:27:23.889095+0000","last_peered":"2024-03-24T12:27:23.889095+0000","last_clean":"2024-03-24T12:27:23.889095+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:27:23.889095+0000","last_undegraded":"2024-03-24T12:27:23.889095+0000","last_fullsized":"2024-03-24T12:27:23.889095+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.308109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886870+0000","last_change":"2024-03-24T12:27:12.437790+0000","last_active":"2024-03-24T12:27:23.886870+0000","last_peered":"2024-03-24T12:27:23.886870+0000","last_clean":"2024-03-24T12:27:23.886870+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:27:23.886870+0000","last_undegraded":"2024-03-24T12:27:23.886870+0000","last_fullsized":"2024-03-24T12:27:23.886870+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.437708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:25.841012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:26.763391+0000","last_change":"2024-03-24T12:27:26.763428+0000","last_active":"2024-03-24T12:27:26.763391+0000","last_peered":"2024-03-24T12:27:26.763391+0000","last_clean":"2024-03-24T12:27:26.763391+0000","last_became_active":"2024-03-24T12:26:05.317587+0000","last_became_peered":"2024-03-24T12:26:05.317587+0000","last_unstale":"2024-03-24T12:27:26.763391+0000","last_undegraded":"2024-03-24T12:27:26.763391+0000","last_fullsized":"2024-03-24T12:27:26.763391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:26.763317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.763317+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.728888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437135+0000","last_change":"2024-03-24T12:26:09.345188+0000","last_active":"2024-03-24T12:27:24.437135+0000","last_peered":"2024-03-24T12:27:24.437135+0000","last_clean":"2024-03-24T12:27:24.437135+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:27:24.437135+0000","last_undegraded":"2024-03-24T12:27:24.437135+0000","last_fullsized":"2024-03-24T12:27:24.437135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:37.908175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.138974+0000","last_change":"2024-03-24T12:26:07.335668+0000","last_active":"2024-03-24T12:27:24.138974+0000","last_peered":"2024-03-24T12:27:24.138974+0000","last_clean":"2024-03-24T12:27:24.138974+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:27:24.138974+0000","last_undegraded":"2024-03-24T12:27:24.138974+0000","last_fullsized":"2024-03-24T12:27:24.138974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.115036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889353+0000","last_change":"2024-03-24T12:26:03.171104+0000","last_active":"2024-03-24T12:27:23.889353+0000","last_peered":"2024-03-24T12:27:23.889353+0000","last_clean":"2024-03-24T12:27:23.889353+0000","last_became_active":"2024-03-24T12:26:03.170395+0000","last_became_peered":"2024-03-24T12:26:03.170395+0000","last_unstale":"2024-03-24T12:27:23.889353+0000","last_undegraded":"2024-03-24T12:27:23.889353+0000","last_fullsized":"2024-03-24T12:27:23.889353+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:25.060953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887019+0000","last_change":"2024-03-24T12:27:15.467117+0000","last_active":"2024-03-24T12:27:23.887019+0000","last_peered":"2024-03-24T12:27:23.887019+0000","last_clean":"2024-03-24T12:27:23.887019+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:27:23.887019+0000","last_undegraded":"2024-03-24T12:27:23.887019+0000","last_fullsized":"2024-03-24T12:27:23.887019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:27:15.466966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:15.466966+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:23.327159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886994+0000","last_change":"2024-03-24T12:26:09.646151+0000","last_active":"2024-03-24T12:27:23.886994+0000","last_peered":"2024-03-24T12:27:23.886994+0000","last_clean":"2024-03-24T12:27:23.886994+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:27:23.886994+0000","last_undegraded":"2024-03-24T12:27:23.886994+0000","last_fullsized":"2024-03-24T12:27:23.886994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.849429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437414+0000","last_change":"2024-03-24T12:27:21.335557+0000","last_active":"2024-03-24T12:27:24.437414+0000","last_peered":"2024-03-24T12:27:24.437414+0000","last_clean":"2024-03-24T12:27:24.437414+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:27:24.437414+0000","last_undegraded":"2024-03-24T12:27:24.437414+0000","last_fullsized":"2024-03-24T12:27:24.437414+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:21.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.789829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373007+0000","last_change":"2024-03-24T12:26:03.166070+0000","last_active":"2024-03-24T12:27:24.373007+0000","last_peered":"2024-03-24T12:27:24.373007+0000","last_clean":"2024-03-24T12:27:24.373007+0000","last_became_active":"2024-03-24T12:26:03.165083+0000","last_became_peered":"2024-03-24T12:26:03.165083+0000","last_unstale":"2024-03-24T12:27:24.373007+0000","last_undegraded":"2024-03-24T12:27:24.373007+0000","last_fullsized":"2024-03-24T12:27:24.373007+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887607+0000","last_change":"2024-03-24T12:27:16.497904+0000","last_active":"2024-03-24T12:27:23.887607+0000","last_peered":"2024-03-24T12:27:23.887607+0000","last_clean":"2024-03-24T12:27:23.887607+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:27:23.887607+0000","last_undegraded":"2024-03-24T12:27:23.887607+0000","last_fullsized":"2024-03-24T12:27:23.887607+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:16.497755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.497755+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.894323+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886620+0000","last_change":"2024-03-24T12:26:09.351371+0000","last_active":"2024-03-24T12:27:23.886620+0000","last_peered":"2024-03-24T12:27:23.886620+0000","last_clean":"2024-03-24T12:27:23.886620+0000","last_became_active":"2024-03-24T12:26:09.350656+0000","last_became_peered":"2024-03-24T12:26:09.350656+0000","last_unstale":"2024-03-24T12:27:23.886620+0000","last_undegraded":"2024-03-24T12:27:23.886620+0000","last_fullsized":"2024-03-24T12:27:23.886620+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887306+0000","last_change":"2024-03-24T12:27:20.458514+0000","last_active":"2024-03-24T12:27:23.887306+0000","last_peered":"2024-03-24T12:27:23.887306+0000","last_clean":"2024-03-24T12:27:23.887306+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:27:23.887306+0000","last_undegraded":"2024-03-24T12:27:23.887306+0000","last_fullsized":"2024-03-24T12:27:23.887306+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.458419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:24.491860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437613+0000","last_change":"2024-03-24T12:27:14.329516+0000","last_active":"2024-03-24T12:27:24.437613+0000","last_peered":"2024-03-24T12:27:24.437613+0000","last_clean":"2024-03-24T12:27:24.437613+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:27:24.437613+0000","last_undegraded":"2024-03-24T12:27:24.437613+0000","last_fullsized":"2024-03-24T12:27:24.437613+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:14.329357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:14.329357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:21.223400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372275+0000","last_change":"2024-03-24T12:26:16.882174+0000","last_active":"2024-03-24T12:27:24.372275+0000","last_peered":"2024-03-24T12:27:24.372275+0000","last_clean":"2024-03-24T12:27:24.372275+0000","last_became_active":"2024-03-24T12:26:16.874170+0000","last_became_peered":"2024-03-24T12:26:16.874170+0000","last_unstale":"2024-03-24T12:27:24.372275+0000","last_undegraded":"2024-03-24T12:27:24.372275+0000","last_fullsized":"2024-03-24T12:27:24.372275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:26.431453+0000","last_change":"2024-03-24T12:27:26.431474+0000","last_active":"2024-03-24T12:27:26.431453+0000","last_peered":"2024-03-24T12:27:26.431453+0000","last_clean":"2024-03-24T12:27:26.431453+0000","last_became_active":"2024-03-24T12:26:09.343296+0000","last_became_peered":"2024-03-24T12:26:09.343296+0000","last_unstale":"2024-03-24T12:27:26.431453+0000","last_undegraded":"2024-03-24T12:27:26.431453+0000","last_fullsized":"2024-03-24T12:27:26.431453+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.431382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.685011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.137320+0000","last_change":"2024-03-24T12:26:07.331137+0000","last_active":"2024-03-24T12:27:24.137320+0000","last_peered":"2024-03-24T12:27:24.137320+0000","last_clean":"2024-03-24T12:27:24.137320+0000","last_became_active":"2024-03-24T12:26:07.330811+0000","last_became_peered":"2024-03-24T12:26:07.330811+0000","last_unstale":"2024-03-24T12:27:24.137320+0000","last_undegraded":"2024-03-24T12:27:24.137320+0000","last_fullsized":"2024-03-24T12:27:24.137320+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:25.889786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888093+0000","last_change":"2024-03-24T12:27:09.137207+0000","last_active":"2024-03-24T12:27:23.888093+0000","last_peered":"2024-03-24T12:27:23.888093+0000","last_clean":"2024-03-24T12:27:23.888093+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:27:23.888093+0000","last_undegraded":"2024-03-24T12:27:23.888093+0000","last_fullsized":"2024-03-24T12:27:23.888093+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:09.137104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:09.137104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.329586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"88'16","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888125+0000","last_change":"2024-03-24T12:27:17.084272+0000","last_active":"2024-03-24T12:27:23.888125+0000","last_peered":"2024-03-24T12:27:23.888125+0000","last_clean":"2024-03-24T12:27:23.888125+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:27:23.888125+0000","last_undegraded":"2024-03-24T12:27:23.888125+0000","last_fullsized":"2024-03-24T12:27:23.888125+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.084148+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.541132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888504+0000","last_change":"2024-03-24T12:26:09.349851+0000","last_active":"2024-03-24T12:27:23.888504+0000","last_peered":"2024-03-24T12:27:23.888504+0000","last_clean":"2024-03-24T12:27:23.888504+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:27:23.888504+0000","last_undegraded":"2024-03-24T12:27:23.888504+0000","last_fullsized":"2024-03-24T12:27:23.888504+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.575676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372823+0000","last_change":"2024-03-24T12:26:07.330555+0000","last_active":"2024-03-24T12:27:24.372823+0000","last_peered":"2024-03-24T12:27:24.372823+0000","last_clean":"2024-03-24T12:27:24.372823+0000","last_became_active":"2024-03-24T12:26:07.329487+0000","last_became_peered":"2024-03-24T12:26:07.329487+0000","last_unstale":"2024-03-24T12:27:24.372823+0000","last_undegraded":"2024-03-24T12:27:24.372823+0000","last_fullsized":"2024-03-24T12:27:24.372823+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437318+0000","last_change":"2024-03-24T12:26:03.163577+0000","last_active":"2024-03-24T12:27:24.437318+0000","last_peered":"2024-03-24T12:27:24.437318+0000","last_clean":"2024-03-24T12:27:24.437318+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:27:24.437318+0000","last_undegraded":"2024-03-24T12:27:24.437318+0000","last_fullsized":"2024-03-24T12:27:24.437318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.200101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887407+0000","last_change":"2024-03-24T12:27:21.451811+0000","last_active":"2024-03-24T12:27:23.887407+0000","last_peered":"2024-03-24T12:27:23.887407+0000","last_clean":"2024-03-24T12:27:23.887407+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:27:23.887407+0000","last_undegraded":"2024-03-24T12:27:23.887407+0000","last_fullsized":"2024-03-24T12:27:23.887407+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:21.451717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.451717+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.819749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888742+0000","last_change":"2024-03-24T12:26:09.346362+0000","last_active":"2024-03-24T12:27:23.888742+0000","last_peered":"2024-03-24T12:27:23.888742+0000","last_clean":"2024-03-24T12:27:23.888742+0000","last_became_active":"2024-03-24T12:26:09.346062+0000","last_became_peered":"2024-03-24T12:26:09.346062+0000","last_unstale":"2024-03-24T12:27:23.888742+0000","last_undegraded":"2024-03-24T12:27:23.888742+0000","last_fullsized":"2024-03-24T12:27:23.888742+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888507+0000","last_change":"2024-03-24T12:26:07.338058+0000","last_active":"2024-03-24T12:27:23.888507+0000","last_peered":"2024-03-24T12:27:23.888507+0000","last_clean":"2024-03-24T12:27:23.888507+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:27:23.888507+0000","last_undegraded":"2024-03-24T12:27:23.888507+0000","last_fullsized":"2024-03-24T12:27:23.888507+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:31.504792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888704+0000","last_change":"2024-03-24T12:27:07.317120+0000","last_active":"2024-03-24T12:27:23.888704+0000","last_peered":"2024-03-24T12:27:23.888704+0000","last_clean":"2024-03-24T12:27:23.888704+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:27:23.888704+0000","last_undegraded":"2024-03-24T12:27:23.888704+0000","last_fullsized":"2024-03-24T12:27:23.888704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:07.317021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:07.317021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.086017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'13","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373017+0000","last_change":"2024-03-24T12:27:16.242881+0000","last_active":"2024-03-24T12:27:24.373017+0000","last_peered":"2024-03-24T12:27:24.373017+0000","last_clean":"2024-03-24T12:27:24.373017+0000","last_became_active":"2024-03-24T12:26:05.358213+0000","last_became_peered":"2024-03-24T12:26:05.358213+0000","last_unstale":"2024-03-24T12:27:24.373017+0000","last_undegraded":"2024-03-24T12:27:24.373017+0000","last_fullsized":"2024-03-24T12:27:24.373017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.242776+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.762237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478346+0000","last_change":"2024-03-24T12:26:09.343216+0000","last_active":"2024-03-24T12:27:24.478346+0000","last_peered":"2024-03-24T12:27:24.478346+0000","last_clean":"2024-03-24T12:27:24.478346+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:27:24.478346+0000","last_undegraded":"2024-03-24T12:27:24.478346+0000","last_fullsized":"2024-03-24T12:27:24.478346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:25.062739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888192+0000","last_change":"2024-03-24T12:26:07.343233+0000","last_active":"2024-03-24T12:27:23.888192+0000","last_peered":"2024-03-24T12:27:23.888192+0000","last_clean":"2024-03-24T12:27:23.888192+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:27:23.888192+0000","last_undegraded":"2024-03-24T12:27:23.888192+0000","last_fullsized":"2024-03-24T12:27:23.888192+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.890619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437199+0000","last_change":"2024-03-24T12:27:06.422517+0000","last_active":"2024-03-24T12:27:24.437199+0000","last_peered":"2024-03-24T12:27:24.437199+0000","last_clean":"2024-03-24T12:27:24.437199+0000","last_became_active":"2024-03-24T12:26:03.166149+0000","last_became_peered":"2024-03-24T12:26:03.166149+0000","last_unstale":"2024-03-24T12:27:24.437199+0000","last_undegraded":"2024-03-24T12:27:24.437199+0000","last_fullsized":"2024-03-24T12:27:24.437199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:06.422387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.422387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:12.269750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888930+0000","last_change":"2024-03-24T12:26:05.358122+0000","last_active":"2024-03-24T12:27:23.888930+0000","last_peered":"2024-03-24T12:27:23.888930+0000","last_clean":"2024-03-24T12:27:23.888930+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:27:23.888930+0000","last_undegraded":"2024-03-24T12:27:23.888930+0000","last_fullsized":"2024-03-24T12:27:23.888930+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437608+0000","last_change":"2024-03-24T12:26:09.646175+0000","last_active":"2024-03-24T12:27:24.437608+0000","last_peered":"2024-03-24T12:27:24.437608+0000","last_clean":"2024-03-24T12:27:24.437608+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:27:24.437608+0000","last_undegraded":"2024-03-24T12:27:24.437608+0000","last_fullsized":"2024-03-24T12:27:24.437608+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373084+0000","last_change":"2024-03-24T12:26:07.329432+0000","last_active":"2024-03-24T12:27:24.373084+0000","last_peered":"2024-03-24T12:27:24.373084+0000","last_clean":"2024-03-24T12:27:24.373084+0000","last_became_active":"2024-03-24T12:26:07.329117+0000","last_became_peered":"2024-03-24T12:26:07.329117+0000","last_unstale":"2024-03-24T12:27:24.373084+0000","last_undegraded":"2024-03-24T12:27:24.373084+0000","last_fullsized":"2024-03-24T12:27:24.373084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:25.438312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.138906+0000","last_change":"2024-03-24T12:27:06.584912+0000","last_active":"2024-03-24T12:27:24.138906+0000","last_peered":"2024-03-24T12:27:24.138906+0000","last_clean":"2024-03-24T12:27:24.138906+0000","last_became_active":"2024-03-24T12:26:03.161414+0000","last_became_peered":"2024-03-24T12:26:03.161414+0000","last_unstale":"2024-03-24T12:27:24.138906+0000","last_undegraded":"2024-03-24T12:27:24.138906+0000","last_fullsized":"2024-03-24T12:27:24.138906+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:06.584809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.584809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:16.866693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888362+0000","last_change":"2024-03-24T12:26:05.357244+0000","last_active":"2024-03-24T12:27:23.888362+0000","last_peered":"2024-03-24T12:27:23.888362+0000","last_clean":"2024-03-24T12:27:23.888362+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:27:23.888362+0000","last_undegraded":"2024-03-24T12:27:23.888362+0000","last_fullsized":"2024-03-24T12:27:23.888362+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:31.433340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372719+0000","last_change":"2024-03-24T12:26:09.643942+0000","last_active":"2024-03-24T12:27:24.372719+0000","last_peered":"2024-03-24T12:27:24.372719+0000","last_clean":"2024-03-24T12:27:24.372719+0000","last_became_active":"2024-03-24T12:26:09.643602+0000","last_became_peered":"2024-03-24T12:26:09.643602+0000","last_unstale":"2024-03-24T12:27:24.372719+0000","last_undegraded":"2024-03-24T12:27:24.372719+0000","last_fullsized":"2024-03-24T12:27:24.372719+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:34.205128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437939+0000","last_change":"2024-03-24T12:26:03.161676+0000","last_active":"2024-03-24T12:27:24.437939+0000","last_peered":"2024-03-24T12:27:24.437939+0000","last_clean":"2024-03-24T12:27:24.437939+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:27:24.437939+0000","last_undegraded":"2024-03-24T12:27:24.437939+0000","last_fullsized":"2024-03-24T12:27:24.437939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372273+0000","last_change":"2024-03-24T12:27:17.288726+0000","last_active":"2024-03-24T12:27:24.372273+0000","last_peered":"2024-03-24T12:27:24.372273+0000","last_clean":"2024-03-24T12:27:24.372273+0000","last_became_active":"2024-03-24T12:26:05.358217+0000","last_became_peered":"2024-03-24T12:26:05.358217+0000","last_unstale":"2024-03-24T12:27:24.372273+0000","last_undegraded":"2024-03-24T12:27:24.372273+0000","last_fullsized":"2024-03-24T12:27:24.372273+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:27:17.288616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.288616+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:31.276817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437451+0000","last_change":"2024-03-24T12:27:20.354464+0000","last_active":"2024-03-24T12:27:24.437451+0000","last_peered":"2024-03-24T12:27:24.437451+0000","last_clean":"2024-03-24T12:27:24.437451+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:27:24.437451+0000","last_undegraded":"2024-03-24T12:27:24.437451+0000","last_fullsized":"2024-03-24T12:27:24.437451+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:20.354367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.354367+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.013310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887122+0000","last_change":"2024-03-24T12:26:09.350769+0000","last_active":"2024-03-24T12:27:23.887122+0000","last_peered":"2024-03-24T12:27:23.887122+0000","last_clean":"2024-03-24T12:27:23.887122+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:27:23.887122+0000","last_undegraded":"2024-03-24T12:27:23.887122+0000","last_fullsized":"2024-03-24T12:27:23.887122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:38.324576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437855+0000","last_change":"2024-03-24T12:26:03.162567+0000","last_active":"2024-03-24T12:27:24.437855+0000","last_peered":"2024-03-24T12:27:24.437855+0000","last_clean":"2024-03-24T12:27:24.437855+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:27:24.437855+0000","last_undegraded":"2024-03-24T12:27:24.437855+0000","last_fullsized":"2024-03-24T12:27:24.437855+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:31.590674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888020+0000","last_change":"2024-03-24T12:27:23.131685+0000","last_active":"2024-03-24T12:27:23.888020+0000","last_peered":"2024-03-24T12:27:23.888020+0000","last_clean":"2024-03-24T12:27:23.888020+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:27:23.888020+0000","last_undegraded":"2024-03-24T12:27:23.888020+0000","last_fullsized":"2024-03-24T12:27:23.888020+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:23.131573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.131573+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.851864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.477937+0000","last_change":"2024-03-24T12:26:07.335582+0000","last_active":"2024-03-24T12:27:24.477937+0000","last_peered":"2024-03-24T12:27:24.477937+0000","last_clean":"2024-03-24T12:27:24.477937+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:27:24.477937+0000","last_undegraded":"2024-03-24T12:27:24.477937+0000","last_fullsized":"2024-03-24T12:27:24.477937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.841365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.436974+0000","last_change":"2024-03-24T12:26:09.339988+0000","last_active":"2024-03-24T12:27:24.436974+0000","last_peered":"2024-03-24T12:27:24.436974+0000","last_clean":"2024-03-24T12:27:24.436974+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:27:24.436974+0000","last_undegraded":"2024-03-24T12:27:24.436974+0000","last_fullsized":"2024-03-24T12:27:24.436974+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.803044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478137+0000","last_change":"2024-03-24T12:26:07.331773+0000","last_active":"2024-03-24T12:27:24.478137+0000","last_peered":"2024-03-24T12:27:24.478137+0000","last_clean":"2024-03-24T12:27:24.478137+0000","last_became_active":"2024-03-24T12:26:07.331416+0000","last_became_peered":"2024-03-24T12:26:07.331416+0000","last_unstale":"2024-03-24T12:27:24.478137+0000","last_undegraded":"2024-03-24T12:27:24.478137+0000","last_fullsized":"2024-03-24T12:27:24.478137+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.501587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437852+0000","last_change":"2024-03-24T12:26:03.165503+0000","last_active":"2024-03-24T12:27:24.437852+0000","last_peered":"2024-03-24T12:27:24.437852+0000","last_clean":"2024-03-24T12:27:24.437852+0000","last_became_active":"2024-03-24T12:26:03.165214+0000","last_became_peered":"2024-03-24T12:26:03.165214+0000","last_unstale":"2024-03-24T12:27:24.437852+0000","last_undegraded":"2024-03-24T12:27:24.437852+0000","last_fullsized":"2024-03-24T12:27:24.437852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.139487+0000","last_change":"2024-03-24T12:27:15.668713+0000","last_active":"2024-03-24T12:27:24.139487+0000","last_peered":"2024-03-24T12:27:24.139487+0000","last_clean":"2024-03-24T12:27:24.139487+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:27:24.139487+0000","last_undegraded":"2024-03-24T12:27:24.139487+0000","last_fullsized":"2024-03-24T12:27:24.139487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:27:15.668556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:15.668556+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:43.755371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886863+0000","last_change":"2024-03-24T12:26:09.350031+0000","last_active":"2024-03-24T12:27:23.886863+0000","last_peered":"2024-03-24T12:27:23.886863+0000","last_clean":"2024-03-24T12:27:23.886863+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:27:23.886863+0000","last_undegraded":"2024-03-24T12:27:23.886863+0000","last_fullsized":"2024-03-24T12:27:23.886863+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.187535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478091+0000","last_change":"2024-03-24T12:26:07.329850+0000","last_active":"2024-03-24T12:27:24.478091+0000","last_peered":"2024-03-24T12:27:24.478091+0000","last_clean":"2024-03-24T12:27:24.478091+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:27:24.478091+0000","last_undegraded":"2024-03-24T12:27:24.478091+0000","last_fullsized":"2024-03-24T12:27:24.478091+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.923438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372614+0000","last_change":"2024-03-24T12:27:09.234408+0000","last_active":"2024-03-24T12:27:24.372614+0000","last_peered":"2024-03-24T12:27:24.372614+0000","last_clean":"2024-03-24T12:27:24.372614+0000","last_became_active":"2024-03-24T12:26:03.162562+0000","last_became_peered":"2024-03-24T12:26:03.162562+0000","last_unstale":"2024-03-24T12:27:24.372614+0000","last_undegraded":"2024-03-24T12:27:24.372614+0000","last_fullsized":"2024-03-24T12:27:24.372614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:27:09.234299+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:17.263964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889082+0000","last_change":"2024-03-24T12:26:16.882489+0000","last_active":"2024-03-24T12:27:23.889082+0000","last_peered":"2024-03-24T12:27:23.889082+0000","last_clean":"2024-03-24T12:27:23.889082+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:27:23.889082+0000","last_undegraded":"2024-03-24T12:27:23.889082+0000","last_fullsized":"2024-03-24T12:27:23.889082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:28.651922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.439361+0000","last_change":"2024-03-24T12:27:24.439386+0000","last_active":"2024-03-24T12:27:24.439361+0000","last_peered":"2024-03-24T12:27:24.439361+0000","last_clean":"2024-03-24T12:27:24.439361+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:27:24.439361+0000","last_undegraded":"2024-03-24T12:27:24.439361+0000","last_fullsized":"2024-03-24T12:27:24.439361+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.439296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.495934+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372454+0000","last_change":"2024-03-24T12:26:07.318867+0000","last_active":"2024-03-24T12:27:24.372454+0000","last_peered":"2024-03-24T12:27:24.372454+0000","last_clean":"2024-03-24T12:27:24.372454+0000","last_became_active":"2024-03-24T12:26:03.165771+0000","last_became_peered":"2024-03-24T12:26:03.165771+0000","last_unstale":"2024-03-24T12:27:24.372454+0000","last_undegraded":"2024-03-24T12:27:24.372454+0000","last_fullsized":"2024-03-24T12:27:24.372454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.390729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887665+0000","last_change":"2024-03-24T12:26:03.161787+0000","last_active":"2024-03-24T12:27:23.887665+0000","last_peered":"2024-03-24T12:27:23.887665+0000","last_clean":"2024-03-24T12:27:23.887665+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:27:23.887665+0000","last_undegraded":"2024-03-24T12:27:23.887665+0000","last_fullsized":"2024-03-24T12:27:23.887665+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":118,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.137053+0000","last_change":"2024-03-24T12:26:05.322639+0000","last_active":"2024-03-24T12:27:24.137053+0000","last_peered":"2024-03-24T12:27:24.137053+0000","last_clean":"2024-03-24T12:27:24.137053+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:27:24.137053+0000","last_undegraded":"2024-03-24T12:27:24.137053+0000","last_fullsized":"2024-03-24T12:27:24.137053+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:28.944748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888397+0000","last_change":"2024-03-24T12:26:07.332154+0000","last_active":"2024-03-24T12:27:23.888397+0000","last_peered":"2024-03-24T12:27:23.888397+0000","last_clean":"2024-03-24T12:27:23.888397+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:27:23.888397+0000","last_undegraded":"2024-03-24T12:27:23.888397+0000","last_fullsized":"2024-03-24T12:27:23.888397+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.438013+0000","last_change":"2024-03-24T12:27:18.336245+0000","last_active":"2024-03-24T12:27:24.438013+0000","last_peered":"2024-03-24T12:27:24.438013+0000","last_clean":"2024-03-24T12:27:24.438013+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:27:24.438013+0000","last_undegraded":"2024-03-24T12:27:24.438013+0000","last_fullsized":"2024-03-24T12:27:24.438013+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:18.336147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:18.336147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.167902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.136626+0000","last_change":"2024-03-24T12:27:16.330188+0000","last_active":"2024-03-24T12:27:24.136626+0000","last_peered":"2024-03-24T12:27:24.136626+0000","last_clean":"2024-03-24T12:27:24.136626+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:27:24.136626+0000","last_undegraded":"2024-03-24T12:27:24.136626+0000","last_fullsized":"2024-03-24T12:27:24.136626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:16.330050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.330050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.775200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.614963+0000","last_change":"2024-03-24T12:27:24.614983+0000","last_active":"2024-03-24T12:27:24.614963+0000","last_peered":"2024-03-24T12:27:24.614963+0000","last_clean":"2024-03-24T12:27:24.614963+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:27:24.614963+0000","last_undegraded":"2024-03-24T12:27:24.614963+0000","last_fullsized":"2024-03-24T12:27:24.614963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.614896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.614896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.137138+0000","last_change":"2024-03-24T12:27:06.412196+0000","last_active":"2024-03-24T12:27:24.137138+0000","last_peered":"2024-03-24T12:27:24.137138+0000","last_clean":"2024-03-24T12:27:24.137138+0000","last_became_active":"2024-03-24T12:26:05.315639+0000","last_became_peered":"2024-03-24T12:26:05.315639+0000","last_unstale":"2024-03-24T12:27:24.137138+0000","last_undegraded":"2024-03-24T12:27:24.137138+0000","last_fullsized":"2024-03-24T12:27:24.137138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:27:06.412074+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:27:06.412074+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.412074+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.288814+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478340+0000","last_change":"2024-03-24T12:27:19.658489+0000","last_active":"2024-03-24T12:27:24.478340+0000","last_peered":"2024-03-24T12:27:24.478340+0000","last_clean":"2024-03-24T12:27:24.478340+0000","last_became_active":"2024-03-24T12:26:07.343010+0000","last_became_peered":"2024-03-24T12:26:07.343010+0000","last_unstale":"2024-03-24T12:27:24.478340+0000","last_undegraded":"2024-03-24T12:27:24.478340+0000","last_fullsized":"2024-03-24T12:27:24.478340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:19.658367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:19.658367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.033169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373000+0000","last_change":"2024-03-24T12:27:12.274502+0000","last_active":"2024-03-24T12:27:24.373000+0000","last_peered":"2024-03-24T12:27:24.373000+0000","last_clean":"2024-03-24T12:27:24.373000+0000","last_became_active":"2024-03-24T12:26:09.347821+0000","last_became_peered":"2024-03-24T12:26:09.347821+0000","last_unstale":"2024-03-24T12:27:24.373000+0000","last_undegraded":"2024-03-24T12:27:24.373000+0000","last_fullsized":"2024-03-24T12:27:24.373000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.274393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.274393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.457433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":318,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437077+0000","last_change":"2024-03-24T12:27:08.342835+0000","last_active":"2024-03-24T12:27:24.437077+0000","last_peered":"2024-03-24T12:27:24.437077+0000","last_clean":"2024-03-24T12:27:24.437077+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:27:24.437077+0000","last_undegraded":"2024-03-24T12:27:24.437077+0000","last_fullsized":"2024-03-24T12:27:24.437077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:27:08.342708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:27:08.342708+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:25.314000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887255+0000","last_change":"2024-03-24T12:26:03.166742+0000","last_active":"2024-03-24T12:27:23.887255+0000","last_peered":"2024-03-24T12:27:23.887255+0000","last_clean":"2024-03-24T12:27:23.887255+0000","last_became_active":"2024-03-24T12:26:03.166467+0000","last_became_peered":"2024-03-24T12:26:03.166467+0000","last_unstale":"2024-03-24T12:27:23.887255+0000","last_undegraded":"2024-03-24T12:27:23.887255+0000","last_fullsized":"2024-03-24T12:27:23.887255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":100,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.139603+0000","last_change":"2024-03-24T12:27:21.712724+0000","last_active":"2024-03-24T12:27:24.139603+0000","last_peered":"2024-03-24T12:27:24.139603+0000","last_clean":"2024-03-24T12:27:24.139603+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:27:24.139603+0000","last_undegraded":"2024-03-24T12:27:24.139603+0000","last_fullsized":"2024-03-24T12:27:24.139603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:27:21.712602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.712602+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.700708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437699+0000","last_change":"2024-03-24T12:26:07.337813+0000","last_active":"2024-03-24T12:27:24.437699+0000","last_peered":"2024-03-24T12:27:24.437699+0000","last_clean":"2024-03-24T12:27:24.437699+0000","last_became_active":"2024-03-24T12:26:07.337548+0000","last_became_peered":"2024-03-24T12:26:07.337548+0000","last_unstale":"2024-03-24T12:27:24.437699+0000","last_undegraded":"2024-03-24T12:27:24.437699+0000","last_fullsized":"2024-03-24T12:27:24.437699+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.316558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:25.409425+0000","last_change":"2024-03-24T12:27:25.409449+0000","last_active":"2024-03-24T12:27:25.409425+0000","last_peered":"2024-03-24T12:27:25.409425+0000","last_clean":"2024-03-24T12:27:25.409425+0000","last_became_active":"2024-03-24T12:26:09.349435+0000","last_became_peered":"2024-03-24T12:26:09.349435+0000","last_unstale":"2024-03-24T12:27:25.409425+0000","last_undegraded":"2024-03-24T12:27:25.409425+0000","last_fullsized":"2024-03-24T12:27:25.409425+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:25.409346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.409346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.841693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:27.333389+0000","last_change":"2024-03-24T12:27:27.333417+0000","last_active":"2024-03-24T12:27:27.333389+0000","last_peered":"2024-03-24T12:27:27.333389+0000","last_clean":"2024-03-24T12:27:27.333389+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:27:27.333389+0000","last_undegraded":"2024-03-24T12:27:27.333389+0000","last_fullsized":"2024-03-24T12:27:27.333389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:27.333274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.333274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.779264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":152,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:25.506718+0000","last_change":"2024-03-24T12:27:16.322773+0000","last_active":"2024-03-24T12:27:25.506718+0000","last_peered":"2024-03-24T12:27:25.506718+0000","last_clean":"2024-03-24T12:27:25.506718+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:27:25.506718+0000","last_undegraded":"2024-03-24T12:27:25.506718+0000","last_fullsized":"2024-03-24T12:27:25.506718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:27:16.322655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.322655+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:22.799348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":77,"num_read_kb":72,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888625+0000","last_change":"2024-03-24T12:26:05.314629+0000","last_active":"2024-03-24T12:27:23.888625+0000","last_peered":"2024-03-24T12:27:23.888625+0000","last_clean":"2024-03-24T12:27:23.888625+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:27:23.888625+0000","last_undegraded":"2024-03-24T12:27:23.888625+0000","last_fullsized":"2024-03-24T12:27:23.888625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373242+0000","last_change":"2024-03-24T12:27:23.276910+0000","last_active":"2024-03-24T12:27:24.373242+0000","last_peered":"2024-03-24T12:27:24.373242+0000","last_clean":"2024-03-24T12:27:24.373242+0000","last_became_active":"2024-03-24T12:26:07.326457+0000","last_became_peered":"2024-03-24T12:26:07.326457+0000","last_unstale":"2024-03-24T12:27:24.373242+0000","last_undegraded":"2024-03-24T12:27:24.373242+0000","last_fullsized":"2024-03-24T12:27:24.373242+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.276789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.276789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.022163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.136565+0000","last_change":"2024-03-24T12:26:09.644098+0000","last_active":"2024-03-24T12:27:24.136565+0000","last_peered":"2024-03-24T12:27:24.136565+0000","last_clean":"2024-03-24T12:27:24.136565+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:27:24.136565+0000","last_undegraded":"2024-03-24T12:27:24.136565+0000","last_fullsized":"2024-03-24T12:27:24.136565+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:33.107640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.136995+0000","last_change":"2024-03-24T12:26:03.167532+0000","last_active":"2024-03-24T12:27:24.136995+0000","last_peered":"2024-03-24T12:27:24.136995+0000","last_clean":"2024-03-24T12:27:24.136995+0000","last_became_active":"2024-03-24T12:26:03.167244+0000","last_became_peered":"2024-03-24T12:26:03.167244+0000","last_unstale":"2024-03-24T12:27:24.136995+0000","last_undegraded":"2024-03-24T12:27:24.136995+0000","last_fullsized":"2024-03-24T12:27:24.136995+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.503467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889108+0000","last_change":"2024-03-24T12:26:05.357149+0000","last_active":"2024-03-24T12:27:23.889108+0000","last_peered":"2024-03-24T12:27:23.889108+0000","last_clean":"2024-03-24T12:27:23.889108+0000","last_became_active":"2024-03-24T12:26:05.356745+0000","last_became_peered":"2024-03-24T12:26:05.356745+0000","last_unstale":"2024-03-24T12:27:23.889108+0000","last_undegraded":"2024-03-24T12:27:23.889108+0000","last_fullsized":"2024-03-24T12:27:23.889108+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:32.355424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.139406+0000","last_change":"2024-03-24T12:27:23.763463+0000","last_active":"2024-03-24T12:27:24.139406+0000","last_peered":"2024-03-24T12:27:24.139406+0000","last_clean":"2024-03-24T12:27:24.139406+0000","last_became_active":"2024-03-24T12:26:07.337303+0000","last_became_peered":"2024-03-24T12:26:07.337303+0000","last_unstale":"2024-03-24T12:27:24.139406+0000","last_undegraded":"2024-03-24T12:27:24.139406+0000","last_fullsized":"2024-03-24T12:27:24.139406+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.763326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:36.340669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886634+0000","last_change":"2024-03-24T12:26:09.341274+0000","last_active":"2024-03-24T12:27:23.886634+0000","last_peered":"2024-03-24T12:27:23.886634+0000","last_clean":"2024-03-24T12:27:23.886634+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:27:23.886634+0000","last_undegraded":"2024-03-24T12:27:23.886634+0000","last_fullsized":"2024-03-24T12:27:23.886634+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.825455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:26.215424+0000","last_change":"2024-03-24T12:27:26.215446+0000","last_active":"2024-03-24T12:27:26.215424+0000","last_peered":"2024-03-24T12:27:26.215424+0000","last_clean":"2024-03-24T12:27:26.215424+0000","last_became_active":"2024-03-24T12:26:03.168184+0000","last_became_peered":"2024-03-24T12:26:03.168184+0000","last_unstale":"2024-03-24T12:27:26.215424+0000","last_undegraded":"2024-03-24T12:27:26.215424+0000","last_fullsized":"2024-03-24T12:27:26.215424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:26.215348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.215348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.879847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.375005+0000","last_change":"2024-03-24T12:27:24.375037+0000","last_active":"2024-03-24T12:27:24.375005+0000","last_peered":"2024-03-24T12:27:24.375005+0000","last_clean":"2024-03-24T12:27:24.375005+0000","last_became_active":"2024-03-24T12:26:05.316739+0000","last_became_peered":"2024-03-24T12:26:05.316739+0000","last_unstale":"2024-03-24T12:27:24.375005+0000","last_undegraded":"2024-03-24T12:27:24.375005+0000","last_fullsized":"2024-03-24T12:27:24.375005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:24.374935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.374935+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:47.083756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888978+0000","last_change":"2024-03-24T12:26:07.336444+0000","last_active":"2024-03-24T12:27:23.888978+0000","last_peered":"2024-03-24T12:27:23.888978+0000","last_clean":"2024-03-24T12:27:23.888978+0000","last_became_active":"2024-03-24T12:26:07.336080+0000","last_became_peered":"2024-03-24T12:26:07.336080+0000","last_unstale":"2024-03-24T12:27:23.888978+0000","last_undegraded":"2024-03-24T12:27:23.888978+0000","last_fullsized":"2024-03-24T12:27:23.888978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437580+0000","last_change":"2024-03-24T12:26:09.645942+0000","last_active":"2024-03-24T12:27:24.437580+0000","last_peered":"2024-03-24T12:27:24.437580+0000","last_clean":"2024-03-24T12:27:24.437580+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:27:24.437580+0000","last_undegraded":"2024-03-24T12:27:24.437580+0000","last_fullsized":"2024-03-24T12:27:24.437580+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888729+0000","last_change":"2024-03-24T12:27:12.294371+0000","last_active":"2024-03-24T12:27:23.888729+0000","last_peered":"2024-03-24T12:27:23.888729+0000","last_clean":"2024-03-24T12:27:23.888729+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:27:23.888729+0000","last_undegraded":"2024-03-24T12:27:23.888729+0000","last_fullsized":"2024-03-24T12:27:23.888729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:12.294292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.294292+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.602110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887735+0000","last_change":"2024-03-24T12:26:05.315771+0000","last_active":"2024-03-24T12:27:23.887735+0000","last_peered":"2024-03-24T12:27:23.887735+0000","last_clean":"2024-03-24T12:27:23.887735+0000","last_became_active":"2024-03-24T12:26:05.315454+0000","last_became_peered":"2024-03-24T12:26:05.315454+0000","last_unstale":"2024-03-24T12:27:23.887735+0000","last_undegraded":"2024-03-24T12:27:23.887735+0000","last_fullsized":"2024-03-24T12:27:23.887735+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.552414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887173+0000","last_change":"2024-03-24T12:27:10.466243+0000","last_active":"2024-03-24T12:27:23.887173+0000","last_peered":"2024-03-24T12:27:23.887173+0000","last_clean":"2024-03-24T12:27:23.887173+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:27:23.887173+0000","last_undegraded":"2024-03-24T12:27:23.887173+0000","last_fullsized":"2024-03-24T12:27:23.887173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:10.466147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.466147+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:12.236572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887676+0000","last_change":"2024-03-24T12:26:09.352908+0000","last_active":"2024-03-24T12:27:23.887676+0000","last_peered":"2024-03-24T12:27:23.887676+0000","last_clean":"2024-03-24T12:27:23.887676+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:27:23.887676+0000","last_undegraded":"2024-03-24T12:27:23.887676+0000","last_fullsized":"2024-03-24T12:27:23.887676+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.176448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888045+0000","last_change":"2024-03-24T12:26:03.161516+0000","last_active":"2024-03-24T12:27:23.888045+0000","last_peered":"2024-03-24T12:27:23.888045+0000","last_clean":"2024-03-24T12:27:23.888045+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:27:23.888045+0000","last_undegraded":"2024-03-24T12:27:23.888045+0000","last_fullsized":"2024-03-24T12:27:23.888045+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.493344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887069+0000","last_change":"2024-03-24T12:27:11.454130+0000","last_active":"2024-03-24T12:27:23.887069+0000","last_peered":"2024-03-24T12:27:23.887069+0000","last_clean":"2024-03-24T12:27:23.887069+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:27:23.887069+0000","last_undegraded":"2024-03-24T12:27:23.887069+0000","last_fullsized":"2024-03-24T12:27:23.887069+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T12:27:11.453973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:11.453973+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:18.945222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887522+0000","last_change":"2024-03-24T12:26:07.336955+0000","last_active":"2024-03-24T12:27:23.887522+0000","last_peered":"2024-03-24T12:27:23.887522+0000","last_clean":"2024-03-24T12:27:23.887522+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:27:23.887522+0000","last_undegraded":"2024-03-24T12:27:23.887522+0000","last_fullsized":"2024-03-24T12:27:23.887522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:35.459896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372686+0000","last_change":"2024-03-24T12:26:09.351354+0000","last_active":"2024-03-24T12:27:24.372686+0000","last_peered":"2024-03-24T12:27:24.372686+0000","last_clean":"2024-03-24T12:27:24.372686+0000","last_became_active":"2024-03-24T12:26:09.351061+0000","last_became_peered":"2024-03-24T12:26:09.351061+0000","last_unstale":"2024-03-24T12:27:24.372686+0000","last_undegraded":"2024-03-24T12:27:24.372686+0000","last_fullsized":"2024-03-24T12:27:24.372686+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:34.667028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888654+0000","last_change":"2024-03-24T12:27:20.101666+0000","last_active":"2024-03-24T12:27:23.888654+0000","last_peered":"2024-03-24T12:27:23.888654+0000","last_clean":"2024-03-24T12:27:23.888654+0000","last_became_active":"2024-03-24T12:26:07.328595+0000","last_became_peered":"2024-03-24T12:26:07.328595+0000","last_unstale":"2024-03-24T12:27:23.888654+0000","last_undegraded":"2024-03-24T12:27:23.888654+0000","last_fullsized":"2024-03-24T12:27:23.888654+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:20.101554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.101554+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:34.528554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886980+0000","last_change":"2024-03-24T12:26:03.163624+0000","last_active":"2024-03-24T12:27:23.886980+0000","last_peered":"2024-03-24T12:27:23.886980+0000","last_clean":"2024-03-24T12:27:23.886980+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:27:23.886980+0000","last_undegraded":"2024-03-24T12:27:23.886980+0000","last_fullsized":"2024-03-24T12:27:23.886980+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.332538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478393+0000","last_change":"2024-03-24T12:26:16.885041+0000","last_active":"2024-03-24T12:27:24.478393+0000","last_peered":"2024-03-24T12:27:24.478393+0000","last_clean":"2024-03-24T12:27:24.478393+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:27:24.478393+0000","last_undegraded":"2024-03-24T12:27:24.478393+0000","last_fullsized":"2024-03-24T12:27:24.478393+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:31.314485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887362+0000","last_change":"2024-03-24T12:26:09.644131+0000","last_active":"2024-03-24T12:27:23.887362+0000","last_peered":"2024-03-24T12:27:23.887362+0000","last_clean":"2024-03-24T12:27:23.887362+0000","last_became_active":"2024-03-24T12:26:09.643739+0000","last_became_peered":"2024-03-24T12:26:09.643739+0000","last_unstale":"2024-03-24T12:27:23.887362+0000","last_undegraded":"2024-03-24T12:27:23.887362+0000","last_fullsized":"2024-03-24T12:27:23.887362+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:28.010994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373205+0000","last_change":"2024-03-24T12:27:08.227925+0000","last_active":"2024-03-24T12:27:24.373205+0000","last_peered":"2024-03-24T12:27:24.373205+0000","last_clean":"2024-03-24T12:27:24.373205+0000","last_became_active":"2024-03-24T12:26:07.332766+0000","last_became_peered":"2024-03-24T12:26:07.332766+0000","last_unstale":"2024-03-24T12:27:24.373205+0000","last_undegraded":"2024-03-24T12:27:24.373205+0000","last_fullsized":"2024-03-24T12:27:24.373205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:08.227812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:08.227812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:14.184705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887504+0000","last_change":"2024-03-24T12:26:03.159947+0000","last_active":"2024-03-24T12:27:23.887504+0000","last_peered":"2024-03-24T12:27:23.887504+0000","last_clean":"2024-03-24T12:27:23.887504+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:27:23.887504+0000","last_undegraded":"2024-03-24T12:27:23.887504+0000","last_fullsized":"2024-03-24T12:27:23.887504+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888468+0000","last_change":"2024-03-24T12:26:05.315880+0000","last_active":"2024-03-24T12:27:23.888468+0000","last_peered":"2024-03-24T12:27:23.888468+0000","last_clean":"2024-03-24T12:27:23.888468+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:27:23.888468+0000","last_undegraded":"2024-03-24T12:27:23.888468+0000","last_fullsized":"2024-03-24T12:27:23.888468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478465+0000","last_change":"2024-03-24T12:26:09.348104+0000","last_active":"2024-03-24T12:27:24.478465+0000","last_peered":"2024-03-24T12:27:24.478465+0000","last_clean":"2024-03-24T12:27:24.478465+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:27:24.478465+0000","last_undegraded":"2024-03-24T12:27:24.478465+0000","last_fullsized":"2024-03-24T12:27:24.478465+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.236528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.137182+0000","last_change":"2024-03-24T12:27:23.411568+0000","last_active":"2024-03-24T12:27:24.137182+0000","last_peered":"2024-03-24T12:27:24.137182+0000","last_clean":"2024-03-24T12:27:24.137182+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:27:24.137182+0000","last_undegraded":"2024-03-24T12:27:24.137182+0000","last_fullsized":"2024-03-24T12:27:24.137182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.411477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.411477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.665294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.139209+0000","last_change":"2024-03-24T12:27:12.636522+0000","last_active":"2024-03-24T12:27:24.139209+0000","last_peered":"2024-03-24T12:27:24.139209+0000","last_clean":"2024-03-24T12:27:24.139209+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:27:24.139209+0000","last_undegraded":"2024-03-24T12:27:24.139209+0000","last_fullsized":"2024-03-24T12:27:24.139209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.636423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.636423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.538773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888362+0000","last_change":"2024-03-24T12:26:05.316692+0000","last_active":"2024-03-24T12:27:23.888362+0000","last_peered":"2024-03-24T12:27:23.888362+0000","last_clean":"2024-03-24T12:27:23.888362+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:27:23.888362+0000","last_undegraded":"2024-03-24T12:27:23.888362+0000","last_fullsized":"2024-03-24T12:27:23.888362+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.077480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888702+0000","last_change":"2024-03-24T12:27:10.145724+0000","last_active":"2024-03-24T12:27:23.888702+0000","last_peered":"2024-03-24T12:27:23.888702+0000","last_clean":"2024-03-24T12:27:23.888702+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:27:23.888702+0000","last_undegraded":"2024-03-24T12:27:23.888702+0000","last_fullsized":"2024-03-24T12:27:23.888702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:10.145613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.145613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:22.594263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":82,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478196+0000","last_change":"2024-03-24T12:27:21.649282+0000","last_active":"2024-03-24T12:27:24.478196+0000","last_peered":"2024-03-24T12:27:24.478196+0000","last_clean":"2024-03-24T12:27:24.478196+0000","last_became_active":"2024-03-24T12:26:07.334798+0000","last_became_peered":"2024-03-24T12:26:07.334798+0000","last_unstale":"2024-03-24T12:27:24.478196+0000","last_undegraded":"2024-03-24T12:27:24.478196+0000","last_fullsized":"2024-03-24T12:27:24.478196+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:21.649181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.649181+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.389315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887325+0000","last_change":"2024-03-24T12:26:03.162618+0000","last_active":"2024-03-24T12:27:23.887325+0000","last_peered":"2024-03-24T12:27:23.887325+0000","last_clean":"2024-03-24T12:27:23.887325+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:27:23.887325+0000","last_undegraded":"2024-03-24T12:27:23.887325+0000","last_fullsized":"2024-03-24T12:27:23.887325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.490100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887782+0000","last_change":"2024-03-24T12:27:22.098052+0000","last_active":"2024-03-24T12:27:23.887782+0000","last_peered":"2024-03-24T12:27:23.887782+0000","last_clean":"2024-03-24T12:27:23.887782+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:27:23.887782+0000","last_undegraded":"2024-03-24T12:27:23.887782+0000","last_fullsized":"2024-03-24T12:27:23.887782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:22.097943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:22.097943+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.541070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":77,"num_read_kb":72,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037781,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28416,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716736,"statfs":{"total":95995035648,"available":95965937664,"internally_reserved":0,"allocated":1429504,"data_stored":951998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332123,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32556,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712596,"statfs":{"total":95995035648,"available":95961698304,"internally_reserved":0,"allocated":1404928,"data_stored":951410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27864,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717288,"statfs":{"total":95995035648,"available":95966502912,"internally_reserved":0,"allocated":864256,"data_stored":362020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32576,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712576,"statfs":{"total":95995035648,"available":95961677824,"internally_reserved":0,"allocated":1429504,"data_stored":951342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713180,"statfs":{"total":95995035648,"available":95962296320,"internally_reserved":0,"allocated":811008,"data_stored":360502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117033,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27852,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717300,"statfs":{"total":95995035648,"available":95966515200,"internally_reserved":0,"allocated":851968,"data_stored":361946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920805,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27836,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717316,"statfs":{"total":95995035648,"available":95966531584,"internally_reserved":0,"allocated":835584,"data_stored":360772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626464,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27824,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717328,"statfs":{"total":95995035648,"available":95966543872,"internally_reserved":0,"allocated":823296,"data_stored":361876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:27:30.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:27:31.213 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 podman[79785]: 2024-03-24 12:27:31.098934069 +0000 UTC m=+1.158381443 container init 28b4d8a961099913ae6effbb628288a788d351cbdaa1c7ad2a330f79001f6c30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:27:31.214 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 podman[79785]: 2024-03-24 12:27:31.104562816 +0000 UTC m=+1.164010190 container start 28b4d8a961099913ae6effbb628288a788d351cbdaa1c7ad2a330f79001f6c30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:27:31.214 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:27:31.121Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-24T12:27:31.214 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:27:31.121Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-24T12:27:31.214 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:27:31.123Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.29 port=9094 2024-03-24T12:27:31.214 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:27:31.124Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T12:27:31.214 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:27:31.148Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T12:27:31.214 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:27:31.148Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T12:27:31.214 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:27:31.151Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-24T12:27:31.214 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:27:31.151Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-24T12:27:31.214 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 bash[79785]: 28b4d8a961099913ae6effbb628288a788d351cbdaa1c7ad2a330f79001f6c30 2024-03-24T12:27:31.298 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:31.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:30 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:27:31.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:30 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:27:31.500 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:31 smithi029 systemd[1]: Started Ceph alertmanager.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:27:31.501 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:31] ENGINE Bus STOPPING 2024-03-24T12:27:31.501 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:31] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:27:31.501 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:31] ENGINE Bus STOPPED 2024-03-24T12:27:31.501 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:31] ENGINE Bus STARTING 2024-03-24T12:27:31.501 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:31] ENGINE Serving on http://:::9283 2024-03-24T12:27:31.501 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:31] ENGINE Bus STARTED 2024-03-24T12:27:32.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:32.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:32.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: 6.1f scrub starts 2024-03-24T12:27:32.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: 6.1f scrub ok 2024-03-24T12:27:32.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: 4.4 deep-scrub starts 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: 4.4 deep-scrub ok 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:32.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: 6.1f scrub starts 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: 6.1f scrub ok 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: 4.4 deep-scrub starts 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: 4.4 deep-scrub ok 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='client.24611 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: 6.1f scrub starts 2024-03-24T12:27:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: 6.1f scrub ok 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: 4.4 deep-scrub starts 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: 4.4 deep-scrub ok 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:33.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[29919]: 3.9 scrub starts 2024-03-24T12:27:33.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[29919]: 3.9 scrub ok 2024-03-24T12:27:33.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[29919]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T12:27:33.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[29919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T12:27:33.074 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[29919]: Deploying daemon grafana.a on smithi190 2024-03-24T12:27:33.075 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[29919]: 3.4 scrub starts 2024-03-24T12:27:33.075 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[29919]: 3.4 scrub ok 2024-03-24T12:27:33.075 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[41500]: 3.9 scrub starts 2024-03-24T12:27:33.075 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[41500]: 3.9 scrub ok 2024-03-24T12:27:33.075 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[41500]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T12:27:33.075 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[41500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T12:27:33.075 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[41500]: Deploying daemon grafana.a on smithi190 2024-03-24T12:27:33.075 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[41500]: 3.4 scrub starts 2024-03-24T12:27:33.075 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:32 smithi029 ceph-mon[41500]: 3.4 scrub ok 2024-03-24T12:27:33.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:32 smithi190 ceph-mon[33748]: 3.9 scrub starts 2024-03-24T12:27:33.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:32 smithi190 ceph-mon[33748]: 3.9 scrub ok 2024-03-24T12:27:33.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:32 smithi190 ceph-mon[33748]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-24T12:27:33.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:32 smithi190 ceph-mon[33748]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T12:27:33.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:32 smithi190 ceph-mon[33748]: Deploying daemon grafana.a on smithi190 2024-03-24T12:27:33.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:32 smithi190 ceph-mon[33748]: 3.4 scrub starts 2024-03-24T12:27:33.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:32 smithi190 ceph-mon[33748]: 3.4 scrub ok 2024-03-24T12:27:33.405 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:27:33.405 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:27:33.124Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000881035s 2024-03-24T12:27:33.411 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:27:34.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:27:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:27:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: 4.1d scrub starts 2024-03-24T12:27:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: 4.1d scrub ok 2024-03-24T12:27:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: 5.b scrub starts 2024-03-24T12:27:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: 5.b scrub ok 2024-03-24T12:27:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: 6.1d scrub starts 2024-03-24T12:27:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: 6.1d scrub ok 2024-03-24T12:27:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: 6.3 scrub starts 2024-03-24T12:27:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[29919]: 6.3 scrub ok 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: 4.1d scrub starts 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: 4.1d scrub ok 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: 5.b scrub starts 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: 5.b scrub ok 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: 6.1d scrub starts 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: 6.1d scrub ok 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: 6.3 scrub starts 2024-03-24T12:27:34.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:33 smithi029 ceph-mon[41500]: 6.3 scrub ok 2024-03-24T12:27:34.070 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-03-24T12:27:32.389891+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1016,"num_read_kb":862,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":632,"ondisk_log_size":632,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":236860,"kb_used_data":8252,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724356,"statfs":{"total":767960285184,"available":767717740544,"internally_reserved":0,"allocated":8450048,"data_stored":4661866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002424"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887551+0000","last_change":"2024-03-24T12:26:09.350829+0000","last_active":"2024-03-24T12:27:23.887551+0000","last_peered":"2024-03-24T12:27:23.887551+0000","last_clean":"2024-03-24T12:27:23.887551+0000","last_became_active":"2024-03-24T12:26:09.350521+0000","last_became_peered":"2024-03-24T12:26:09.350521+0000","last_unstale":"2024-03-24T12:27:23.887551+0000","last_undegraded":"2024-03-24T12:27:23.887551+0000","last_fullsized":"2024-03-24T12:27:23.887551+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.019266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.136629+0000","last_change":"2024-03-24T12:26:07.335814+0000","last_active":"2024-03-24T12:27:24.136629+0000","last_peered":"2024-03-24T12:27:24.136629+0000","last_clean":"2024-03-24T12:27:24.136629+0000","last_became_active":"2024-03-24T12:26:07.335517+0000","last_became_peered":"2024-03-24T12:26:07.335517+0000","last_unstale":"2024-03-24T12:27:24.136629+0000","last_undegraded":"2024-03-24T12:27:24.136629+0000","last_fullsized":"2024-03-24T12:27:24.136629+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:34.813926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889332+0000","last_change":"2024-03-24T12:26:03.171110+0000","last_active":"2024-03-24T12:27:23.889332+0000","last_peered":"2024-03-24T12:27:23.889332+0000","last_clean":"2024-03-24T12:27:23.889332+0000","last_became_active":"2024-03-24T12:26:03.170822+0000","last_became_peered":"2024-03-24T12:26:03.170822+0000","last_unstale":"2024-03-24T12:27:23.889332+0000","last_undegraded":"2024-03-24T12:27:23.889332+0000","last_fullsized":"2024-03-24T12:27:23.889332+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:31.894081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.141423+0000","last_change":"2024-03-24T12:27:24.141489+0000","last_active":"2024-03-24T12:27:24.141423+0000","last_peered":"2024-03-24T12:27:24.141423+0000","last_clean":"2024-03-24T12:27:24.141423+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:27:24.141423+0000","last_undegraded":"2024-03-24T12:27:24.141423+0000","last_fullsized":"2024-03-24T12:27:24.141423+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:24.141325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.141325+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.796727+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887179+0000","last_change":"2024-03-24T12:26:09.646438+0000","last_active":"2024-03-24T12:27:23.887179+0000","last_peered":"2024-03-24T12:27:23.887179+0000","last_clean":"2024-03-24T12:27:23.887179+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:27:23.887179+0000","last_undegraded":"2024-03-24T12:27:23.887179+0000","last_fullsized":"2024-03-24T12:27:23.887179+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:25.350589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887385+0000","last_change":"2024-03-24T12:26:07.335524+0000","last_active":"2024-03-24T12:27:23.887385+0000","last_peered":"2024-03-24T12:27:23.887385+0000","last_clean":"2024-03-24T12:27:23.887385+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:27:23.887385+0000","last_undegraded":"2024-03-24T12:27:23.887385+0000","last_fullsized":"2024-03-24T12:27:23.887385+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888488+0000","last_change":"2024-03-24T12:26:03.164752+0000","last_active":"2024-03-24T12:27:23.888488+0000","last_peered":"2024-03-24T12:27:23.888488+0000","last_clean":"2024-03-24T12:27:23.888488+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:27:23.888488+0000","last_undegraded":"2024-03-24T12:27:23.888488+0000","last_fullsized":"2024-03-24T12:27:23.888488+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.612593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"90'22","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:25.396603+0000","last_change":"2024-03-24T12:27:25.396689+0000","last_active":"2024-03-24T12:27:25.396603+0000","last_peered":"2024-03-24T12:27:25.396603+0000","last_clean":"2024-03-24T12:27:25.396603+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:27:25.396603+0000","last_undegraded":"2024-03-24T12:27:25.396603+0000","last_fullsized":"2024-03-24T12:27:25.396603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.396514+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.161411+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886854+0000","last_change":"2024-03-24T12:27:23.289755+0000","last_active":"2024-03-24T12:27:23.886854+0000","last_peered":"2024-03-24T12:27:23.886854+0000","last_clean":"2024-03-24T12:27:23.886854+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:27:23.886854+0000","last_undegraded":"2024-03-24T12:27:23.886854+0000","last_fullsized":"2024-03-24T12:27:23.886854+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.289666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.684236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372511+0000","last_change":"2024-03-24T12:26:07.331880+0000","last_active":"2024-03-24T12:27:24.372511+0000","last_peered":"2024-03-24T12:27:24.372511+0000","last_clean":"2024-03-24T12:27:24.372511+0000","last_became_active":"2024-03-24T12:26:07.331562+0000","last_became_peered":"2024-03-24T12:26:07.331562+0000","last_unstale":"2024-03-24T12:27:24.372511+0000","last_undegraded":"2024-03-24T12:27:24.372511+0000","last_fullsized":"2024-03-24T12:27:24.372511+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372688+0000","last_change":"2024-03-24T12:27:20.255011+0000","last_active":"2024-03-24T12:27:24.372688+0000","last_peered":"2024-03-24T12:27:24.372688+0000","last_clean":"2024-03-24T12:27:24.372688+0000","last_became_active":"2024-03-24T12:26:03.161544+0000","last_became_peered":"2024-03-24T12:26:03.161544+0000","last_unstale":"2024-03-24T12:27:24.372688+0000","last_undegraded":"2024-03-24T12:27:24.372688+0000","last_fullsized":"2024-03-24T12:27:24.372688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:20.254898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.254898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.151491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887903+0000","last_change":"2024-03-24T12:26:05.318208+0000","last_active":"2024-03-24T12:27:23.887903+0000","last_peered":"2024-03-24T12:27:23.887903+0000","last_clean":"2024-03-24T12:27:23.887903+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:27:23.887903+0000","last_undegraded":"2024-03-24T12:27:23.887903+0000","last_fullsized":"2024-03-24T12:27:23.887903+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.433393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372248+0000","last_change":"2024-03-24T12:27:21.260071+0000","last_active":"2024-03-24T12:27:24.372248+0000","last_peered":"2024-03-24T12:27:24.372248+0000","last_clean":"2024-03-24T12:27:24.372248+0000","last_became_active":"2024-03-24T12:26:09.350721+0000","last_became_peered":"2024-03-24T12:26:09.350721+0000","last_unstale":"2024-03-24T12:27:24.372248+0000","last_undegraded":"2024-03-24T12:27:24.372248+0000","last_fullsized":"2024-03-24T12:27:24.372248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.259962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.801612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437152+0000","last_change":"2024-03-24T12:27:17.365794+0000","last_active":"2024-03-24T12:27:24.437152+0000","last_peered":"2024-03-24T12:27:24.437152+0000","last_clean":"2024-03-24T12:27:24.437152+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:27:24.437152+0000","last_undegraded":"2024-03-24T12:27:24.437152+0000","last_fullsized":"2024-03-24T12:27:24.437152+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.365696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.873148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372612+0000","last_change":"2024-03-24T12:27:10.251781+0000","last_active":"2024-03-24T12:27:24.372612+0000","last_peered":"2024-03-24T12:27:24.372612+0000","last_clean":"2024-03-24T12:27:24.372612+0000","last_became_active":"2024-03-24T12:26:03.159997+0000","last_became_peered":"2024-03-24T12:26:03.159997+0000","last_unstale":"2024-03-24T12:27:24.372612+0000","last_undegraded":"2024-03-24T12:27:24.372612+0000","last_fullsized":"2024-03-24T12:27:24.372612+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.251686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.902316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887583+0000","last_change":"2024-03-24T12:26:05.322947+0000","last_active":"2024-03-24T12:27:23.887583+0000","last_peered":"2024-03-24T12:27:23.887583+0000","last_clean":"2024-03-24T12:27:23.887583+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:27:23.887583+0000","last_undegraded":"2024-03-24T12:27:23.887583+0000","last_fullsized":"2024-03-24T12:27:23.887583+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:25.164535+0000","last_change":"2024-03-24T12:27:25.164561+0000","last_active":"2024-03-24T12:27:25.164535+0000","last_peered":"2024-03-24T12:27:25.164535+0000","last_clean":"2024-03-24T12:27:25.164535+0000","last_became_active":"2024-03-24T12:26:09.643671+0000","last_became_peered":"2024-03-24T12:26:09.643671+0000","last_unstale":"2024-03-24T12:27:25.164535+0000","last_undegraded":"2024-03-24T12:27:25.164535+0000","last_fullsized":"2024-03-24T12:27:25.164535+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:25.164460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.164460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.284324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:28.397253+0000","last_change":"2024-03-24T12:27:28.397274+0000","last_active":"2024-03-24T12:27:28.397253+0000","last_peered":"2024-03-24T12:27:28.397253+0000","last_clean":"2024-03-24T12:27:28.397253+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:27:28.397253+0000","last_undegraded":"2024-03-24T12:27:28.397253+0000","last_fullsized":"2024-03-24T12:27:28.397253+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.397179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.397179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.594543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:28.360991+0000","last_change":"2024-03-24T12:27:28.361014+0000","last_active":"2024-03-24T12:27:28.360991+0000","last_peered":"2024-03-24T12:27:28.360991+0000","last_clean":"2024-03-24T12:27:28.360991+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:27:28.360991+0000","last_undegraded":"2024-03-24T12:27:28.360991+0000","last_fullsized":"2024-03-24T12:27:28.360991+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.360926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.835122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478184+0000","last_change":"2024-03-24T12:26:05.322833+0000","last_active":"2024-03-24T12:27:24.478184+0000","last_peered":"2024-03-24T12:27:24.478184+0000","last_clean":"2024-03-24T12:27:24.478184+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:27:24.478184+0000","last_undegraded":"2024-03-24T12:27:24.478184+0000","last_fullsized":"2024-03-24T12:27:24.478184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.619352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:30.441550+0000","last_change":"2024-03-24T12:27:30.441572+0000","last_active":"2024-03-24T12:27:30.441550+0000","last_peered":"2024-03-24T12:27:30.441550+0000","last_clean":"2024-03-24T12:27:30.441550+0000","last_became_active":"2024-03-24T12:26:09.643738+0000","last_became_peered":"2024-03-24T12:26:09.643738+0000","last_unstale":"2024-03-24T12:27:30.441550+0000","last_undegraded":"2024-03-24T12:27:30.441550+0000","last_fullsized":"2024-03-24T12:27:30.441550+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:30.441497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:30.441497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.867059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887737+0000","last_change":"2024-03-24T12:26:07.333844+0000","last_active":"2024-03-24T12:27:23.887737+0000","last_peered":"2024-03-24T12:27:23.887737+0000","last_clean":"2024-03-24T12:27:23.887737+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:27:23.887737+0000","last_undegraded":"2024-03-24T12:27:23.887737+0000","last_fullsized":"2024-03-24T12:27:23.887737+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887942+0000","last_change":"2024-03-24T12:26:03.160687+0000","last_active":"2024-03-24T12:27:23.887942+0000","last_peered":"2024-03-24T12:27:23.887942+0000","last_clean":"2024-03-24T12:27:23.887942+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:27:23.887942+0000","last_undegraded":"2024-03-24T12:27:23.887942+0000","last_fullsized":"2024-03-24T12:27:23.887942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.827198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887683+0000","last_change":"2024-03-24T12:26:05.321693+0000","last_active":"2024-03-24T12:27:23.887683+0000","last_peered":"2024-03-24T12:27:23.887683+0000","last_clean":"2024-03-24T12:27:23.887683+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:27:23.887683+0000","last_undegraded":"2024-03-24T12:27:23.887683+0000","last_fullsized":"2024-03-24T12:27:23.887683+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437511+0000","last_change":"2024-03-24T12:26:09.344099+0000","last_active":"2024-03-24T12:27:24.437511+0000","last_peered":"2024-03-24T12:27:24.437511+0000","last_clean":"2024-03-24T12:27:24.437511+0000","last_became_active":"2024-03-24T12:26:09.343786+0000","last_became_peered":"2024-03-24T12:26:09.343786+0000","last_unstale":"2024-03-24T12:27:24.437511+0000","last_undegraded":"2024-03-24T12:27:24.437511+0000","last_fullsized":"2024-03-24T12:27:24.437511+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.139928+0000","last_change":"2024-03-24T12:26:07.335552+0000","last_active":"2024-03-24T12:27:24.139928+0000","last_peered":"2024-03-24T12:27:24.139928+0000","last_clean":"2024-03-24T12:27:24.139928+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:27:24.139928+0000","last_undegraded":"2024-03-24T12:27:24.139928+0000","last_fullsized":"2024-03-24T12:27:24.139928+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:35.440267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:27.409004+0000","last_change":"2024-03-24T12:27:27.409032+0000","last_active":"2024-03-24T12:27:27.409004+0000","last_peered":"2024-03-24T12:27:27.409004+0000","last_clean":"2024-03-24T12:27:27.409004+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:27:27.409004+0000","last_undegraded":"2024-03-24T12:27:27.409004+0000","last_fullsized":"2024-03-24T12:27:27.409004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:27.408937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.408937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.920390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888827+0000","last_change":"2024-03-24T12:27:11.269339+0000","last_active":"2024-03-24T12:27:23.888827+0000","last_peered":"2024-03-24T12:27:23.888827+0000","last_clean":"2024-03-24T12:27:23.888827+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:27:23.888827+0000","last_undegraded":"2024-03-24T12:27:23.888827+0000","last_fullsized":"2024-03-24T12:27:23.888827+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:11.269181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:11.269181+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:34.970841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.137202+0000","last_change":"2024-03-24T12:26:09.348873+0000","last_active":"2024-03-24T12:27:24.137202+0000","last_peered":"2024-03-24T12:27:24.137202+0000","last_clean":"2024-03-24T12:27:24.137202+0000","last_became_active":"2024-03-24T12:26:09.348355+0000","last_became_peered":"2024-03-24T12:26:09.348355+0000","last_unstale":"2024-03-24T12:27:24.137202+0000","last_undegraded":"2024-03-24T12:27:24.137202+0000","last_fullsized":"2024-03-24T12:27:24.137202+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:33.087921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889095+0000","last_change":"2024-03-24T12:26:07.331932+0000","last_active":"2024-03-24T12:27:23.889095+0000","last_peered":"2024-03-24T12:27:23.889095+0000","last_clean":"2024-03-24T12:27:23.889095+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:27:23.889095+0000","last_undegraded":"2024-03-24T12:27:23.889095+0000","last_fullsized":"2024-03-24T12:27:23.889095+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.308109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886870+0000","last_change":"2024-03-24T12:27:12.437790+0000","last_active":"2024-03-24T12:27:23.886870+0000","last_peered":"2024-03-24T12:27:23.886870+0000","last_clean":"2024-03-24T12:27:23.886870+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:27:23.886870+0000","last_undegraded":"2024-03-24T12:27:23.886870+0000","last_fullsized":"2024-03-24T12:27:23.886870+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.437708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:25.841012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:26.763391+0000","last_change":"2024-03-24T12:27:26.763428+0000","last_active":"2024-03-24T12:27:26.763391+0000","last_peered":"2024-03-24T12:27:26.763391+0000","last_clean":"2024-03-24T12:27:26.763391+0000","last_became_active":"2024-03-24T12:26:05.317587+0000","last_became_peered":"2024-03-24T12:26:05.317587+0000","last_unstale":"2024-03-24T12:27:26.763391+0000","last_undegraded":"2024-03-24T12:27:26.763391+0000","last_fullsized":"2024-03-24T12:27:26.763391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:26.763317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.763317+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.728888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437135+0000","last_change":"2024-03-24T12:26:09.345188+0000","last_active":"2024-03-24T12:27:24.437135+0000","last_peered":"2024-03-24T12:27:24.437135+0000","last_clean":"2024-03-24T12:27:24.437135+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:27:24.437135+0000","last_undegraded":"2024-03-24T12:27:24.437135+0000","last_fullsized":"2024-03-24T12:27:24.437135+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:37.908175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.138974+0000","last_change":"2024-03-24T12:26:07.335668+0000","last_active":"2024-03-24T12:27:24.138974+0000","last_peered":"2024-03-24T12:27:24.138974+0000","last_clean":"2024-03-24T12:27:24.138974+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:27:24.138974+0000","last_undegraded":"2024-03-24T12:27:24.138974+0000","last_fullsized":"2024-03-24T12:27:24.138974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.115036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889353+0000","last_change":"2024-03-24T12:26:03.171104+0000","last_active":"2024-03-24T12:27:23.889353+0000","last_peered":"2024-03-24T12:27:23.889353+0000","last_clean":"2024-03-24T12:27:23.889353+0000","last_became_active":"2024-03-24T12:26:03.170395+0000","last_became_peered":"2024-03-24T12:26:03.170395+0000","last_unstale":"2024-03-24T12:27:23.889353+0000","last_undegraded":"2024-03-24T12:27:23.889353+0000","last_fullsized":"2024-03-24T12:27:23.889353+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:25.060953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887019+0000","last_change":"2024-03-24T12:27:15.467117+0000","last_active":"2024-03-24T12:27:23.887019+0000","last_peered":"2024-03-24T12:27:23.887019+0000","last_clean":"2024-03-24T12:27:23.887019+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:27:23.887019+0000","last_undegraded":"2024-03-24T12:27:23.887019+0000","last_fullsized":"2024-03-24T12:27:23.887019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:27:15.466966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:15.466966+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:23.327159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886994+0000","last_change":"2024-03-24T12:26:09.646151+0000","last_active":"2024-03-24T12:27:23.886994+0000","last_peered":"2024-03-24T12:27:23.886994+0000","last_clean":"2024-03-24T12:27:23.886994+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:27:23.886994+0000","last_undegraded":"2024-03-24T12:27:23.886994+0000","last_fullsized":"2024-03-24T12:27:23.886994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.849429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437414+0000","last_change":"2024-03-24T12:27:21.335557+0000","last_active":"2024-03-24T12:27:24.437414+0000","last_peered":"2024-03-24T12:27:24.437414+0000","last_clean":"2024-03-24T12:27:24.437414+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:27:24.437414+0000","last_undegraded":"2024-03-24T12:27:24.437414+0000","last_fullsized":"2024-03-24T12:27:24.437414+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:21.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.789829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373007+0000","last_change":"2024-03-24T12:26:03.166070+0000","last_active":"2024-03-24T12:27:24.373007+0000","last_peered":"2024-03-24T12:27:24.373007+0000","last_clean":"2024-03-24T12:27:24.373007+0000","last_became_active":"2024-03-24T12:26:03.165083+0000","last_became_peered":"2024-03-24T12:26:03.165083+0000","last_unstale":"2024-03-24T12:27:24.373007+0000","last_undegraded":"2024-03-24T12:27:24.373007+0000","last_fullsized":"2024-03-24T12:27:24.373007+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887607+0000","last_change":"2024-03-24T12:27:16.497904+0000","last_active":"2024-03-24T12:27:23.887607+0000","last_peered":"2024-03-24T12:27:23.887607+0000","last_clean":"2024-03-24T12:27:23.887607+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:27:23.887607+0000","last_undegraded":"2024-03-24T12:27:23.887607+0000","last_fullsized":"2024-03-24T12:27:23.887607+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:16.497755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.497755+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.894323+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886620+0000","last_change":"2024-03-24T12:26:09.351371+0000","last_active":"2024-03-24T12:27:23.886620+0000","last_peered":"2024-03-24T12:27:23.886620+0000","last_clean":"2024-03-24T12:27:23.886620+0000","last_became_active":"2024-03-24T12:26:09.350656+0000","last_became_peered":"2024-03-24T12:26:09.350656+0000","last_unstale":"2024-03-24T12:27:23.886620+0000","last_undegraded":"2024-03-24T12:27:23.886620+0000","last_fullsized":"2024-03-24T12:27:23.886620+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887306+0000","last_change":"2024-03-24T12:27:20.458514+0000","last_active":"2024-03-24T12:27:23.887306+0000","last_peered":"2024-03-24T12:27:23.887306+0000","last_clean":"2024-03-24T12:27:23.887306+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:27:23.887306+0000","last_undegraded":"2024-03-24T12:27:23.887306+0000","last_fullsized":"2024-03-24T12:27:23.887306+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.458419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:24.491860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437613+0000","last_change":"2024-03-24T12:27:14.329516+0000","last_active":"2024-03-24T12:27:24.437613+0000","last_peered":"2024-03-24T12:27:24.437613+0000","last_clean":"2024-03-24T12:27:24.437613+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:27:24.437613+0000","last_undegraded":"2024-03-24T12:27:24.437613+0000","last_fullsized":"2024-03-24T12:27:24.437613+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:14.329357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:14.329357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:21.223400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372275+0000","last_change":"2024-03-24T12:26:16.882174+0000","last_active":"2024-03-24T12:27:24.372275+0000","last_peered":"2024-03-24T12:27:24.372275+0000","last_clean":"2024-03-24T12:27:24.372275+0000","last_became_active":"2024-03-24T12:26:16.874170+0000","last_became_peered":"2024-03-24T12:26:16.874170+0000","last_unstale":"2024-03-24T12:27:24.372275+0000","last_undegraded":"2024-03-24T12:27:24.372275+0000","last_fullsized":"2024-03-24T12:27:24.372275+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:26.431453+0000","last_change":"2024-03-24T12:27:26.431474+0000","last_active":"2024-03-24T12:27:26.431453+0000","last_peered":"2024-03-24T12:27:26.431453+0000","last_clean":"2024-03-24T12:27:26.431453+0000","last_became_active":"2024-03-24T12:26:09.343296+0000","last_became_peered":"2024-03-24T12:26:09.343296+0000","last_unstale":"2024-03-24T12:27:26.431453+0000","last_undegraded":"2024-03-24T12:27:26.431453+0000","last_fullsized":"2024-03-24T12:27:26.431453+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.431382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.685011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:29.397536+0000","last_change":"2024-03-24T12:27:29.397575+0000","last_active":"2024-03-24T12:27:29.397536+0000","last_peered":"2024-03-24T12:27:29.397536+0000","last_clean":"2024-03-24T12:27:29.397536+0000","last_became_active":"2024-03-24T12:26:07.330811+0000","last_became_peered":"2024-03-24T12:26:07.330811+0000","last_unstale":"2024-03-24T12:27:29.397536+0000","last_undegraded":"2024-03-24T12:27:29.397536+0000","last_fullsized":"2024-03-24T12:27:29.397536+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:29.397463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:29.397463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:52.953466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888093+0000","last_change":"2024-03-24T12:27:09.137207+0000","last_active":"2024-03-24T12:27:23.888093+0000","last_peered":"2024-03-24T12:27:23.888093+0000","last_clean":"2024-03-24T12:27:23.888093+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:27:23.888093+0000","last_undegraded":"2024-03-24T12:27:23.888093+0000","last_fullsized":"2024-03-24T12:27:23.888093+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:09.137104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:09.137104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.329586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"88'16","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888125+0000","last_change":"2024-03-24T12:27:17.084272+0000","last_active":"2024-03-24T12:27:23.888125+0000","last_peered":"2024-03-24T12:27:23.888125+0000","last_clean":"2024-03-24T12:27:23.888125+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:27:23.888125+0000","last_undegraded":"2024-03-24T12:27:23.888125+0000","last_fullsized":"2024-03-24T12:27:23.888125+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.084148+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.541132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888504+0000","last_change":"2024-03-24T12:26:09.349851+0000","last_active":"2024-03-24T12:27:23.888504+0000","last_peered":"2024-03-24T12:27:23.888504+0000","last_clean":"2024-03-24T12:27:23.888504+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:27:23.888504+0000","last_undegraded":"2024-03-24T12:27:23.888504+0000","last_fullsized":"2024-03-24T12:27:23.888504+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.575676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372891+0000","last_change":"2024-03-24T12:26:07.330555+0000","last_active":"2024-03-24T12:27:24.372891+0000","last_peered":"2024-03-24T12:27:24.372891+0000","last_clean":"2024-03-24T12:27:24.372891+0000","last_became_active":"2024-03-24T12:26:07.329487+0000","last_became_peered":"2024-03-24T12:26:07.329487+0000","last_unstale":"2024-03-24T12:27:24.372891+0000","last_undegraded":"2024-03-24T12:27:24.372891+0000","last_fullsized":"2024-03-24T12:27:24.372891+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437318+0000","last_change":"2024-03-24T12:26:03.163577+0000","last_active":"2024-03-24T12:27:24.437318+0000","last_peered":"2024-03-24T12:27:24.437318+0000","last_clean":"2024-03-24T12:27:24.437318+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:27:24.437318+0000","last_undegraded":"2024-03-24T12:27:24.437318+0000","last_fullsized":"2024-03-24T12:27:24.437318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.200101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887407+0000","last_change":"2024-03-24T12:27:21.451811+0000","last_active":"2024-03-24T12:27:23.887407+0000","last_peered":"2024-03-24T12:27:23.887407+0000","last_clean":"2024-03-24T12:27:23.887407+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:27:23.887407+0000","last_undegraded":"2024-03-24T12:27:23.887407+0000","last_fullsized":"2024-03-24T12:27:23.887407+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:21.451717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.451717+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.819749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888742+0000","last_change":"2024-03-24T12:26:09.346362+0000","last_active":"2024-03-24T12:27:23.888742+0000","last_peered":"2024-03-24T12:27:23.888742+0000","last_clean":"2024-03-24T12:27:23.888742+0000","last_became_active":"2024-03-24T12:26:09.346062+0000","last_became_peered":"2024-03-24T12:26:09.346062+0000","last_unstale":"2024-03-24T12:27:23.888742+0000","last_undegraded":"2024-03-24T12:27:23.888742+0000","last_fullsized":"2024-03-24T12:27:23.888742+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888561+0000","last_change":"2024-03-24T12:26:07.338058+0000","last_active":"2024-03-24T12:27:23.888561+0000","last_peered":"2024-03-24T12:27:23.888561+0000","last_clean":"2024-03-24T12:27:23.888561+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:27:23.888561+0000","last_undegraded":"2024-03-24T12:27:23.888561+0000","last_fullsized":"2024-03-24T12:27:23.888561+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:31.504792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888704+0000","last_change":"2024-03-24T12:27:07.317120+0000","last_active":"2024-03-24T12:27:23.888704+0000","last_peered":"2024-03-24T12:27:23.888704+0000","last_clean":"2024-03-24T12:27:23.888704+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:27:23.888704+0000","last_undegraded":"2024-03-24T12:27:23.888704+0000","last_fullsized":"2024-03-24T12:27:23.888704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:07.317021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:07.317021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.086017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'13","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373017+0000","last_change":"2024-03-24T12:27:16.242881+0000","last_active":"2024-03-24T12:27:24.373017+0000","last_peered":"2024-03-24T12:27:24.373017+0000","last_clean":"2024-03-24T12:27:24.373017+0000","last_became_active":"2024-03-24T12:26:05.358213+0000","last_became_peered":"2024-03-24T12:26:05.358213+0000","last_unstale":"2024-03-24T12:27:24.373017+0000","last_undegraded":"2024-03-24T12:27:24.373017+0000","last_fullsized":"2024-03-24T12:27:24.373017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.242776+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.762237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:27.661039+0000","last_change":"2024-03-24T12:27:27.661082+0000","last_active":"2024-03-24T12:27:27.661039+0000","last_peered":"2024-03-24T12:27:27.661039+0000","last_clean":"2024-03-24T12:27:27.661039+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:27:27.661039+0000","last_undegraded":"2024-03-24T12:27:27.661039+0000","last_fullsized":"2024-03-24T12:27:27.661039+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.660932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.997979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888192+0000","last_change":"2024-03-24T12:26:07.343233+0000","last_active":"2024-03-24T12:27:23.888192+0000","last_peered":"2024-03-24T12:27:23.888192+0000","last_clean":"2024-03-24T12:27:23.888192+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:27:23.888192+0000","last_undegraded":"2024-03-24T12:27:23.888192+0000","last_fullsized":"2024-03-24T12:27:23.888192+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.890619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437199+0000","last_change":"2024-03-24T12:27:06.422517+0000","last_active":"2024-03-24T12:27:24.437199+0000","last_peered":"2024-03-24T12:27:24.437199+0000","last_clean":"2024-03-24T12:27:24.437199+0000","last_became_active":"2024-03-24T12:26:03.166149+0000","last_became_peered":"2024-03-24T12:26:03.166149+0000","last_unstale":"2024-03-24T12:27:24.437199+0000","last_undegraded":"2024-03-24T12:27:24.437199+0000","last_fullsized":"2024-03-24T12:27:24.437199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:06.422387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.422387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:12.269750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":77,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:30.384094+0000","last_change":"2024-03-24T12:27:30.384123+0000","last_active":"2024-03-24T12:27:30.384094+0000","last_peered":"2024-03-24T12:27:30.384094+0000","last_clean":"2024-03-24T12:27:30.384094+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:27:30.384094+0000","last_undegraded":"2024-03-24T12:27:30.384094+0000","last_fullsized":"2024-03-24T12:27:30.384094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:30.384035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:30.384035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:48.792180+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437608+0000","last_change":"2024-03-24T12:26:09.646175+0000","last_active":"2024-03-24T12:27:24.437608+0000","last_peered":"2024-03-24T12:27:24.437608+0000","last_clean":"2024-03-24T12:27:24.437608+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:27:24.437608+0000","last_undegraded":"2024-03-24T12:27:24.437608+0000","last_fullsized":"2024-03-24T12:27:24.437608+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373084+0000","last_change":"2024-03-24T12:26:07.329432+0000","last_active":"2024-03-24T12:27:24.373084+0000","last_peered":"2024-03-24T12:27:24.373084+0000","last_clean":"2024-03-24T12:27:24.373084+0000","last_became_active":"2024-03-24T12:26:07.329117+0000","last_became_peered":"2024-03-24T12:26:07.329117+0000","last_unstale":"2024-03-24T12:27:24.373084+0000","last_undegraded":"2024-03-24T12:27:24.373084+0000","last_fullsized":"2024-03-24T12:27:24.373084+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:25.438312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.138906+0000","last_change":"2024-03-24T12:27:06.584912+0000","last_active":"2024-03-24T12:27:24.138906+0000","last_peered":"2024-03-24T12:27:24.138906+0000","last_clean":"2024-03-24T12:27:24.138906+0000","last_became_active":"2024-03-24T12:26:03.161414+0000","last_became_peered":"2024-03-24T12:26:03.161414+0000","last_unstale":"2024-03-24T12:27:24.138906+0000","last_undegraded":"2024-03-24T12:27:24.138906+0000","last_fullsized":"2024-03-24T12:27:24.138906+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:06.584809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.584809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:16.866693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":62,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888362+0000","last_change":"2024-03-24T12:26:05.357244+0000","last_active":"2024-03-24T12:27:23.888362+0000","last_peered":"2024-03-24T12:27:23.888362+0000","last_clean":"2024-03-24T12:27:23.888362+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:27:23.888362+0000","last_undegraded":"2024-03-24T12:27:23.888362+0000","last_fullsized":"2024-03-24T12:27:23.888362+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:31.433340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372719+0000","last_change":"2024-03-24T12:26:09.643942+0000","last_active":"2024-03-24T12:27:24.372719+0000","last_peered":"2024-03-24T12:27:24.372719+0000","last_clean":"2024-03-24T12:27:24.372719+0000","last_became_active":"2024-03-24T12:26:09.643602+0000","last_became_peered":"2024-03-24T12:26:09.643602+0000","last_unstale":"2024-03-24T12:27:24.372719+0000","last_undegraded":"2024-03-24T12:27:24.372719+0000","last_fullsized":"2024-03-24T12:27:24.372719+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:34.205128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437939+0000","last_change":"2024-03-24T12:26:03.161676+0000","last_active":"2024-03-24T12:27:24.437939+0000","last_peered":"2024-03-24T12:27:24.437939+0000","last_clean":"2024-03-24T12:27:24.437939+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:27:24.437939+0000","last_undegraded":"2024-03-24T12:27:24.437939+0000","last_fullsized":"2024-03-24T12:27:24.437939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372273+0000","last_change":"2024-03-24T12:27:17.288726+0000","last_active":"2024-03-24T12:27:24.372273+0000","last_peered":"2024-03-24T12:27:24.372273+0000","last_clean":"2024-03-24T12:27:24.372273+0000","last_became_active":"2024-03-24T12:26:05.358217+0000","last_became_peered":"2024-03-24T12:26:05.358217+0000","last_unstale":"2024-03-24T12:27:24.372273+0000","last_undegraded":"2024-03-24T12:27:24.372273+0000","last_fullsized":"2024-03-24T12:27:24.372273+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:27:17.288616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.288616+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:31.276817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":86,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437516+0000","last_change":"2024-03-24T12:27:20.354464+0000","last_active":"2024-03-24T12:27:24.437516+0000","last_peered":"2024-03-24T12:27:24.437516+0000","last_clean":"2024-03-24T12:27:24.437516+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:27:24.437516+0000","last_undegraded":"2024-03-24T12:27:24.437516+0000","last_fullsized":"2024-03-24T12:27:24.437516+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:20.354367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.354367+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.013310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887122+0000","last_change":"2024-03-24T12:26:09.350769+0000","last_active":"2024-03-24T12:27:23.887122+0000","last_peered":"2024-03-24T12:27:23.887122+0000","last_clean":"2024-03-24T12:27:23.887122+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:27:23.887122+0000","last_undegraded":"2024-03-24T12:27:23.887122+0000","last_fullsized":"2024-03-24T12:27:23.887122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:38.324576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437855+0000","last_change":"2024-03-24T12:26:03.162567+0000","last_active":"2024-03-24T12:27:24.437855+0000","last_peered":"2024-03-24T12:27:24.437855+0000","last_clean":"2024-03-24T12:27:24.437855+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:27:24.437855+0000","last_undegraded":"2024-03-24T12:27:24.437855+0000","last_fullsized":"2024-03-24T12:27:24.437855+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:31.590674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":84,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888020+0000","last_change":"2024-03-24T12:27:23.131685+0000","last_active":"2024-03-24T12:27:23.888020+0000","last_peered":"2024-03-24T12:27:23.888020+0000","last_clean":"2024-03-24T12:27:23.888020+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:27:23.888020+0000","last_undegraded":"2024-03-24T12:27:23.888020+0000","last_fullsized":"2024-03-24T12:27:23.888020+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:23.131573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.131573+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.851864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.477937+0000","last_change":"2024-03-24T12:26:07.335582+0000","last_active":"2024-03-24T12:27:24.477937+0000","last_peered":"2024-03-24T12:27:24.477937+0000","last_clean":"2024-03-24T12:27:24.477937+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:27:24.477937+0000","last_undegraded":"2024-03-24T12:27:24.477937+0000","last_fullsized":"2024-03-24T12:27:24.477937+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.841365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.436974+0000","last_change":"2024-03-24T12:26:09.339988+0000","last_active":"2024-03-24T12:27:24.436974+0000","last_peered":"2024-03-24T12:27:24.436974+0000","last_clean":"2024-03-24T12:27:24.436974+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:27:24.436974+0000","last_undegraded":"2024-03-24T12:27:24.436974+0000","last_fullsized":"2024-03-24T12:27:24.436974+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.803044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478137+0000","last_change":"2024-03-24T12:26:07.331773+0000","last_active":"2024-03-24T12:27:24.478137+0000","last_peered":"2024-03-24T12:27:24.478137+0000","last_clean":"2024-03-24T12:27:24.478137+0000","last_became_active":"2024-03-24T12:26:07.331416+0000","last_became_peered":"2024-03-24T12:26:07.331416+0000","last_unstale":"2024-03-24T12:27:24.478137+0000","last_undegraded":"2024-03-24T12:27:24.478137+0000","last_fullsized":"2024-03-24T12:27:24.478137+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.501587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437852+0000","last_change":"2024-03-24T12:26:03.165503+0000","last_active":"2024-03-24T12:27:24.437852+0000","last_peered":"2024-03-24T12:27:24.437852+0000","last_clean":"2024-03-24T12:27:24.437852+0000","last_became_active":"2024-03-24T12:26:03.165214+0000","last_became_peered":"2024-03-24T12:26:03.165214+0000","last_unstale":"2024-03-24T12:27:24.437852+0000","last_undegraded":"2024-03-24T12:27:24.437852+0000","last_fullsized":"2024-03-24T12:27:24.437852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":92,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.139487+0000","last_change":"2024-03-24T12:27:15.668713+0000","last_active":"2024-03-24T12:27:24.139487+0000","last_peered":"2024-03-24T12:27:24.139487+0000","last_clean":"2024-03-24T12:27:24.139487+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:27:24.139487+0000","last_undegraded":"2024-03-24T12:27:24.139487+0000","last_fullsized":"2024-03-24T12:27:24.139487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:27:15.668556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:15.668556+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:43.755371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886863+0000","last_change":"2024-03-24T12:26:09.350031+0000","last_active":"2024-03-24T12:27:23.886863+0000","last_peered":"2024-03-24T12:27:23.886863+0000","last_clean":"2024-03-24T12:27:23.886863+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:27:23.886863+0000","last_undegraded":"2024-03-24T12:27:23.886863+0000","last_fullsized":"2024-03-24T12:27:23.886863+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.187535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478091+0000","last_change":"2024-03-24T12:26:07.329850+0000","last_active":"2024-03-24T12:27:24.478091+0000","last_peered":"2024-03-24T12:27:24.478091+0000","last_clean":"2024-03-24T12:27:24.478091+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:27:24.478091+0000","last_undegraded":"2024-03-24T12:27:24.478091+0000","last_fullsized":"2024-03-24T12:27:24.478091+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.923438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372614+0000","last_change":"2024-03-24T12:27:09.234408+0000","last_active":"2024-03-24T12:27:24.372614+0000","last_peered":"2024-03-24T12:27:24.372614+0000","last_clean":"2024-03-24T12:27:24.372614+0000","last_became_active":"2024-03-24T12:26:03.162562+0000","last_became_peered":"2024-03-24T12:26:03.162562+0000","last_unstale":"2024-03-24T12:27:24.372614+0000","last_undegraded":"2024-03-24T12:27:24.372614+0000","last_fullsized":"2024-03-24T12:27:24.372614+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:27:09.234299+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:17.263964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889082+0000","last_change":"2024-03-24T12:26:16.882489+0000","last_active":"2024-03-24T12:27:23.889082+0000","last_peered":"2024-03-24T12:27:23.889082+0000","last_clean":"2024-03-24T12:27:23.889082+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:27:23.889082+0000","last_undegraded":"2024-03-24T12:27:23.889082+0000","last_fullsized":"2024-03-24T12:27:23.889082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:28.651922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.439361+0000","last_change":"2024-03-24T12:27:24.439386+0000","last_active":"2024-03-24T12:27:24.439361+0000","last_peered":"2024-03-24T12:27:24.439361+0000","last_clean":"2024-03-24T12:27:24.439361+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:27:24.439361+0000","last_undegraded":"2024-03-24T12:27:24.439361+0000","last_fullsized":"2024-03-24T12:27:24.439361+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.439296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.495934+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372454+0000","last_change":"2024-03-24T12:26:07.318867+0000","last_active":"2024-03-24T12:27:24.372454+0000","last_peered":"2024-03-24T12:27:24.372454+0000","last_clean":"2024-03-24T12:27:24.372454+0000","last_became_active":"2024-03-24T12:26:03.165771+0000","last_became_peered":"2024-03-24T12:26:03.165771+0000","last_unstale":"2024-03-24T12:27:24.372454+0000","last_undegraded":"2024-03-24T12:27:24.372454+0000","last_fullsized":"2024-03-24T12:27:24.372454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:29.390729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:32.227040+0000","last_change":"2024-03-24T12:27:32.227062+0000","last_active":"2024-03-24T12:27:32.227040+0000","last_peered":"2024-03-24T12:27:32.227040+0000","last_clean":"2024-03-24T12:27:32.227040+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:27:32.227040+0000","last_undegraded":"2024-03-24T12:27:32.227040+0000","last_fullsized":"2024-03-24T12:27:32.227040+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.226982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.226982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.516101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"90'38","reported_seq":137,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:31.375718+0000","last_change":"2024-03-24T12:27:31.375798+0000","last_active":"2024-03-24T12:27:31.375718+0000","last_peered":"2024-03-24T12:27:31.375718+0000","last_clean":"2024-03-24T12:27:31.375718+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:27:31.375718+0000","last_undegraded":"2024-03-24T12:27:31.375718+0000","last_fullsized":"2024-03-24T12:27:31.375718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'38","last_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:27:31.375629+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:55.069683+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888397+0000","last_change":"2024-03-24T12:26:07.332154+0000","last_active":"2024-03-24T12:27:23.888397+0000","last_peered":"2024-03-24T12:27:23.888397+0000","last_clean":"2024-03-24T12:27:23.888397+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:27:23.888397+0000","last_undegraded":"2024-03-24T12:27:23.888397+0000","last_fullsized":"2024-03-24T12:27:23.888397+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.438013+0000","last_change":"2024-03-24T12:27:18.336245+0000","last_active":"2024-03-24T12:27:24.438013+0000","last_peered":"2024-03-24T12:27:24.438013+0000","last_clean":"2024-03-24T12:27:24.438013+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:27:24.438013+0000","last_undegraded":"2024-03-24T12:27:24.438013+0000","last_fullsized":"2024-03-24T12:27:24.438013+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:18.336147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:18.336147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.167902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.136626+0000","last_change":"2024-03-24T12:27:16.330188+0000","last_active":"2024-03-24T12:27:24.136626+0000","last_peered":"2024-03-24T12:27:24.136626+0000","last_clean":"2024-03-24T12:27:24.136626+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:27:24.136626+0000","last_undegraded":"2024-03-24T12:27:24.136626+0000","last_fullsized":"2024-03-24T12:27:24.136626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:16.330050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.330050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.775200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.614963+0000","last_change":"2024-03-24T12:27:24.614983+0000","last_active":"2024-03-24T12:27:24.614963+0000","last_peered":"2024-03-24T12:27:24.614963+0000","last_clean":"2024-03-24T12:27:24.614963+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:27:24.614963+0000","last_undegraded":"2024-03-24T12:27:24.614963+0000","last_fullsized":"2024-03-24T12:27:24.614963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.614896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.614896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.137138+0000","last_change":"2024-03-24T12:27:06.412196+0000","last_active":"2024-03-24T12:27:24.137138+0000","last_peered":"2024-03-24T12:27:24.137138+0000","last_clean":"2024-03-24T12:27:24.137138+0000","last_became_active":"2024-03-24T12:26:05.315639+0000","last_became_peered":"2024-03-24T12:26:05.315639+0000","last_unstale":"2024-03-24T12:27:24.137138+0000","last_undegraded":"2024-03-24T12:27:24.137138+0000","last_fullsized":"2024-03-24T12:27:24.137138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:27:06.412074+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:27:06.412074+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.412074+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.288814+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478340+0000","last_change":"2024-03-24T12:27:19.658489+0000","last_active":"2024-03-24T12:27:24.478340+0000","last_peered":"2024-03-24T12:27:24.478340+0000","last_clean":"2024-03-24T12:27:24.478340+0000","last_became_active":"2024-03-24T12:26:07.343010+0000","last_became_peered":"2024-03-24T12:26:07.343010+0000","last_unstale":"2024-03-24T12:27:24.478340+0000","last_undegraded":"2024-03-24T12:27:24.478340+0000","last_fullsized":"2024-03-24T12:27:24.478340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:19.658367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:19.658367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.033169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373000+0000","last_change":"2024-03-24T12:27:12.274502+0000","last_active":"2024-03-24T12:27:24.373000+0000","last_peered":"2024-03-24T12:27:24.373000+0000","last_clean":"2024-03-24T12:27:24.373000+0000","last_became_active":"2024-03-24T12:26:09.347821+0000","last_became_peered":"2024-03-24T12:26:09.347821+0000","last_unstale":"2024-03-24T12:27:24.373000+0000","last_undegraded":"2024-03-24T12:27:24.373000+0000","last_fullsized":"2024-03-24T12:27:24.373000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.274393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.274393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.457433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":318,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437077+0000","last_change":"2024-03-24T12:27:08.342835+0000","last_active":"2024-03-24T12:27:24.437077+0000","last_peered":"2024-03-24T12:27:24.437077+0000","last_clean":"2024-03-24T12:27:24.437077+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:27:24.437077+0000","last_undegraded":"2024-03-24T12:27:24.437077+0000","last_fullsized":"2024-03-24T12:27:24.437077+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:27:08.342708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:27:08.342708+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:25.314000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:28.436469+0000","last_change":"2024-03-24T12:27:28.436490+0000","last_active":"2024-03-24T12:27:28.436469+0000","last_peered":"2024-03-24T12:27:28.436469+0000","last_clean":"2024-03-24T12:27:28.436469+0000","last_became_active":"2024-03-24T12:26:03.166467+0000","last_became_peered":"2024-03-24T12:26:03.166467+0000","last_unstale":"2024-03-24T12:27:28.436469+0000","last_undegraded":"2024-03-24T12:27:28.436469+0000","last_fullsized":"2024-03-24T12:27:28.436469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.436402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.436402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.490307+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":101,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.139654+0000","last_change":"2024-03-24T12:27:21.712724+0000","last_active":"2024-03-24T12:27:24.139654+0000","last_peered":"2024-03-24T12:27:24.139654+0000","last_clean":"2024-03-24T12:27:24.139654+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:27:24.139654+0000","last_undegraded":"2024-03-24T12:27:24.139654+0000","last_fullsized":"2024-03-24T12:27:24.139654+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:27:21.712602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.712602+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.700708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437699+0000","last_change":"2024-03-24T12:26:07.337813+0000","last_active":"2024-03-24T12:27:24.437699+0000","last_peered":"2024-03-24T12:27:24.437699+0000","last_clean":"2024-03-24T12:27:24.437699+0000","last_became_active":"2024-03-24T12:26:07.337548+0000","last_became_peered":"2024-03-24T12:26:07.337548+0000","last_unstale":"2024-03-24T12:27:24.437699+0000","last_undegraded":"2024-03-24T12:27:24.437699+0000","last_fullsized":"2024-03-24T12:27:24.437699+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.316558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:25.409425+0000","last_change":"2024-03-24T12:27:25.409449+0000","last_active":"2024-03-24T12:27:25.409425+0000","last_peered":"2024-03-24T12:27:25.409425+0000","last_clean":"2024-03-24T12:27:25.409425+0000","last_became_active":"2024-03-24T12:26:09.349435+0000","last_became_peered":"2024-03-24T12:26:09.349435+0000","last_unstale":"2024-03-24T12:27:25.409425+0000","last_undegraded":"2024-03-24T12:27:25.409425+0000","last_fullsized":"2024-03-24T12:27:25.409425+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:25.409346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.409346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.841693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:27.333389+0000","last_change":"2024-03-24T12:27:27.333417+0000","last_active":"2024-03-24T12:27:27.333389+0000","last_peered":"2024-03-24T12:27:27.333389+0000","last_clean":"2024-03-24T12:27:27.333389+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:27:27.333389+0000","last_undegraded":"2024-03-24T12:27:27.333389+0000","last_fullsized":"2024-03-24T12:27:27.333389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:27.333274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.333274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.779264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":157,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:30.514451+0000","last_change":"2024-03-24T12:27:16.322773+0000","last_active":"2024-03-24T12:27:30.514451+0000","last_peered":"2024-03-24T12:27:30.514451+0000","last_clean":"2024-03-24T12:27:30.514451+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:27:30.514451+0000","last_undegraded":"2024-03-24T12:27:30.514451+0000","last_fullsized":"2024-03-24T12:27:30.514451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:27:16.322655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.322655+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:22.799348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":82,"num_read_kb":77,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888625+0000","last_change":"2024-03-24T12:26:05.314629+0000","last_active":"2024-03-24T12:27:23.888625+0000","last_peered":"2024-03-24T12:27:23.888625+0000","last_clean":"2024-03-24T12:27:23.888625+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:27:23.888625+0000","last_undegraded":"2024-03-24T12:27:23.888625+0000","last_fullsized":"2024-03-24T12:27:23.888625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373242+0000","last_change":"2024-03-24T12:27:23.276910+0000","last_active":"2024-03-24T12:27:24.373242+0000","last_peered":"2024-03-24T12:27:24.373242+0000","last_clean":"2024-03-24T12:27:24.373242+0000","last_became_active":"2024-03-24T12:26:07.326457+0000","last_became_peered":"2024-03-24T12:26:07.326457+0000","last_unstale":"2024-03-24T12:27:24.373242+0000","last_undegraded":"2024-03-24T12:27:24.373242+0000","last_fullsized":"2024-03-24T12:27:24.373242+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.276789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.276789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.022163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.136565+0000","last_change":"2024-03-24T12:26:09.644098+0000","last_active":"2024-03-24T12:27:24.136565+0000","last_peered":"2024-03-24T12:27:24.136565+0000","last_clean":"2024-03-24T12:27:24.136565+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:27:24.136565+0000","last_undegraded":"2024-03-24T12:27:24.136565+0000","last_fullsized":"2024-03-24T12:27:24.136565+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:33.107640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:32.343958+0000","last_change":"2024-03-24T12:27:32.343982+0000","last_active":"2024-03-24T12:27:32.343958+0000","last_peered":"2024-03-24T12:27:32.343958+0000","last_clean":"2024-03-24T12:27:32.343958+0000","last_became_active":"2024-03-24T12:26:03.167244+0000","last_became_peered":"2024-03-24T12:26:03.167244+0000","last_unstale":"2024-03-24T12:27:32.343958+0000","last_undegraded":"2024-03-24T12:27:32.343958+0000","last_fullsized":"2024-03-24T12:27:32.343958+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.343880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.343880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:53.358101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":89,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889108+0000","last_change":"2024-03-24T12:26:05.357149+0000","last_active":"2024-03-24T12:27:23.889108+0000","last_peered":"2024-03-24T12:27:23.889108+0000","last_clean":"2024-03-24T12:27:23.889108+0000","last_became_active":"2024-03-24T12:26:05.356745+0000","last_became_peered":"2024-03-24T12:26:05.356745+0000","last_unstale":"2024-03-24T12:27:23.889108+0000","last_undegraded":"2024-03-24T12:27:23.889108+0000","last_fullsized":"2024-03-24T12:27:23.889108+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:32.355424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.139406+0000","last_change":"2024-03-24T12:27:23.763463+0000","last_active":"2024-03-24T12:27:24.139406+0000","last_peered":"2024-03-24T12:27:24.139406+0000","last_clean":"2024-03-24T12:27:24.139406+0000","last_became_active":"2024-03-24T12:26:07.337303+0000","last_became_peered":"2024-03-24T12:26:07.337303+0000","last_unstale":"2024-03-24T12:27:24.139406+0000","last_undegraded":"2024-03-24T12:27:24.139406+0000","last_fullsized":"2024-03-24T12:27:24.139406+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.763326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:36.340669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886634+0000","last_change":"2024-03-24T12:26:09.341274+0000","last_active":"2024-03-24T12:27:23.886634+0000","last_peered":"2024-03-24T12:27:23.886634+0000","last_clean":"2024-03-24T12:27:23.886634+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:27:23.886634+0000","last_undegraded":"2024-03-24T12:27:23.886634+0000","last_fullsized":"2024-03-24T12:27:23.886634+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.825455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:26.215424+0000","last_change":"2024-03-24T12:27:26.215446+0000","last_active":"2024-03-24T12:27:26.215424+0000","last_peered":"2024-03-24T12:27:26.215424+0000","last_clean":"2024-03-24T12:27:26.215424+0000","last_became_active":"2024-03-24T12:26:03.168184+0000","last_became_peered":"2024-03-24T12:26:03.168184+0000","last_unstale":"2024-03-24T12:27:26.215424+0000","last_undegraded":"2024-03-24T12:27:26.215424+0000","last_fullsized":"2024-03-24T12:27:26.215424+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:26.215348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.215348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.879847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.375005+0000","last_change":"2024-03-24T12:27:24.375037+0000","last_active":"2024-03-24T12:27:24.375005+0000","last_peered":"2024-03-24T12:27:24.375005+0000","last_clean":"2024-03-24T12:27:24.375005+0000","last_became_active":"2024-03-24T12:26:05.316739+0000","last_became_peered":"2024-03-24T12:26:05.316739+0000","last_unstale":"2024-03-24T12:27:24.375005+0000","last_undegraded":"2024-03-24T12:27:24.375005+0000","last_fullsized":"2024-03-24T12:27:24.375005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:24.374935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.374935+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:47.083756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":75,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.889025+0000","last_change":"2024-03-24T12:26:07.336444+0000","last_active":"2024-03-24T12:27:23.889025+0000","last_peered":"2024-03-24T12:27:23.889025+0000","last_clean":"2024-03-24T12:27:23.889025+0000","last_became_active":"2024-03-24T12:26:07.336080+0000","last_became_peered":"2024-03-24T12:26:07.336080+0000","last_unstale":"2024-03-24T12:27:23.889025+0000","last_undegraded":"2024-03-24T12:27:23.889025+0000","last_fullsized":"2024-03-24T12:27:23.889025+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.437580+0000","last_change":"2024-03-24T12:26:09.645942+0000","last_active":"2024-03-24T12:27:24.437580+0000","last_peered":"2024-03-24T12:27:24.437580+0000","last_clean":"2024-03-24T12:27:24.437580+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:27:24.437580+0000","last_undegraded":"2024-03-24T12:27:24.437580+0000","last_fullsized":"2024-03-24T12:27:24.437580+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888729+0000","last_change":"2024-03-24T12:27:12.294371+0000","last_active":"2024-03-24T12:27:23.888729+0000","last_peered":"2024-03-24T12:27:23.888729+0000","last_clean":"2024-03-24T12:27:23.888729+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:27:23.888729+0000","last_undegraded":"2024-03-24T12:27:23.888729+0000","last_fullsized":"2024-03-24T12:27:23.888729+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:12.294292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.294292+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.602110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887735+0000","last_change":"2024-03-24T12:26:05.315771+0000","last_active":"2024-03-24T12:27:23.887735+0000","last_peered":"2024-03-24T12:27:23.887735+0000","last_clean":"2024-03-24T12:27:23.887735+0000","last_became_active":"2024-03-24T12:26:05.315454+0000","last_became_peered":"2024-03-24T12:26:05.315454+0000","last_unstale":"2024-03-24T12:27:23.887735+0000","last_undegraded":"2024-03-24T12:27:23.887735+0000","last_fullsized":"2024-03-24T12:27:23.887735+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:27.552414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887223+0000","last_change":"2024-03-24T12:27:10.466243+0000","last_active":"2024-03-24T12:27:23.887223+0000","last_peered":"2024-03-24T12:27:23.887223+0000","last_clean":"2024-03-24T12:27:23.887223+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:27:23.887223+0000","last_undegraded":"2024-03-24T12:27:23.887223+0000","last_fullsized":"2024-03-24T12:27:23.887223+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:10.466147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.466147+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:12.236572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887676+0000","last_change":"2024-03-24T12:26:09.352908+0000","last_active":"2024-03-24T12:27:23.887676+0000","last_peered":"2024-03-24T12:27:23.887676+0000","last_clean":"2024-03-24T12:27:23.887676+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:27:23.887676+0000","last_undegraded":"2024-03-24T12:27:23.887676+0000","last_fullsized":"2024-03-24T12:27:23.887676+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.176448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888045+0000","last_change":"2024-03-24T12:26:03.161516+0000","last_active":"2024-03-24T12:27:23.888045+0000","last_peered":"2024-03-24T12:27:23.888045+0000","last_clean":"2024-03-24T12:27:23.888045+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:27:23.888045+0000","last_undegraded":"2024-03-24T12:27:23.888045+0000","last_fullsized":"2024-03-24T12:27:23.888045+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:26.493344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":93,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887069+0000","last_change":"2024-03-24T12:27:11.454130+0000","last_active":"2024-03-24T12:27:23.887069+0000","last_peered":"2024-03-24T12:27:23.887069+0000","last_clean":"2024-03-24T12:27:23.887069+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:27:23.887069+0000","last_undegraded":"2024-03-24T12:27:23.887069+0000","last_fullsized":"2024-03-24T12:27:23.887069+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T12:27:11.453973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:11.453973+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:18.945222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":49,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887522+0000","last_change":"2024-03-24T12:26:07.336955+0000","last_active":"2024-03-24T12:27:23.887522+0000","last_peered":"2024-03-24T12:27:23.887522+0000","last_clean":"2024-03-24T12:27:23.887522+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:27:23.887522+0000","last_undegraded":"2024-03-24T12:27:23.887522+0000","last_fullsized":"2024-03-24T12:27:23.887522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:35.459896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.372686+0000","last_change":"2024-03-24T12:26:09.351354+0000","last_active":"2024-03-24T12:27:24.372686+0000","last_peered":"2024-03-24T12:27:24.372686+0000","last_clean":"2024-03-24T12:27:24.372686+0000","last_became_active":"2024-03-24T12:26:09.351061+0000","last_became_peered":"2024-03-24T12:26:09.351061+0000","last_unstale":"2024-03-24T12:27:24.372686+0000","last_undegraded":"2024-03-24T12:27:24.372686+0000","last_fullsized":"2024-03-24T12:27:24.372686+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:34.667028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888722+0000","last_change":"2024-03-24T12:27:20.101666+0000","last_active":"2024-03-24T12:27:23.888722+0000","last_peered":"2024-03-24T12:27:23.888722+0000","last_clean":"2024-03-24T12:27:23.888722+0000","last_became_active":"2024-03-24T12:26:07.328595+0000","last_became_peered":"2024-03-24T12:26:07.328595+0000","last_unstale":"2024-03-24T12:27:23.888722+0000","last_undegraded":"2024-03-24T12:27:23.888722+0000","last_fullsized":"2024-03-24T12:27:23.888722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:20.101554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.101554+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:34.528554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.886980+0000","last_change":"2024-03-24T12:26:03.163624+0000","last_active":"2024-03-24T12:27:23.886980+0000","last_peered":"2024-03-24T12:27:23.886980+0000","last_clean":"2024-03-24T12:27:23.886980+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:27:23.886980+0000","last_undegraded":"2024-03-24T12:27:23.886980+0000","last_fullsized":"2024-03-24T12:27:23.886980+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.332538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478393+0000","last_change":"2024-03-24T12:26:16.885041+0000","last_active":"2024-03-24T12:27:24.478393+0000","last_peered":"2024-03-24T12:27:24.478393+0000","last_clean":"2024-03-24T12:27:24.478393+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:27:24.478393+0000","last_undegraded":"2024-03-24T12:27:24.478393+0000","last_fullsized":"2024-03-24T12:27:24.478393+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:31.314485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887362+0000","last_change":"2024-03-24T12:26:09.644131+0000","last_active":"2024-03-24T12:27:23.887362+0000","last_peered":"2024-03-24T12:27:23.887362+0000","last_clean":"2024-03-24T12:27:23.887362+0000","last_became_active":"2024-03-24T12:26:09.643739+0000","last_became_peered":"2024-03-24T12:26:09.643739+0000","last_unstale":"2024-03-24T12:27:23.887362+0000","last_undegraded":"2024-03-24T12:27:23.887362+0000","last_fullsized":"2024-03-24T12:27:23.887362+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:28.010994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.373205+0000","last_change":"2024-03-24T12:27:08.227925+0000","last_active":"2024-03-24T12:27:24.373205+0000","last_peered":"2024-03-24T12:27:24.373205+0000","last_clean":"2024-03-24T12:27:24.373205+0000","last_became_active":"2024-03-24T12:26:07.332766+0000","last_became_peered":"2024-03-24T12:26:07.332766+0000","last_unstale":"2024-03-24T12:27:24.373205+0000","last_undegraded":"2024-03-24T12:27:24.373205+0000","last_fullsized":"2024-03-24T12:27:24.373205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:08.227812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:08.227812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:14.184705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:31.177623+0000","last_change":"2024-03-24T12:27:31.177660+0000","last_active":"2024-03-24T12:27:31.177623+0000","last_peered":"2024-03-24T12:27:31.177623+0000","last_clean":"2024-03-24T12:27:31.177623+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:27:31.177623+0000","last_undegraded":"2024-03-24T12:27:31.177623+0000","last_fullsized":"2024-03-24T12:27:31.177623+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:31.177529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:31.177529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.777930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888468+0000","last_change":"2024-03-24T12:26:05.315880+0000","last_active":"2024-03-24T12:27:23.888468+0000","last_peered":"2024-03-24T12:27:23.888468+0000","last_clean":"2024-03-24T12:27:23.888468+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:27:23.888468+0000","last_undegraded":"2024-03-24T12:27:23.888468+0000","last_fullsized":"2024-03-24T12:27:23.888468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478465+0000","last_change":"2024-03-24T12:26:09.348104+0000","last_active":"2024-03-24T12:27:24.478465+0000","last_peered":"2024-03-24T12:27:24.478465+0000","last_clean":"2024-03-24T12:27:24.478465+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:27:24.478465+0000","last_undegraded":"2024-03-24T12:27:24.478465+0000","last_fullsized":"2024-03-24T12:27:24.478465+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.236528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.137182+0000","last_change":"2024-03-24T12:27:23.411568+0000","last_active":"2024-03-24T12:27:24.137182+0000","last_peered":"2024-03-24T12:27:24.137182+0000","last_clean":"2024-03-24T12:27:24.137182+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:27:24.137182+0000","last_undegraded":"2024-03-24T12:27:24.137182+0000","last_fullsized":"2024-03-24T12:27:24.137182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.411477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.411477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.665294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.139209+0000","last_change":"2024-03-24T12:27:12.636522+0000","last_active":"2024-03-24T12:27:24.139209+0000","last_peered":"2024-03-24T12:27:24.139209+0000","last_clean":"2024-03-24T12:27:24.139209+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:27:24.139209+0000","last_undegraded":"2024-03-24T12:27:24.139209+0000","last_fullsized":"2024-03-24T12:27:24.139209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.636423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.636423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.538773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888362+0000","last_change":"2024-03-24T12:26:05.316692+0000","last_active":"2024-03-24T12:27:23.888362+0000","last_peered":"2024-03-24T12:27:23.888362+0000","last_clean":"2024-03-24T12:27:23.888362+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:27:23.888362+0000","last_undegraded":"2024-03-24T12:27:23.888362+0000","last_fullsized":"2024-03-24T12:27:23.888362+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.077480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":53,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.888702+0000","last_change":"2024-03-24T12:27:10.145724+0000","last_active":"2024-03-24T12:27:23.888702+0000","last_peered":"2024-03-24T12:27:23.888702+0000","last_clean":"2024-03-24T12:27:23.888702+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:27:23.888702+0000","last_undegraded":"2024-03-24T12:27:23.888702+0000","last_fullsized":"2024-03-24T12:27:23.888702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:10.145613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.145613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:22.594263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":83,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:24.478237+0000","last_change":"2024-03-24T12:27:21.649282+0000","last_active":"2024-03-24T12:27:24.478237+0000","last_peered":"2024-03-24T12:27:24.478237+0000","last_clean":"2024-03-24T12:27:24.478237+0000","last_became_active":"2024-03-24T12:26:07.334798+0000","last_became_peered":"2024-03-24T12:26:07.334798+0000","last_unstale":"2024-03-24T12:27:24.478237+0000","last_undegraded":"2024-03-24T12:27:24.478237+0000","last_fullsized":"2024-03-24T12:27:24.478237+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:21.649181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.649181+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.389315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887325+0000","last_change":"2024-03-24T12:26:03.162618+0000","last_active":"2024-03-24T12:27:23.887325+0000","last_peered":"2024-03-24T12:27:23.887325+0000","last_clean":"2024-03-24T12:27:23.887325+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:27:23.887325+0000","last_undegraded":"2024-03-24T12:27:23.887325+0000","last_fullsized":"2024-03-24T12:27:23.887325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:30.490100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T12:27:23.887782+0000","last_change":"2024-03-24T12:27:22.098052+0000","last_active":"2024-03-24T12:27:23.887782+0000","last_peered":"2024-03-24T12:27:23.887782+0000","last_clean":"2024-03-24T12:27:23.887782+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:27:23.887782+0000","last_undegraded":"2024-03-24T12:27:23.887782+0000","last_fullsized":"2024-03-24T12:27:23.887782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:22.097943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:22.097943+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.541070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":424,"ondisk_log_size":424,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":77,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037782,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28412,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716740,"statfs":{"total":95995035648,"available":95965941760,"internally_reserved":0,"allocated":1429504,"data_stored":951998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332124,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32548,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712604,"statfs":{"total":95995035648,"available":95961706496,"internally_reserved":0,"allocated":1404928,"data_stored":951410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":362020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32572,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712580,"statfs":{"total":95995035648,"available":95961681920,"internally_reserved":0,"allocated":1429504,"data_stored":951342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31968,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713184,"statfs":{"total":95995035648,"available":95962300416,"internally_reserved":0,"allocated":811008,"data_stored":360502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117034,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27848,"kb_used_data":832,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717304,"statfs":{"total":95995035648,"available":95966519296,"internally_reserved":0,"allocated":851968,"data_stored":361946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920806,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27832,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717320,"statfs":{"total":95995035648,"available":95966535680,"internally_reserved":0,"allocated":835584,"data_stored":360772,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626465,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27820,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717332,"statfs":{"total":95995035648,"available":95966547968,"internally_reserved":0,"allocated":823296,"data_stored":361876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:27:34.076 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T12:27:34.076 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T12:27:34.077 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T12:27:34.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T12:27:34.325 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:27:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:27:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: 4.1d scrub starts 2024-03-24T12:27:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: 4.1d scrub ok 2024-03-24T12:27:34.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: 5.b scrub starts 2024-03-24T12:27:34.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: 5.b scrub ok 2024-03-24T12:27:34.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: 6.1d scrub starts 2024-03-24T12:27:34.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: 6.1d scrub ok 2024-03-24T12:27:34.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: 6.3 scrub starts 2024-03-24T12:27:34.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:33 smithi190 ceph-mon[33748]: 6.3 scrub ok 2024-03-24T12:27:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:27:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:27:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[29919]: from='client.24617 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:27:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:27:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[29919]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T12:27:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[29919]: 6.1c scrub starts 2024-03-24T12:27:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[29919]: 6.1c scrub ok 2024-03-24T12:27:35.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[29919]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T12:27:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:27:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:27:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[41500]: from='client.24617 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:27:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:27:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[41500]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T12:27:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[41500]: 6.1c scrub starts 2024-03-24T12:27:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[41500]: 6.1c scrub ok 2024-03-24T12:27:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:34 smithi029 ceph-mon[41500]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T12:27:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:34 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:27:35.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:34 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:27:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:34 smithi190 ceph-mon[33748]: from='client.24617 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:34 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:27:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:34 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:27:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:34 smithi190 ceph-mon[33748]: osdmap e91: 8 total, 8 up, 8 in 2024-03-24T12:27:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:34 smithi190 ceph-mon[33748]: 6.1c scrub starts 2024-03-24T12:27:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:34 smithi190 ceph-mon[33748]: 6.1c scrub ok 2024-03-24T12:27:35.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:34 smithi190 ceph-mon[33748]: osdmap e92: 8 total, 8 up, 8 in 2024-03-24T12:27:36.230 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[29919]: 3.16 scrub starts 2024-03-24T12:27:36.230 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[29919]: 3.16 scrub ok 2024-03-24T12:27:36.230 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[29919]: 5.18 scrub starts 2024-03-24T12:27:36.231 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[29919]: 5.18 scrub ok 2024-03-24T12:27:36.231 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[29919]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:36.231 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:27:36.231 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:27:36.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[41500]: 3.16 scrub starts 2024-03-24T12:27:36.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[41500]: 3.16 scrub ok 2024-03-24T12:27:36.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[41500]: 5.18 scrub starts 2024-03-24T12:27:36.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[41500]: 5.18 scrub ok 2024-03-24T12:27:36.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[41500]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:36.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:27:36.232 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:35 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:27:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:35 smithi190 ceph-mon[33748]: 3.16 scrub starts 2024-03-24T12:27:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:35 smithi190 ceph-mon[33748]: 3.16 scrub ok 2024-03-24T12:27:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:35 smithi190 ceph-mon[33748]: 5.18 scrub starts 2024-03-24T12:27:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:35 smithi190 ceph-mon[33748]: 5.18 scrub ok 2024-03-24T12:27:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:35 smithi190 ceph-mon[33748]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:35 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:27:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:35 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:27:36.592 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:27:36.592 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T12:27:37.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:36 smithi029 ceph-mon[29919]: 6.1b scrub starts 2024-03-24T12:27:37.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:36 smithi029 ceph-mon[29919]: 6.1b scrub ok 2024-03-24T12:27:37.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:36 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:27:37.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:36 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:27:37.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:36 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3914131306' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T12:27:37.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:36 smithi029 ceph-mon[41500]: 6.1b scrub starts 2024-03-24T12:27:37.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:36 smithi029 ceph-mon[41500]: 6.1b scrub ok 2024-03-24T12:27:37.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:36 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:27:37.259 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:36 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:27:37.259 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:36 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3914131306' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T12:27:37.288 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T12:27:37.288 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T12:27:37.289 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-24T12:27:37.300 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-24T12:27:37.300 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr1': False, 'ms bind msgr2': True, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'squid', 'sha1': 'f73f1dfd96849300c4322bd7259a3ab37f94c200'}} 2024-03-24T12:27:37.300 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-24T12:27:37.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-24T12:27:37.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:36 smithi190 ceph-mon[33748]: 6.1b scrub starts 2024-03-24T12:27:37.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:36 smithi190 ceph-mon[33748]: 6.1b scrub ok 2024-03-24T12:27:37.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:36 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:27:37.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:36 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:27:37.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:36 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3914131306' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T12:27:37.554 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:37.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:27:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:27:37.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:27:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:27:38.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:37 smithi190 ceph-mon[33748]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:38.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:37 smithi190 ceph-mon[33748]: 3.7 scrub starts 2024-03-24T12:27:38.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:37 smithi190 ceph-mon[33748]: 3.7 scrub ok 2024-03-24T12:27:38.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:37 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:27:38.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:37 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:27:38.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:37 smithi190 ceph-mon[33748]: 3.d scrub starts 2024-03-24T12:27:38.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:37 smithi190 ceph-mon[33748]: 3.d scrub ok 2024-03-24T12:27:38.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[29919]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:38.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[29919]: 3.7 scrub starts 2024-03-24T12:27:38.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[29919]: 3.7 scrub ok 2024-03-24T12:27:38.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:27:38.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:27:38.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[29919]: 3.d scrub starts 2024-03-24T12:27:38.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[29919]: 3.d scrub ok 2024-03-24T12:27:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[41500]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[41500]: 3.7 scrub starts 2024-03-24T12:27:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[41500]: 3.7 scrub ok 2024-03-24T12:27:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:27:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:27:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[41500]: 3.d scrub starts 2024-03-24T12:27:38.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:37 smithi029 ceph-mon[41500]: 3.d scrub ok 2024-03-24T12:27:39.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[29919]: 6.17 scrub starts 2024-03-24T12:27:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[29919]: 6.17 scrub ok 2024-03-24T12:27:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[29919]: 3.1e scrub starts 2024-03-24T12:27:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[29919]: 3.1e scrub ok 2024-03-24T12:27:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[29919]: 4.6 scrub starts 2024-03-24T12:27:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[29919]: 4.6 scrub ok 2024-03-24T12:27:39.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[41500]: 6.17 scrub starts 2024-03-24T12:27:39.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[41500]: 6.17 scrub ok 2024-03-24T12:27:39.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:39.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[41500]: 3.1e scrub starts 2024-03-24T12:27:39.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[41500]: 3.1e scrub ok 2024-03-24T12:27:39.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[41500]: 4.6 scrub starts 2024-03-24T12:27:39.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:38 smithi029 ceph-mon[41500]: 4.6 scrub ok 2024-03-24T12:27:39.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:38 smithi190 ceph-mon[33748]: 6.17 scrub starts 2024-03-24T12:27:39.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:38 smithi190 ceph-mon[33748]: 6.17 scrub ok 2024-03-24T12:27:39.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:38 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:39.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:38 smithi190 ceph-mon[33748]: 3.1e scrub starts 2024-03-24T12:27:39.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:38 smithi190 ceph-mon[33748]: 3.1e scrub ok 2024-03-24T12:27:39.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:38 smithi190 ceph-mon[33748]: 4.6 scrub starts 2024-03-24T12:27:39.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:38 smithi190 ceph-mon[33748]: 4.6 scrub ok 2024-03-24T12:27:39.763 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:39] ENGINE Bus STOPPING 2024-03-24T12:27:39.763 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:27:39.763 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:39] ENGINE Bus STOPPED 2024-03-24T12:27:39.763 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:39] ENGINE Bus STARTING 2024-03-24T12:27:39.764 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:39] ENGINE Serving on http://:::9283 2024-03-24T12:27:40.032 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:27:39] ENGINE Bus STARTED 2024-03-24T12:27:40.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:27:40.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:27:40.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[29919]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:40.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[29919]: 6.13 scrub starts 2024-03-24T12:27:40.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[29919]: 6.13 scrub ok 2024-03-24T12:27:40.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[29919]: 5.a deep-scrub starts 2024-03-24T12:27:40.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[29919]: 5.a deep-scrub ok 2024-03-24T12:27:40.033 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2665060751' entity='client.admin' 2024-03-24T12:27:40.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:27:40.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:27:40.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[41500]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:40.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[41500]: 6.13 scrub starts 2024-03-24T12:27:40.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[41500]: 6.13 scrub ok 2024-03-24T12:27:40.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[41500]: 5.a deep-scrub starts 2024-03-24T12:27:40.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[41500]: 5.a deep-scrub ok 2024-03-24T12:27:40.034 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:39 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2665060751' entity='client.admin' 2024-03-24T12:27:40.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:39 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:27:40.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:39 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:27:40.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:39 smithi190 ceph-mon[33748]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:40.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:39 smithi190 ceph-mon[33748]: 6.13 scrub starts 2024-03-24T12:27:40.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:39 smithi190 ceph-mon[33748]: 6.13 scrub ok 2024-03-24T12:27:40.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:39 smithi190 ceph-mon[33748]: 5.a deep-scrub starts 2024-03-24T12:27:40.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:39 smithi190 ceph-mon[33748]: 5.a deep-scrub ok 2024-03-24T12:27:40.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:39 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2665060751' entity='client.admin' 2024-03-24T12:27:40.446 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-24T12:27:40.446 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T12:27:40.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:27:40.693 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:41.110 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:40 smithi029 ceph-mon[41500]: 5.1c deep-scrub starts 2024-03-24T12:27:41.110 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:40 smithi029 ceph-mon[41500]: 5.1c deep-scrub ok 2024-03-24T12:27:41.110 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:40 smithi029 ceph-mon[41500]: 5.17 scrub starts 2024-03-24T12:27:41.110 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:40 smithi029 ceph-mon[41500]: 5.17 scrub ok 2024-03-24T12:27:41.110 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:40 smithi029 ceph-mon[41500]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:41.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:40 smithi029 ceph-mon[29919]: 5.1c deep-scrub starts 2024-03-24T12:27:41.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:40 smithi029 ceph-mon[29919]: 5.1c deep-scrub ok 2024-03-24T12:27:41.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:40 smithi029 ceph-mon[29919]: 5.17 scrub starts 2024-03-24T12:27:41.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:40 smithi029 ceph-mon[29919]: 5.17 scrub ok 2024-03-24T12:27:41.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:40 smithi029 ceph-mon[29919]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:41.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:40 smithi190 ceph-mon[33748]: 5.1c deep-scrub starts 2024-03-24T12:27:41.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:40 smithi190 ceph-mon[33748]: 5.1c deep-scrub ok 2024-03-24T12:27:41.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:40 smithi190 ceph-mon[33748]: 5.17 scrub starts 2024-03-24T12:27:41.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:40 smithi190 ceph-mon[33748]: 5.17 scrub ok 2024-03-24T12:27:41.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:40 smithi190 ceph-mon[33748]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:27:41.419 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:27:41 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:27:41.127Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003500246s 2024-03-24T12:27:42.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:41 smithi029 ceph-mon[29919]: 6.c scrub starts 2024-03-24T12:27:42.098 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:41 smithi029 ceph-mon[29919]: 6.c scrub ok 2024-03-24T12:27:42.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:41 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:27:42.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:41 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:27:42.099 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:41 smithi029 ceph-mon[41500]: 6.c scrub starts 2024-03-24T12:27:42.099 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:41 smithi029 ceph-mon[41500]: 6.c scrub ok 2024-03-24T12:27:42.099 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:41 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:27:42.099 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:41 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:27:42.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:41 smithi190 ceph-mon[33748]: 6.c scrub starts 2024-03-24T12:27:42.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:41 smithi190 ceph-mon[33748]: 6.c scrub ok 2024-03-24T12:27:42.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:41 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:27:42.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:41 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:27:42.864 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:27:42.868 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:27:43.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:42 smithi190 ceph-mon[33748]: 6.15 scrub starts 2024-03-24T12:27:43.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:42 smithi190 ceph-mon[33748]: 6.15 scrub ok 2024-03-24T12:27:43.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:42 smithi190 ceph-mon[33748]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:43.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:42 smithi190 ceph-mon[33748]: 5.19 scrub starts 2024-03-24T12:27:43.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:42 smithi190 ceph-mon[33748]: 5.19 scrub ok 2024-03-24T12:27:43.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:42 smithi190 ceph-mon[33748]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:43.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[29919]: 6.15 scrub starts 2024-03-24T12:27:43.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[29919]: 6.15 scrub ok 2024-03-24T12:27:43.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[29919]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:43.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[29919]: 5.19 scrub starts 2024-03-24T12:27:43.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[29919]: 5.19 scrub ok 2024-03-24T12:27:43.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[29919]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:43.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[41500]: 6.15 scrub starts 2024-03-24T12:27:43.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[41500]: 6.15 scrub ok 2024-03-24T12:27:43.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[41500]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:43.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[41500]: 5.19 scrub starts 2024-03-24T12:27:43.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[41500]: 5.19 scrub ok 2024-03-24T12:27:43.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:42 smithi029 ceph-mon[41500]: from='client.24635 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:43.640 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-24T12:27:42.392041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1026,"num_read_kb":872,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":632,"ondisk_log_size":632,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":237052,"kb_used_data":8444,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724164,"statfs":{"total":767960285184,"available":767717543936,"internally_reserved":0,"allocated":8646656,"data_stored":4765610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.466346"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870600+0000","last_change":"2024-03-24T12:26:09.350829+0000","last_active":"2024-03-24T12:27:34.870600+0000","last_peered":"2024-03-24T12:27:34.870600+0000","last_clean":"2024-03-24T12:27:34.870600+0000","last_became_active":"2024-03-24T12:26:09.350521+0000","last_became_peered":"2024-03-24T12:26:09.350521+0000","last_unstale":"2024-03-24T12:27:34.870600+0000","last_undegraded":"2024-03-24T12:27:34.870600+0000","last_fullsized":"2024-03-24T12:27:34.870600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870389+0000","last_change":"2024-03-24T12:26:07.335814+0000","last_active":"2024-03-24T12:27:34.870389+0000","last_peered":"2024-03-24T12:27:34.870389+0000","last_clean":"2024-03-24T12:27:34.870389+0000","last_became_active":"2024-03-24T12:26:07.335517+0000","last_became_peered":"2024-03-24T12:26:07.335517+0000","last_unstale":"2024-03-24T12:27:34.870389+0000","last_undegraded":"2024-03-24T12:27:34.870389+0000","last_fullsized":"2024-03-24T12:27:34.870389+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044306+0000","last_change":"2024-03-24T12:26:03.171110+0000","last_active":"2024-03-24T12:27:35.044306+0000","last_peered":"2024-03-24T12:27:35.044306+0000","last_clean":"2024-03-24T12:27:35.044306+0000","last_became_active":"2024-03-24T12:26:03.170822+0000","last_became_peered":"2024-03-24T12:26:03.170822+0000","last_unstale":"2024-03-24T12:27:35.044306+0000","last_undegraded":"2024-03-24T12:27:35.044306+0000","last_fullsized":"2024-03-24T12:27:35.044306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870319+0000","last_change":"2024-03-24T12:27:24.141489+0000","last_active":"2024-03-24T12:27:34.870319+0000","last_peered":"2024-03-24T12:27:34.870319+0000","last_clean":"2024-03-24T12:27:34.870319+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:27:34.870319+0000","last_undegraded":"2024-03-24T12:27:34.870319+0000","last_fullsized":"2024-03-24T12:27:34.870319+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:24.141325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.141325+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.796727+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494713+0000","last_change":"2024-03-24T12:27:35.494737+0000","last_active":"2024-03-24T12:27:35.494713+0000","last_peered":"2024-03-24T12:27:35.494713+0000","last_clean":"2024-03-24T12:27:35.494713+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:27:35.494713+0000","last_undegraded":"2024-03-24T12:27:35.494713+0000","last_fullsized":"2024-03-24T12:27:35.494713+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:35.494635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:35.494635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:48.670706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870321+0000","last_change":"2024-03-24T12:27:34.234548+0000","last_active":"2024-03-24T12:27:34.870321+0000","last_peered":"2024-03-24T12:27:34.870321+0000","last_clean":"2024-03-24T12:27:34.870321+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:27:34.870321+0000","last_undegraded":"2024-03-24T12:27:34.870321+0000","last_fullsized":"2024-03-24T12:27:34.870321+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:34.234417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.234417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.166756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:37.470875+0000","last_change":"2024-03-24T12:27:37.470895+0000","last_active":"2024-03-24T12:27:37.470875+0000","last_peered":"2024-03-24T12:27:37.470875+0000","last_clean":"2024-03-24T12:27:37.470875+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:27:37.470875+0000","last_undegraded":"2024-03-24T12:27:37.470875+0000","last_fullsized":"2024-03-24T12:27:37.470875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:37.470809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:37.470809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:01.645277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"90'22","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493873+0000","last_change":"2024-03-24T12:27:25.396689+0000","last_active":"2024-03-24T12:27:35.493873+0000","last_peered":"2024-03-24T12:27:35.493873+0000","last_clean":"2024-03-24T12:27:35.493873+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:27:35.493873+0000","last_undegraded":"2024-03-24T12:27:35.493873+0000","last_fullsized":"2024-03-24T12:27:35.493873+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.396514+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.161411+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043813+0000","last_change":"2024-03-24T12:27:23.289755+0000","last_active":"2024-03-24T12:27:35.043813+0000","last_peered":"2024-03-24T12:27:35.043813+0000","last_clean":"2024-03-24T12:27:35.043813+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:27:35.043813+0000","last_undegraded":"2024-03-24T12:27:35.043813+0000","last_fullsized":"2024-03-24T12:27:35.043813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.289666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.684236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569461+0000","last_change":"2024-03-24T12:26:07.331880+0000","last_active":"2024-03-24T12:27:35.569461+0000","last_peered":"2024-03-24T12:27:35.569461+0000","last_clean":"2024-03-24T12:27:35.569461+0000","last_became_active":"2024-03-24T12:26:07.331562+0000","last_became_peered":"2024-03-24T12:26:07.331562+0000","last_unstale":"2024-03-24T12:27:35.569461+0000","last_undegraded":"2024-03-24T12:27:35.569461+0000","last_fullsized":"2024-03-24T12:27:35.569461+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570257+0000","last_change":"2024-03-24T12:27:20.255011+0000","last_active":"2024-03-24T12:27:35.570257+0000","last_peered":"2024-03-24T12:27:35.570257+0000","last_clean":"2024-03-24T12:27:35.570257+0000","last_became_active":"2024-03-24T12:26:03.161544+0000","last_became_peered":"2024-03-24T12:26:03.161544+0000","last_unstale":"2024-03-24T12:27:35.570257+0000","last_undegraded":"2024-03-24T12:27:35.570257+0000","last_fullsized":"2024-03-24T12:27:35.570257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:20.254898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.254898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.151491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870711+0000","last_change":"2024-03-24T12:26:05.318208+0000","last_active":"2024-03-24T12:27:34.870711+0000","last_peered":"2024-03-24T12:27:34.870711+0000","last_clean":"2024-03-24T12:27:34.870711+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:27:34.870711+0000","last_undegraded":"2024-03-24T12:27:34.870711+0000","last_fullsized":"2024-03-24T12:27:34.870711+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569689+0000","last_change":"2024-03-24T12:27:21.260071+0000","last_active":"2024-03-24T12:27:35.569689+0000","last_peered":"2024-03-24T12:27:35.569689+0000","last_clean":"2024-03-24T12:27:35.569689+0000","last_became_active":"2024-03-24T12:26:09.350721+0000","last_became_peered":"2024-03-24T12:26:09.350721+0000","last_unstale":"2024-03-24T12:27:35.569689+0000","last_undegraded":"2024-03-24T12:27:35.569689+0000","last_fullsized":"2024-03-24T12:27:35.569689+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.259962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.801612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391054+0000","last_change":"2024-03-24T12:27:17.365794+0000","last_active":"2024-03-24T12:27:35.391054+0000","last_peered":"2024-03-24T12:27:35.391054+0000","last_clean":"2024-03-24T12:27:35.391054+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:27:35.391054+0000","last_undegraded":"2024-03-24T12:27:35.391054+0000","last_fullsized":"2024-03-24T12:27:35.391054+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.365696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.873148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570142+0000","last_change":"2024-03-24T12:27:10.251781+0000","last_active":"2024-03-24T12:27:35.570142+0000","last_peered":"2024-03-24T12:27:35.570142+0000","last_clean":"2024-03-24T12:27:35.570142+0000","last_became_active":"2024-03-24T12:26:03.159997+0000","last_became_peered":"2024-03-24T12:26:03.159997+0000","last_unstale":"2024-03-24T12:27:35.570142+0000","last_undegraded":"2024-03-24T12:27:35.570142+0000","last_fullsized":"2024-03-24T12:27:35.570142+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.251686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.902316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":75,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:38.276134+0000","last_change":"2024-03-24T12:27:38.276194+0000","last_active":"2024-03-24T12:27:38.276134+0000","last_peered":"2024-03-24T12:27:38.276134+0000","last_clean":"2024-03-24T12:27:38.276134+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:27:38.276134+0000","last_undegraded":"2024-03-24T12:27:38.276134+0000","last_fullsized":"2024-03-24T12:27:38.276134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T12:27:38.276034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.276034+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:06.748494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871176+0000","last_change":"2024-03-24T12:27:25.164561+0000","last_active":"2024-03-24T12:27:34.871176+0000","last_peered":"2024-03-24T12:27:34.871176+0000","last_clean":"2024-03-24T12:27:34.871176+0000","last_became_active":"2024-03-24T12:26:09.643671+0000","last_became_peered":"2024-03-24T12:26:09.643671+0000","last_unstale":"2024-03-24T12:27:34.871176+0000","last_undegraded":"2024-03-24T12:27:34.871176+0000","last_fullsized":"2024-03-24T12:27:34.871176+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:25.164460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.164460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.284324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871818+0000","last_change":"2024-03-24T12:27:28.397274+0000","last_active":"2024-03-24T12:27:34.871818+0000","last_peered":"2024-03-24T12:27:34.871818+0000","last_clean":"2024-03-24T12:27:34.871818+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:27:34.871818+0000","last_undegraded":"2024-03-24T12:27:34.871818+0000","last_fullsized":"2024-03-24T12:27:34.871818+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.397179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.397179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.594543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044310+0000","last_change":"2024-03-24T12:27:28.361014+0000","last_active":"2024-03-24T12:27:35.044310+0000","last_peered":"2024-03-24T12:27:35.044310+0000","last_clean":"2024-03-24T12:27:35.044310+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:27:35.044310+0000","last_undegraded":"2024-03-24T12:27:35.044310+0000","last_fullsized":"2024-03-24T12:27:35.044310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.360926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.835122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870865+0000","last_change":"2024-03-24T12:27:33.582805+0000","last_active":"2024-03-24T12:27:34.870865+0000","last_peered":"2024-03-24T12:27:34.870865+0000","last_clean":"2024-03-24T12:27:34.870865+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:27:34.870865+0000","last_undegraded":"2024-03-24T12:27:34.870865+0000","last_fullsized":"2024-03-24T12:27:34.870865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:27:33.582711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:33.582711+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.312118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494584+0000","last_change":"2024-03-24T12:27:30.441572+0000","last_active":"2024-03-24T12:27:35.494584+0000","last_peered":"2024-03-24T12:27:35.494584+0000","last_clean":"2024-03-24T12:27:35.494584+0000","last_became_active":"2024-03-24T12:26:09.643738+0000","last_became_peered":"2024-03-24T12:26:09.643738+0000","last_unstale":"2024-03-24T12:27:35.494584+0000","last_undegraded":"2024-03-24T12:27:35.494584+0000","last_fullsized":"2024-03-24T12:27:35.494584+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:30.441497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:30.441497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.867059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:39.295925+0000","last_change":"2024-03-24T12:27:39.295946+0000","last_active":"2024-03-24T12:27:39.295925+0000","last_peered":"2024-03-24T12:27:39.295925+0000","last_clean":"2024-03-24T12:27:39.295925+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:27:39.295925+0000","last_undegraded":"2024-03-24T12:27:39.295925+0000","last_fullsized":"2024-03-24T12:27:39.295925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:27:39.295864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:07.017784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870775+0000","last_change":"2024-03-24T12:26:03.160687+0000","last_active":"2024-03-24T12:27:34.870775+0000","last_peered":"2024-03-24T12:27:34.870775+0000","last_clean":"2024-03-24T12:27:34.870775+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:27:34.870775+0000","last_undegraded":"2024-03-24T12:27:34.870775+0000","last_fullsized":"2024-03-24T12:27:34.870775+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":88,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494411+0000","last_change":"2024-03-24T12:27:32.464080+0000","last_active":"2024-03-24T12:27:35.494411+0000","last_peered":"2024-03-24T12:27:35.494411+0000","last_clean":"2024-03-24T12:27:35.494411+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:27:35.494411+0000","last_undegraded":"2024-03-24T12:27:35.494411+0000","last_fullsized":"2024-03-24T12:27:35.494411+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:32.463972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.463972+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:58.687386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391678+0000","last_change":"2024-03-24T12:27:34.409721+0000","last_active":"2024-03-24T12:27:35.391678+0000","last_peered":"2024-03-24T12:27:35.391678+0000","last_clean":"2024-03-24T12:27:35.391678+0000","last_became_active":"2024-03-24T12:26:09.343786+0000","last_became_peered":"2024-03-24T12:26:09.343786+0000","last_unstale":"2024-03-24T12:27:35.391678+0000","last_undegraded":"2024-03-24T12:27:35.391678+0000","last_fullsized":"2024-03-24T12:27:35.391678+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:27:34.409577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.409577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:55.581370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":92,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:36.723422+0000","last_change":"2024-03-24T12:27:36.723422+0000","last_active":"2024-03-24T12:27:36.723422+0000","last_peered":"2024-03-24T12:27:36.723422+0000","last_clean":"2024-03-24T12:27:36.723422+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:27:36.723422+0000","last_undegraded":"2024-03-24T12:27:36.723422+0000","last_fullsized":"2024-03-24T12:27:36.723422+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:36.723265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:36.723265+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.467176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870414+0000","last_change":"2024-03-24T12:27:27.409032+0000","last_active":"2024-03-24T12:27:34.870414+0000","last_peered":"2024-03-24T12:27:34.870414+0000","last_clean":"2024-03-24T12:27:34.870414+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:27:34.870414+0000","last_undegraded":"2024-03-24T12:27:34.870414+0000","last_fullsized":"2024-03-24T12:27:34.870414+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:27.408937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.408937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.920390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044300+0000","last_change":"2024-03-24T12:27:11.269339+0000","last_active":"2024-03-24T12:27:35.044300+0000","last_peered":"2024-03-24T12:27:35.044300+0000","last_clean":"2024-03-24T12:27:35.044300+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:27:35.044300+0000","last_undegraded":"2024-03-24T12:27:35.044300+0000","last_fullsized":"2024-03-24T12:27:35.044300+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:11.269181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:11.269181+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:34.970841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871884+0000","last_change":"2024-03-24T12:27:33.377434+0000","last_active":"2024-03-24T12:27:34.871884+0000","last_peered":"2024-03-24T12:27:34.871884+0000","last_clean":"2024-03-24T12:27:34.871884+0000","last_became_active":"2024-03-24T12:26:09.348355+0000","last_became_peered":"2024-03-24T12:26:09.348355+0000","last_unstale":"2024-03-24T12:27:34.871884+0000","last_undegraded":"2024-03-24T12:27:34.871884+0000","last_fullsized":"2024-03-24T12:27:34.871884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:33.377351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:33.377351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.631210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044456+0000","last_change":"2024-03-24T12:26:07.331932+0000","last_active":"2024-03-24T12:27:35.044456+0000","last_peered":"2024-03-24T12:27:35.044456+0000","last_clean":"2024-03-24T12:27:35.044456+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:27:35.044456+0000","last_undegraded":"2024-03-24T12:27:35.044456+0000","last_fullsized":"2024-03-24T12:27:35.044456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493361+0000","last_change":"2024-03-24T12:27:12.437790+0000","last_active":"2024-03-24T12:27:35.493361+0000","last_peered":"2024-03-24T12:27:35.493361+0000","last_clean":"2024-03-24T12:27:35.493361+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:27:35.493361+0000","last_undegraded":"2024-03-24T12:27:35.493361+0000","last_fullsized":"2024-03-24T12:27:35.493361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.437708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:25.841012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":84,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.495063+0000","last_change":"2024-03-24T12:27:26.763428+0000","last_active":"2024-03-24T12:27:35.495063+0000","last_peered":"2024-03-24T12:27:35.495063+0000","last_clean":"2024-03-24T12:27:35.495063+0000","last_became_active":"2024-03-24T12:26:05.317587+0000","last_became_peered":"2024-03-24T12:26:05.317587+0000","last_unstale":"2024-03-24T12:27:35.495063+0000","last_undegraded":"2024-03-24T12:27:35.495063+0000","last_fullsized":"2024-03-24T12:27:35.495063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:26.763317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.763317+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.728888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391387+0000","last_change":"2024-03-24T12:26:09.345188+0000","last_active":"2024-03-24T12:27:35.391387+0000","last_peered":"2024-03-24T12:27:35.391387+0000","last_clean":"2024-03-24T12:27:35.391387+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:27:35.391387+0000","last_undegraded":"2024-03-24T12:27:35.391387+0000","last_fullsized":"2024-03-24T12:27:35.391387+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:37.908175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494107+0000","last_change":"2024-03-24T12:27:34.747936+0000","last_active":"2024-03-24T12:27:35.494107+0000","last_peered":"2024-03-24T12:27:35.494107+0000","last_clean":"2024-03-24T12:27:35.494107+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:27:35.494107+0000","last_undegraded":"2024-03-24T12:27:35.494107+0000","last_fullsized":"2024-03-24T12:27:35.494107+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:34.747834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.747834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:50.783426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044307+0000","last_change":"2024-03-24T12:26:03.171104+0000","last_active":"2024-03-24T12:27:35.044307+0000","last_peered":"2024-03-24T12:27:35.044307+0000","last_clean":"2024-03-24T12:27:35.044307+0000","last_became_active":"2024-03-24T12:26:03.170395+0000","last_became_peered":"2024-03-24T12:26:03.170395+0000","last_unstale":"2024-03-24T12:27:35.044307+0000","last_undegraded":"2024-03-24T12:27:35.044307+0000","last_fullsized":"2024-03-24T12:27:35.044307+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493391+0000","last_change":"2024-03-24T12:27:15.467117+0000","last_active":"2024-03-24T12:27:35.493391+0000","last_peered":"2024-03-24T12:27:35.493391+0000","last_clean":"2024-03-24T12:27:35.493391+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:27:35.493391+0000","last_undegraded":"2024-03-24T12:27:35.493391+0000","last_fullsized":"2024-03-24T12:27:35.493391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:27:15.466966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:15.466966+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:23.327159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:38.510313+0000","last_change":"2024-03-24T12:27:38.510336+0000","last_active":"2024-03-24T12:27:38.510313+0000","last_peered":"2024-03-24T12:27:38.510313+0000","last_clean":"2024-03-24T12:27:38.510313+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:27:38.510313+0000","last_undegraded":"2024-03-24T12:27:38.510313+0000","last_fullsized":"2024-03-24T12:27:38.510313+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:38.510237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.510237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.100244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391030+0000","last_change":"2024-03-24T12:27:21.335557+0000","last_active":"2024-03-24T12:27:35.391030+0000","last_peered":"2024-03-24T12:27:35.391030+0000","last_clean":"2024-03-24T12:27:35.391030+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:27:35.391030+0000","last_undegraded":"2024-03-24T12:27:35.391030+0000","last_fullsized":"2024-03-24T12:27:35.391030+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:21.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.789829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":75,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570411+0000","last_change":"2024-03-24T12:27:34.193568+0000","last_active":"2024-03-24T12:27:35.570411+0000","last_peered":"2024-03-24T12:27:35.570411+0000","last_clean":"2024-03-24T12:27:35.570411+0000","last_became_active":"2024-03-24T12:26:03.165083+0000","last_became_peered":"2024-03-24T12:26:03.165083+0000","last_unstale":"2024-03-24T12:27:35.570411+0000","last_undegraded":"2024-03-24T12:27:35.570411+0000","last_fullsized":"2024-03-24T12:27:35.570411+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:34.193480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.193480+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:50.648254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":84,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493768+0000","last_change":"2024-03-24T12:27:16.497904+0000","last_active":"2024-03-24T12:27:35.493768+0000","last_peered":"2024-03-24T12:27:35.493768+0000","last_clean":"2024-03-24T12:27:35.493768+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:27:35.493768+0000","last_undegraded":"2024-03-24T12:27:35.493768+0000","last_fullsized":"2024-03-24T12:27:35.493768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:16.497755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.497755+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.894323+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043666+0000","last_change":"2024-03-24T12:26:09.351371+0000","last_active":"2024-03-24T12:27:35.043666+0000","last_peered":"2024-03-24T12:27:35.043666+0000","last_clean":"2024-03-24T12:27:35.043666+0000","last_became_active":"2024-03-24T12:26:09.350656+0000","last_became_peered":"2024-03-24T12:26:09.350656+0000","last_unstale":"2024-03-24T12:27:35.043666+0000","last_undegraded":"2024-03-24T12:27:35.043666+0000","last_fullsized":"2024-03-24T12:27:35.043666+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.492934+0000","last_change":"2024-03-24T12:27:20.458514+0000","last_active":"2024-03-24T12:27:35.492934+0000","last_peered":"2024-03-24T12:27:35.492934+0000","last_clean":"2024-03-24T12:27:35.492934+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:27:35.492934+0000","last_undegraded":"2024-03-24T12:27:35.492934+0000","last_fullsized":"2024-03-24T12:27:35.492934+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.458419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:24.491860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391814+0000","last_change":"2024-03-24T12:27:14.329516+0000","last_active":"2024-03-24T12:27:35.391814+0000","last_peered":"2024-03-24T12:27:35.391814+0000","last_clean":"2024-03-24T12:27:35.391814+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:27:35.391814+0000","last_undegraded":"2024-03-24T12:27:35.391814+0000","last_fullsized":"2024-03-24T12:27:35.391814+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:14.329357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:14.329357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:21.223400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569825+0000","last_change":"2024-03-24T12:26:16.882174+0000","last_active":"2024-03-24T12:27:35.569825+0000","last_peered":"2024-03-24T12:27:35.569825+0000","last_clean":"2024-03-24T12:27:35.569825+0000","last_became_active":"2024-03-24T12:26:16.874170+0000","last_became_peered":"2024-03-24T12:26:16.874170+0000","last_unstale":"2024-03-24T12:27:35.569825+0000","last_undegraded":"2024-03-24T12:27:35.569825+0000","last_fullsized":"2024-03-24T12:27:35.569825+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493631+0000","last_change":"2024-03-24T12:27:26.431474+0000","last_active":"2024-03-24T12:27:35.493631+0000","last_peered":"2024-03-24T12:27:35.493631+0000","last_clean":"2024-03-24T12:27:35.493631+0000","last_became_active":"2024-03-24T12:26:09.343296+0000","last_became_peered":"2024-03-24T12:26:09.343296+0000","last_unstale":"2024-03-24T12:27:35.493631+0000","last_undegraded":"2024-03-24T12:27:35.493631+0000","last_fullsized":"2024-03-24T12:27:35.493631+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.431382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.685011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871005+0000","last_change":"2024-03-24T12:27:29.397575+0000","last_active":"2024-03-24T12:27:34.871005+0000","last_peered":"2024-03-24T12:27:34.871005+0000","last_clean":"2024-03-24T12:27:34.871005+0000","last_became_active":"2024-03-24T12:26:07.330811+0000","last_became_peered":"2024-03-24T12:26:07.330811+0000","last_unstale":"2024-03-24T12:27:34.871005+0000","last_undegraded":"2024-03-24T12:27:34.871005+0000","last_fullsized":"2024-03-24T12:27:34.871005+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:29.397463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:29.397463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:52.953466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870888+0000","last_change":"2024-03-24T12:27:09.137207+0000","last_active":"2024-03-24T12:27:34.870888+0000","last_peered":"2024-03-24T12:27:34.870888+0000","last_clean":"2024-03-24T12:27:34.870888+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:27:34.870888+0000","last_undegraded":"2024-03-24T12:27:34.870888+0000","last_fullsized":"2024-03-24T12:27:34.870888+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:09.137104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:09.137104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.329586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"88'16","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871349+0000","last_change":"2024-03-24T12:27:17.084272+0000","last_active":"2024-03-24T12:27:34.871349+0000","last_peered":"2024-03-24T12:27:34.871349+0000","last_clean":"2024-03-24T12:27:34.871349+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:27:34.871349+0000","last_undegraded":"2024-03-24T12:27:34.871349+0000","last_fullsized":"2024-03-24T12:27:34.871349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.084148+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.541132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043972+0000","last_change":"2024-03-24T12:26:09.349851+0000","last_active":"2024-03-24T12:27:35.043972+0000","last_peered":"2024-03-24T12:27:35.043972+0000","last_clean":"2024-03-24T12:27:35.043972+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:27:35.043972+0000","last_undegraded":"2024-03-24T12:27:35.043972+0000","last_fullsized":"2024-03-24T12:27:35.043972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569527+0000","last_change":"2024-03-24T12:26:07.330555+0000","last_active":"2024-03-24T12:27:35.569527+0000","last_peered":"2024-03-24T12:27:35.569527+0000","last_clean":"2024-03-24T12:27:35.569527+0000","last_became_active":"2024-03-24T12:26:07.329487+0000","last_became_peered":"2024-03-24T12:26:07.329487+0000","last_unstale":"2024-03-24T12:27:35.569527+0000","last_undegraded":"2024-03-24T12:27:35.569527+0000","last_fullsized":"2024-03-24T12:27:35.569527+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391484+0000","last_change":"2024-03-24T12:26:03.163577+0000","last_active":"2024-03-24T12:27:35.391484+0000","last_peered":"2024-03-24T12:27:35.391484+0000","last_clean":"2024-03-24T12:27:35.391484+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:27:35.391484+0000","last_undegraded":"2024-03-24T12:27:35.391484+0000","last_fullsized":"2024-03-24T12:27:35.391484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494049+0000","last_change":"2024-03-24T12:27:21.451811+0000","last_active":"2024-03-24T12:27:35.494049+0000","last_peered":"2024-03-24T12:27:35.494049+0000","last_clean":"2024-03-24T12:27:35.494049+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:27:35.494049+0000","last_undegraded":"2024-03-24T12:27:35.494049+0000","last_fullsized":"2024-03-24T12:27:35.494049+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:21.451717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.451717+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.819749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:37.276934+0000","last_change":"2024-03-24T12:27:37.276957+0000","last_active":"2024-03-24T12:27:37.276934+0000","last_peered":"2024-03-24T12:27:37.276934+0000","last_clean":"2024-03-24T12:27:37.276934+0000","last_became_active":"2024-03-24T12:26:09.346062+0000","last_became_peered":"2024-03-24T12:26:09.346062+0000","last_unstale":"2024-03-24T12:27:37.276934+0000","last_undegraded":"2024-03-24T12:27:37.276934+0000","last_fullsized":"2024-03-24T12:27:37.276934+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:37.276875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:37.276875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.339012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493061+0000","last_change":"2024-03-24T12:26:07.338058+0000","last_active":"2024-03-24T12:27:35.493061+0000","last_peered":"2024-03-24T12:27:35.493061+0000","last_clean":"2024-03-24T12:27:35.493061+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:27:35.493061+0000","last_undegraded":"2024-03-24T12:27:35.493061+0000","last_fullsized":"2024-03-24T12:27:35.493061+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043881+0000","last_change":"2024-03-24T12:27:07.317120+0000","last_active":"2024-03-24T12:27:35.043881+0000","last_peered":"2024-03-24T12:27:35.043881+0000","last_clean":"2024-03-24T12:27:35.043881+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:27:35.043881+0000","last_undegraded":"2024-03-24T12:27:35.043881+0000","last_fullsized":"2024-03-24T12:27:35.043881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:07.317021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:07.317021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.086017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'13","reported_seq":85,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570422+0000","last_change":"2024-03-24T12:27:16.242881+0000","last_active":"2024-03-24T12:27:35.570422+0000","last_peered":"2024-03-24T12:27:35.570422+0000","last_clean":"2024-03-24T12:27:35.570422+0000","last_became_active":"2024-03-24T12:26:05.358213+0000","last_became_peered":"2024-03-24T12:26:05.358213+0000","last_unstale":"2024-03-24T12:27:35.570422+0000","last_undegraded":"2024-03-24T12:27:35.570422+0000","last_fullsized":"2024-03-24T12:27:35.570422+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.242776+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.762237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871090+0000","last_change":"2024-03-24T12:27:27.661082+0000","last_active":"2024-03-24T12:27:34.871090+0000","last_peered":"2024-03-24T12:27:34.871090+0000","last_clean":"2024-03-24T12:27:34.871090+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:27:34.871090+0000","last_undegraded":"2024-03-24T12:27:34.871090+0000","last_fullsized":"2024-03-24T12:27:34.871090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.660932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.997979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:39.472978+0000","last_change":"2024-03-24T12:27:39.473015+0000","last_active":"2024-03-24T12:27:39.472978+0000","last_peered":"2024-03-24T12:27:39.472978+0000","last_clean":"2024-03-24T12:27:39.472978+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:27:39.472978+0000","last_undegraded":"2024-03-24T12:27:39.472978+0000","last_fullsized":"2024-03-24T12:27:39.472978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:39.472900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:39.472900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:04.877211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391468+0000","last_change":"2024-03-24T12:27:06.422517+0000","last_active":"2024-03-24T12:27:35.391468+0000","last_peered":"2024-03-24T12:27:35.391468+0000","last_clean":"2024-03-24T12:27:35.391468+0000","last_became_active":"2024-03-24T12:26:03.166149+0000","last_became_peered":"2024-03-24T12:26:03.166149+0000","last_unstale":"2024-03-24T12:27:35.391468+0000","last_undegraded":"2024-03-24T12:27:35.391468+0000","last_fullsized":"2024-03-24T12:27:35.391468+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:06.422387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.422387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:12.269750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044352+0000","last_change":"2024-03-24T12:27:30.384123+0000","last_active":"2024-03-24T12:27:35.044352+0000","last_peered":"2024-03-24T12:27:35.044352+0000","last_clean":"2024-03-24T12:27:35.044352+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:27:35.044352+0000","last_undegraded":"2024-03-24T12:27:35.044352+0000","last_fullsized":"2024-03-24T12:27:35.044352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:30.384035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:30.384035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:48.792180+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:42.333908+0000","last_change":"2024-03-24T12:27:42.333953+0000","last_active":"2024-03-24T12:27:42.333908+0000","last_peered":"2024-03-24T12:27:42.333908+0000","last_clean":"2024-03-24T12:27:42.333908+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:27:42.333908+0000","last_undegraded":"2024-03-24T12:27:42.333908+0000","last_fullsized":"2024-03-24T12:27:42.333908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:42.333813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:42.333813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:01.854155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569557+0000","last_change":"2024-03-24T12:26:07.329432+0000","last_active":"2024-03-24T12:27:35.569557+0000","last_peered":"2024-03-24T12:27:35.569557+0000","last_clean":"2024-03-24T12:27:35.569557+0000","last_became_active":"2024-03-24T12:26:07.329117+0000","last_became_peered":"2024-03-24T12:26:07.329117+0000","last_unstale":"2024-03-24T12:27:35.569557+0000","last_undegraded":"2024-03-24T12:27:35.569557+0000","last_fullsized":"2024-03-24T12:27:35.569557+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494270+0000","last_change":"2024-03-24T12:27:06.584912+0000","last_active":"2024-03-24T12:27:35.494270+0000","last_peered":"2024-03-24T12:27:35.494270+0000","last_clean":"2024-03-24T12:27:35.494270+0000","last_became_active":"2024-03-24T12:26:03.161414+0000","last_became_peered":"2024-03-24T12:26:03.161414+0000","last_unstale":"2024-03-24T12:27:35.494270+0000","last_undegraded":"2024-03-24T12:27:35.494270+0000","last_fullsized":"2024-03-24T12:27:35.494270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:06.584809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.584809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:16.866693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":66,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494581+0000","last_change":"2024-03-24T12:26:05.357244+0000","last_active":"2024-03-24T12:27:35.494581+0000","last_peered":"2024-03-24T12:27:35.494581+0000","last_clean":"2024-03-24T12:27:35.494581+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:27:35.494581+0000","last_undegraded":"2024-03-24T12:27:35.494581+0000","last_fullsized":"2024-03-24T12:27:35.494581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570086+0000","last_change":"2024-03-24T12:26:09.643942+0000","last_active":"2024-03-24T12:27:35.570086+0000","last_peered":"2024-03-24T12:27:35.570086+0000","last_clean":"2024-03-24T12:27:35.570086+0000","last_became_active":"2024-03-24T12:26:09.643602+0000","last_became_peered":"2024-03-24T12:26:09.643602+0000","last_unstale":"2024-03-24T12:27:35.570086+0000","last_undegraded":"2024-03-24T12:27:35.570086+0000","last_fullsized":"2024-03-24T12:27:35.570086+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.392046+0000","last_change":"2024-03-24T12:26:03.161676+0000","last_active":"2024-03-24T12:27:35.392046+0000","last_peered":"2024-03-24T12:27:35.392046+0000","last_clean":"2024-03-24T12:27:35.392046+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:27:35.392046+0000","last_undegraded":"2024-03-24T12:27:35.392046+0000","last_fullsized":"2024-03-24T12:27:35.392046+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569814+0000","last_change":"2024-03-24T12:27:17.288726+0000","last_active":"2024-03-24T12:27:35.569814+0000","last_peered":"2024-03-24T12:27:35.569814+0000","last_clean":"2024-03-24T12:27:35.569814+0000","last_became_active":"2024-03-24T12:26:05.358217+0000","last_became_peered":"2024-03-24T12:26:05.358217+0000","last_unstale":"2024-03-24T12:27:35.569814+0000","last_undegraded":"2024-03-24T12:27:35.569814+0000","last_fullsized":"2024-03-24T12:27:35.569814+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:27:17.288616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.288616+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:31.276817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":92,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391251+0000","last_change":"2024-03-24T12:27:20.354464+0000","last_active":"2024-03-24T12:27:35.391251+0000","last_peered":"2024-03-24T12:27:35.391251+0000","last_clean":"2024-03-24T12:27:35.391251+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:27:35.391251+0000","last_undegraded":"2024-03-24T12:27:35.391251+0000","last_fullsized":"2024-03-24T12:27:35.391251+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:20.354367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.354367+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.013310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493952+0000","last_change":"2024-03-24T12:26:09.350769+0000","last_active":"2024-03-24T12:27:35.493952+0000","last_peered":"2024-03-24T12:27:35.493952+0000","last_clean":"2024-03-24T12:27:35.493952+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:27:35.493952+0000","last_undegraded":"2024-03-24T12:27:35.493952+0000","last_fullsized":"2024-03-24T12:27:35.493952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:38.324576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391982+0000","last_change":"2024-03-24T12:26:03.162567+0000","last_active":"2024-03-24T12:27:35.391982+0000","last_peered":"2024-03-24T12:27:35.391982+0000","last_clean":"2024-03-24T12:27:35.391982+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:27:35.391982+0000","last_undegraded":"2024-03-24T12:27:35.391982+0000","last_fullsized":"2024-03-24T12:27:35.391982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":88,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870620+0000","last_change":"2024-03-24T12:27:23.131685+0000","last_active":"2024-03-24T12:27:34.870620+0000","last_peered":"2024-03-24T12:27:34.870620+0000","last_clean":"2024-03-24T12:27:34.870620+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:27:34.870620+0000","last_undegraded":"2024-03-24T12:27:34.870620+0000","last_fullsized":"2024-03-24T12:27:34.870620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:23.131573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.131573+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.851864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.581442+0000","last_change":"2024-03-24T12:27:35.581463+0000","last_active":"2024-03-24T12:27:35.581442+0000","last_peered":"2024-03-24T12:27:35.581442+0000","last_clean":"2024-03-24T12:27:35.581442+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:27:35.581442+0000","last_undegraded":"2024-03-24T12:27:35.581442+0000","last_fullsized":"2024-03-24T12:27:35.581442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:35.581370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:35.581370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.191248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391391+0000","last_change":"2024-03-24T12:26:09.339988+0000","last_active":"2024-03-24T12:27:35.391391+0000","last_peered":"2024-03-24T12:27:35.391391+0000","last_clean":"2024-03-24T12:27:35.391391+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:27:35.391391+0000","last_undegraded":"2024-03-24T12:27:35.391391+0000","last_fullsized":"2024-03-24T12:27:35.391391+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.803044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870725+0000","last_change":"2024-03-24T12:27:32.548089+0000","last_active":"2024-03-24T12:27:34.870725+0000","last_peered":"2024-03-24T12:27:34.870725+0000","last_clean":"2024-03-24T12:27:34.870725+0000","last_became_active":"2024-03-24T12:26:07.331416+0000","last_became_peered":"2024-03-24T12:26:07.331416+0000","last_unstale":"2024-03-24T12:27:34.870725+0000","last_undegraded":"2024-03-24T12:27:34.870725+0000","last_fullsized":"2024-03-24T12:27:34.870725+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.547951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.547951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.882159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:37.366726+0000","last_change":"2024-03-24T12:27:37.366770+0000","last_active":"2024-03-24T12:27:37.366726+0000","last_peered":"2024-03-24T12:27:37.366726+0000","last_clean":"2024-03-24T12:27:37.366726+0000","last_became_active":"2024-03-24T12:26:03.165214+0000","last_became_peered":"2024-03-24T12:26:03.165214+0000","last_unstale":"2024-03-24T12:27:37.366726+0000","last_undegraded":"2024-03-24T12:27:37.366726+0000","last_fullsized":"2024-03-24T12:27:37.366726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:37.366618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:37.366618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.039598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":96,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494684+0000","last_change":"2024-03-24T12:27:15.668713+0000","last_active":"2024-03-24T12:27:35.494684+0000","last_peered":"2024-03-24T12:27:35.494684+0000","last_clean":"2024-03-24T12:27:35.494684+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:27:35.494684+0000","last_undegraded":"2024-03-24T12:27:35.494684+0000","last_fullsized":"2024-03-24T12:27:35.494684+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:27:15.668556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:15.668556+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:43.755371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043720+0000","last_change":"2024-03-24T12:26:09.350031+0000","last_active":"2024-03-24T12:27:35.043720+0000","last_peered":"2024-03-24T12:27:35.043720+0000","last_clean":"2024-03-24T12:27:35.043720+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:27:35.043720+0000","last_undegraded":"2024-03-24T12:27:35.043720+0000","last_fullsized":"2024-03-24T12:27:35.043720+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:38.596198+0000","last_change":"2024-03-24T12:27:38.596223+0000","last_active":"2024-03-24T12:27:38.596198+0000","last_peered":"2024-03-24T12:27:38.596198+0000","last_clean":"2024-03-24T12:27:38.596198+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:27:38.596198+0000","last_undegraded":"2024-03-24T12:27:38.596198+0000","last_fullsized":"2024-03-24T12:27:38.596198+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.596133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:52.862429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570168+0000","last_change":"2024-03-24T12:27:09.234408+0000","last_active":"2024-03-24T12:27:35.570168+0000","last_peered":"2024-03-24T12:27:35.570168+0000","last_clean":"2024-03-24T12:27:35.570168+0000","last_became_active":"2024-03-24T12:26:03.162562+0000","last_became_peered":"2024-03-24T12:26:03.162562+0000","last_unstale":"2024-03-24T12:27:35.570168+0000","last_undegraded":"2024-03-24T12:27:35.570168+0000","last_fullsized":"2024-03-24T12:27:35.570168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:27:09.234299+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:17.263964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044198+0000","last_change":"2024-03-24T12:26:16.882489+0000","last_active":"2024-03-24T12:27:35.044198+0000","last_peered":"2024-03-24T12:27:35.044198+0000","last_clean":"2024-03-24T12:27:35.044198+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:27:35.044198+0000","last_undegraded":"2024-03-24T12:27:35.044198+0000","last_fullsized":"2024-03-24T12:27:35.044198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494257+0000","last_change":"2024-03-24T12:27:24.439386+0000","last_active":"2024-03-24T12:27:35.494257+0000","last_peered":"2024-03-24T12:27:35.494257+0000","last_clean":"2024-03-24T12:27:35.494257+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:27:35.494257+0000","last_undegraded":"2024-03-24T12:27:35.494257+0000","last_fullsized":"2024-03-24T12:27:35.494257+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.439296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.495934+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569976+0000","last_change":"2024-03-24T12:26:07.318867+0000","last_active":"2024-03-24T12:27:35.569976+0000","last_peered":"2024-03-24T12:27:35.569976+0000","last_clean":"2024-03-24T12:27:35.569976+0000","last_became_active":"2024-03-24T12:26:03.165771+0000","last_became_peered":"2024-03-24T12:26:03.165771+0000","last_unstale":"2024-03-24T12:27:35.569976+0000","last_undegraded":"2024-03-24T12:27:35.569976+0000","last_fullsized":"2024-03-24T12:27:35.569976+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870916+0000","last_change":"2024-03-24T12:27:32.227062+0000","last_active":"2024-03-24T12:27:34.870916+0000","last_peered":"2024-03-24T12:27:34.870916+0000","last_clean":"2024-03-24T12:27:34.870916+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:27:34.870916+0000","last_undegraded":"2024-03-24T12:27:34.870916+0000","last_fullsized":"2024-03-24T12:27:34.870916+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.226982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.226982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.516101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"90'38","reported_seq":143,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871376+0000","last_change":"2024-03-24T12:27:31.375798+0000","last_active":"2024-03-24T12:27:34.871376+0000","last_peered":"2024-03-24T12:27:34.871376+0000","last_clean":"2024-03-24T12:27:34.871376+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:27:34.871376+0000","last_undegraded":"2024-03-24T12:27:34.871376+0000","last_fullsized":"2024-03-24T12:27:34.871376+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'38","last_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:27:31.375629+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:55.069683+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044171+0000","last_change":"2024-03-24T12:26:07.332154+0000","last_active":"2024-03-24T12:27:35.044171+0000","last_peered":"2024-03-24T12:27:35.044171+0000","last_clean":"2024-03-24T12:27:35.044171+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:27:35.044171+0000","last_undegraded":"2024-03-24T12:27:35.044171+0000","last_fullsized":"2024-03-24T12:27:35.044171+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.392103+0000","last_change":"2024-03-24T12:27:18.336245+0000","last_active":"2024-03-24T12:27:35.392103+0000","last_peered":"2024-03-24T12:27:35.392103+0000","last_clean":"2024-03-24T12:27:35.392103+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:27:35.392103+0000","last_undegraded":"2024-03-24T12:27:35.392103+0000","last_fullsized":"2024-03-24T12:27:35.392103+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:18.336147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:18.336147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.167902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870392+0000","last_change":"2024-03-24T12:27:16.330188+0000","last_active":"2024-03-24T12:27:34.870392+0000","last_peered":"2024-03-24T12:27:34.870392+0000","last_clean":"2024-03-24T12:27:34.870392+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:27:34.870392+0000","last_undegraded":"2024-03-24T12:27:34.870392+0000","last_fullsized":"2024-03-24T12:27:34.870392+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:16.330050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.330050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.775200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870388+0000","last_change":"2024-03-24T12:27:24.614983+0000","last_active":"2024-03-24T12:27:34.870388+0000","last_peered":"2024-03-24T12:27:34.870388+0000","last_clean":"2024-03-24T12:27:34.870388+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:27:34.870388+0000","last_undegraded":"2024-03-24T12:27:34.870388+0000","last_fullsized":"2024-03-24T12:27:34.870388+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.614896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.614896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871644+0000","last_change":"2024-03-24T12:27:06.412196+0000","last_active":"2024-03-24T12:27:34.871644+0000","last_peered":"2024-03-24T12:27:34.871644+0000","last_clean":"2024-03-24T12:27:34.871644+0000","last_became_active":"2024-03-24T12:26:05.315639+0000","last_became_peered":"2024-03-24T12:26:05.315639+0000","last_unstale":"2024-03-24T12:27:34.871644+0000","last_undegraded":"2024-03-24T12:27:34.871644+0000","last_fullsized":"2024-03-24T12:27:34.871644+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:27:06.412074+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:27:06.412074+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.412074+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.288814+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871108+0000","last_change":"2024-03-24T12:27:19.658489+0000","last_active":"2024-03-24T12:27:34.871108+0000","last_peered":"2024-03-24T12:27:34.871108+0000","last_clean":"2024-03-24T12:27:34.871108+0000","last_became_active":"2024-03-24T12:26:07.343010+0000","last_became_peered":"2024-03-24T12:26:07.343010+0000","last_unstale":"2024-03-24T12:27:34.871108+0000","last_undegraded":"2024-03-24T12:27:34.871108+0000","last_fullsized":"2024-03-24T12:27:34.871108+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:19.658367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:19.658367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.033169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570399+0000","last_change":"2024-03-24T12:27:12.274502+0000","last_active":"2024-03-24T12:27:35.570399+0000","last_peered":"2024-03-24T12:27:35.570399+0000","last_clean":"2024-03-24T12:27:35.570399+0000","last_became_active":"2024-03-24T12:26:09.347821+0000","last_became_peered":"2024-03-24T12:26:09.347821+0000","last_unstale":"2024-03-24T12:27:35.570399+0000","last_undegraded":"2024-03-24T12:27:35.570399+0000","last_fullsized":"2024-03-24T12:27:35.570399+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.274393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.274393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.457433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":322,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391519+0000","last_change":"2024-03-24T12:27:08.342835+0000","last_active":"2024-03-24T12:27:35.391519+0000","last_peered":"2024-03-24T12:27:35.391519+0000","last_clean":"2024-03-24T12:27:35.391519+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:27:35.391519+0000","last_undegraded":"2024-03-24T12:27:35.391519+0000","last_fullsized":"2024-03-24T12:27:35.391519+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:27:08.342708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:27:08.342708+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:25.314000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493810+0000","last_change":"2024-03-24T12:27:28.436490+0000","last_active":"2024-03-24T12:27:35.493810+0000","last_peered":"2024-03-24T12:27:35.493810+0000","last_clean":"2024-03-24T12:27:35.493810+0000","last_became_active":"2024-03-24T12:26:03.166467+0000","last_became_peered":"2024-03-24T12:26:03.166467+0000","last_unstale":"2024-03-24T12:27:35.493810+0000","last_undegraded":"2024-03-24T12:27:35.493810+0000","last_fullsized":"2024-03-24T12:27:35.493810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.436402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.436402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.490307+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":107,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494871+0000","last_change":"2024-03-24T12:27:21.712724+0000","last_active":"2024-03-24T12:27:35.494871+0000","last_peered":"2024-03-24T12:27:35.494871+0000","last_clean":"2024-03-24T12:27:35.494871+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:27:35.494871+0000","last_undegraded":"2024-03-24T12:27:35.494871+0000","last_fullsized":"2024-03-24T12:27:35.494871+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:27:21.712602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.712602+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.700708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391063+0000","last_change":"2024-03-24T12:26:07.337813+0000","last_active":"2024-03-24T12:27:35.391063+0000","last_peered":"2024-03-24T12:27:35.391063+0000","last_clean":"2024-03-24T12:27:35.391063+0000","last_became_active":"2024-03-24T12:26:07.337548+0000","last_became_peered":"2024-03-24T12:26:07.337548+0000","last_unstale":"2024-03-24T12:27:35.391063+0000","last_undegraded":"2024-03-24T12:27:35.391063+0000","last_fullsized":"2024-03-24T12:27:35.391063+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870685+0000","last_change":"2024-03-24T12:27:25.409449+0000","last_active":"2024-03-24T12:27:34.870685+0000","last_peered":"2024-03-24T12:27:34.870685+0000","last_clean":"2024-03-24T12:27:34.870685+0000","last_became_active":"2024-03-24T12:26:09.349435+0000","last_became_peered":"2024-03-24T12:26:09.349435+0000","last_unstale":"2024-03-24T12:27:34.870685+0000","last_undegraded":"2024-03-24T12:27:34.870685+0000","last_fullsized":"2024-03-24T12:27:34.870685+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:25.409346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.409346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.841693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043877+0000","last_change":"2024-03-24T12:27:27.333417+0000","last_active":"2024-03-24T12:27:35.043877+0000","last_peered":"2024-03-24T12:27:35.043877+0000","last_clean":"2024-03-24T12:27:35.043877+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:27:35.043877+0000","last_undegraded":"2024-03-24T12:27:35.043877+0000","last_fullsized":"2024-03-24T12:27:35.043877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:27.333274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.333274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.779264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":171,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:40.525206+0000","last_change":"2024-03-24T12:27:16.322773+0000","last_active":"2024-03-24T12:27:40.525206+0000","last_peered":"2024-03-24T12:27:40.525206+0000","last_clean":"2024-03-24T12:27:40.525206+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:27:40.525206+0000","last_undegraded":"2024-03-24T12:27:40.525206+0000","last_fullsized":"2024-03-24T12:27:40.525206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:27:16.322655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.322655+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:22.799348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":92,"num_read_kb":87,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:38.328522+0000","last_change":"2024-03-24T12:27:38.328555+0000","last_active":"2024-03-24T12:27:38.328522+0000","last_peered":"2024-03-24T12:27:38.328522+0000","last_clean":"2024-03-24T12:27:38.328522+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:27:38.328522+0000","last_undegraded":"2024-03-24T12:27:38.328522+0000","last_fullsized":"2024-03-24T12:27:38.328522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:38.328458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.328458+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.105741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569584+0000","last_change":"2024-03-24T12:27:23.276910+0000","last_active":"2024-03-24T12:27:35.569584+0000","last_peered":"2024-03-24T12:27:35.569584+0000","last_clean":"2024-03-24T12:27:35.569584+0000","last_became_active":"2024-03-24T12:26:07.326457+0000","last_became_peered":"2024-03-24T12:26:07.326457+0000","last_unstale":"2024-03-24T12:27:35.569584+0000","last_undegraded":"2024-03-24T12:27:35.569584+0000","last_fullsized":"2024-03-24T12:27:35.569584+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.276789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.276789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.022163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:41.410555+0000","last_change":"2024-03-24T12:27:41.410577+0000","last_active":"2024-03-24T12:27:41.410555+0000","last_peered":"2024-03-24T12:27:41.410555+0000","last_clean":"2024-03-24T12:27:41.410555+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:27:41.410555+0000","last_undegraded":"2024-03-24T12:27:41.410555+0000","last_fullsized":"2024-03-24T12:27:41.410555+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:41.410495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:41.410495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.007651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871261+0000","last_change":"2024-03-24T12:27:32.343982+0000","last_active":"2024-03-24T12:27:34.871261+0000","last_peered":"2024-03-24T12:27:34.871261+0000","last_clean":"2024-03-24T12:27:34.871261+0000","last_became_active":"2024-03-24T12:26:03.167244+0000","last_became_peered":"2024-03-24T12:26:03.167244+0000","last_unstale":"2024-03-24T12:27:34.871261+0000","last_undegraded":"2024-03-24T12:27:34.871261+0000","last_fullsized":"2024-03-24T12:27:34.871261+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.343880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.343880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:53.358101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":93,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044244+0000","last_change":"2024-03-24T12:26:05.357149+0000","last_active":"2024-03-24T12:27:35.044244+0000","last_peered":"2024-03-24T12:27:35.044244+0000","last_clean":"2024-03-24T12:27:35.044244+0000","last_became_active":"2024-03-24T12:26:05.356745+0000","last_became_peered":"2024-03-24T12:26:05.356745+0000","last_unstale":"2024-03-24T12:27:35.044244+0000","last_undegraded":"2024-03-24T12:27:35.044244+0000","last_fullsized":"2024-03-24T12:27:35.044244+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494114+0000","last_change":"2024-03-24T12:27:23.763463+0000","last_active":"2024-03-24T12:27:35.494114+0000","last_peered":"2024-03-24T12:27:35.494114+0000","last_clean":"2024-03-24T12:27:35.494114+0000","last_became_active":"2024-03-24T12:26:07.337303+0000","last_became_peered":"2024-03-24T12:26:07.337303+0000","last_unstale":"2024-03-24T12:27:35.494114+0000","last_undegraded":"2024-03-24T12:27:35.494114+0000","last_fullsized":"2024-03-24T12:27:35.494114+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.763326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:36.340669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043666+0000","last_change":"2024-03-24T12:26:09.341274+0000","last_active":"2024-03-24T12:27:35.043666+0000","last_peered":"2024-03-24T12:27:35.043666+0000","last_clean":"2024-03-24T12:27:35.043666+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:27:35.043666+0000","last_undegraded":"2024-03-24T12:27:35.043666+0000","last_fullsized":"2024-03-24T12:27:35.043666+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570395+0000","last_change":"2024-03-24T12:27:26.215446+0000","last_active":"2024-03-24T12:27:35.570395+0000","last_peered":"2024-03-24T12:27:35.570395+0000","last_clean":"2024-03-24T12:27:35.570395+0000","last_became_active":"2024-03-24T12:26:03.168184+0000","last_became_peered":"2024-03-24T12:26:03.168184+0000","last_unstale":"2024-03-24T12:27:35.570395+0000","last_undegraded":"2024-03-24T12:27:35.570395+0000","last_fullsized":"2024-03-24T12:27:35.570395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:26.215348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.215348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.879847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871050+0000","last_change":"2024-03-24T12:27:24.375037+0000","last_active":"2024-03-24T12:27:34.871050+0000","last_peered":"2024-03-24T12:27:34.871050+0000","last_clean":"2024-03-24T12:27:34.871050+0000","last_became_active":"2024-03-24T12:26:05.316739+0000","last_became_peered":"2024-03-24T12:26:05.316739+0000","last_unstale":"2024-03-24T12:27:34.871050+0000","last_undegraded":"2024-03-24T12:27:34.871050+0000","last_fullsized":"2024-03-24T12:27:34.871050+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:24.374935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.374935+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:47.083756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044174+0000","last_change":"2024-03-24T12:26:07.336444+0000","last_active":"2024-03-24T12:27:35.044174+0000","last_peered":"2024-03-24T12:27:35.044174+0000","last_clean":"2024-03-24T12:27:35.044174+0000","last_became_active":"2024-03-24T12:26:07.336080+0000","last_became_peered":"2024-03-24T12:26:07.336080+0000","last_unstale":"2024-03-24T12:27:35.044174+0000","last_undegraded":"2024-03-24T12:27:35.044174+0000","last_fullsized":"2024-03-24T12:27:35.044174+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391746+0000","last_change":"2024-03-24T12:27:33.429270+0000","last_active":"2024-03-24T12:27:35.391746+0000","last_peered":"2024-03-24T12:27:35.391746+0000","last_clean":"2024-03-24T12:27:35.391746+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:27:35.391746+0000","last_undegraded":"2024-03-24T12:27:35.391746+0000","last_fullsized":"2024-03-24T12:27:35.391746+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:33.429173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:33.429173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.664982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044198+0000","last_change":"2024-03-24T12:27:12.294371+0000","last_active":"2024-03-24T12:27:35.044198+0000","last_peered":"2024-03-24T12:27:35.044198+0000","last_clean":"2024-03-24T12:27:35.044198+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:27:35.044198+0000","last_undegraded":"2024-03-24T12:27:35.044198+0000","last_fullsized":"2024-03-24T12:27:35.044198+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:12.294292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.294292+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.602110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870646+0000","last_change":"2024-03-24T12:26:05.315771+0000","last_active":"2024-03-24T12:27:34.870646+0000","last_peered":"2024-03-24T12:27:34.870646+0000","last_clean":"2024-03-24T12:27:34.870646+0000","last_became_active":"2024-03-24T12:26:05.315454+0000","last_became_peered":"2024-03-24T12:26:05.315454+0000","last_unstale":"2024-03-24T12:27:34.870646+0000","last_undegraded":"2024-03-24T12:27:34.870646+0000","last_fullsized":"2024-03-24T12:27:34.870646+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.492894+0000","last_change":"2024-03-24T12:27:10.466243+0000","last_active":"2024-03-24T12:27:35.492894+0000","last_peered":"2024-03-24T12:27:35.492894+0000","last_clean":"2024-03-24T12:27:35.492894+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:27:35.492894+0000","last_undegraded":"2024-03-24T12:27:35.492894+0000","last_fullsized":"2024-03-24T12:27:35.492894+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:10.466147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.466147+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:12.236572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870529+0000","last_change":"2024-03-24T12:26:09.352908+0000","last_active":"2024-03-24T12:27:34.870529+0000","last_peered":"2024-03-24T12:27:34.870529+0000","last_clean":"2024-03-24T12:27:34.870529+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:27:34.870529+0000","last_undegraded":"2024-03-24T12:27:34.870529+0000","last_fullsized":"2024-03-24T12:27:34.870529+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.176448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:36.482534+0000","last_change":"2024-03-24T12:27:36.482571+0000","last_active":"2024-03-24T12:27:36.482534+0000","last_peered":"2024-03-24T12:27:36.482534+0000","last_clean":"2024-03-24T12:27:36.482534+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:27:36.482534+0000","last_undegraded":"2024-03-24T12:27:36.482534+0000","last_fullsized":"2024-03-24T12:27:36.482534+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:36.482466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:36.482466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:03.061842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493506+0000","last_change":"2024-03-24T12:27:11.454130+0000","last_active":"2024-03-24T12:27:35.493506+0000","last_peered":"2024-03-24T12:27:35.493506+0000","last_clean":"2024-03-24T12:27:35.493506+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:27:35.493506+0000","last_undegraded":"2024-03-24T12:27:35.493506+0000","last_fullsized":"2024-03-24T12:27:35.493506+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T12:27:11.453973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:11.453973+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:18.945222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870349+0000","last_change":"2024-03-24T12:26:07.336955+0000","last_active":"2024-03-24T12:27:34.870349+0000","last_peered":"2024-03-24T12:27:34.870349+0000","last_clean":"2024-03-24T12:27:34.870349+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:27:34.870349+0000","last_undegraded":"2024-03-24T12:27:34.870349+0000","last_fullsized":"2024-03-24T12:27:34.870349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:35.459896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570232+0000","last_change":"2024-03-24T12:26:09.351354+0000","last_active":"2024-03-24T12:27:35.570232+0000","last_peered":"2024-03-24T12:27:35.570232+0000","last_clean":"2024-03-24T12:27:35.570232+0000","last_became_active":"2024-03-24T12:26:09.351061+0000","last_became_peered":"2024-03-24T12:26:09.351061+0000","last_unstale":"2024-03-24T12:27:35.570232+0000","last_undegraded":"2024-03-24T12:27:35.570232+0000","last_fullsized":"2024-03-24T12:27:35.570232+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871496+0000","last_change":"2024-03-24T12:27:20.101666+0000","last_active":"2024-03-24T12:27:34.871496+0000","last_peered":"2024-03-24T12:27:34.871496+0000","last_clean":"2024-03-24T12:27:34.871496+0000","last_became_active":"2024-03-24T12:26:07.328595+0000","last_became_peered":"2024-03-24T12:26:07.328595+0000","last_unstale":"2024-03-24T12:27:34.871496+0000","last_undegraded":"2024-03-24T12:27:34.871496+0000","last_fullsized":"2024-03-24T12:27:34.871496+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:20.101554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.101554+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:34.528554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493461+0000","last_change":"2024-03-24T12:26:03.163624+0000","last_active":"2024-03-24T12:27:35.493461+0000","last_peered":"2024-03-24T12:27:35.493461+0000","last_clean":"2024-03-24T12:27:35.493461+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:27:35.493461+0000","last_undegraded":"2024-03-24T12:27:35.493461+0000","last_fullsized":"2024-03-24T12:27:35.493461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871458+0000","last_change":"2024-03-24T12:26:16.885041+0000","last_active":"2024-03-24T12:27:34.871458+0000","last_peered":"2024-03-24T12:27:34.871458+0000","last_clean":"2024-03-24T12:27:34.871458+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:27:34.871458+0000","last_undegraded":"2024-03-24T12:27:34.871458+0000","last_fullsized":"2024-03-24T12:27:34.871458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:40.439771+0000","last_change":"2024-03-24T12:27:40.439811+0000","last_active":"2024-03-24T12:27:40.439771+0000","last_peered":"2024-03-24T12:27:40.439771+0000","last_clean":"2024-03-24T12:27:40.439771+0000","last_became_active":"2024-03-24T12:26:09.643739+0000","last_became_peered":"2024-03-24T12:26:09.643739+0000","last_unstale":"2024-03-24T12:27:40.439771+0000","last_undegraded":"2024-03-24T12:27:40.439771+0000","last_fullsized":"2024-03-24T12:27:40.439771+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:40.439695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:40.439695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:04.958905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569735+0000","last_change":"2024-03-24T12:27:08.227925+0000","last_active":"2024-03-24T12:27:35.569735+0000","last_peered":"2024-03-24T12:27:35.569735+0000","last_clean":"2024-03-24T12:27:35.569735+0000","last_became_active":"2024-03-24T12:26:07.332766+0000","last_became_peered":"2024-03-24T12:26:07.332766+0000","last_unstale":"2024-03-24T12:27:35.569735+0000","last_undegraded":"2024-03-24T12:27:35.569735+0000","last_fullsized":"2024-03-24T12:27:35.569735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:08.227812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:08.227812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:14.184705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870384+0000","last_change":"2024-03-24T12:27:31.177660+0000","last_active":"2024-03-24T12:27:34.870384+0000","last_peered":"2024-03-24T12:27:34.870384+0000","last_clean":"2024-03-24T12:27:34.870384+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:27:34.870384+0000","last_undegraded":"2024-03-24T12:27:34.870384+0000","last_fullsized":"2024-03-24T12:27:34.870384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:31.177529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:31.177529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.777930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871082+0000","last_change":"2024-03-24T12:26:05.315880+0000","last_active":"2024-03-24T12:27:34.871082+0000","last_peered":"2024-03-24T12:27:34.871082+0000","last_clean":"2024-03-24T12:27:34.871082+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:27:34.871082+0000","last_undegraded":"2024-03-24T12:27:34.871082+0000","last_fullsized":"2024-03-24T12:27:34.871082+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871619+0000","last_change":"2024-03-24T12:26:09.348104+0000","last_active":"2024-03-24T12:27:34.871619+0000","last_peered":"2024-03-24T12:27:34.871619+0000","last_clean":"2024-03-24T12:27:34.871619+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:27:34.871619+0000","last_undegraded":"2024-03-24T12:27:34.871619+0000","last_fullsized":"2024-03-24T12:27:34.871619+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.236528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871483+0000","last_change":"2024-03-24T12:27:23.411568+0000","last_active":"2024-03-24T12:27:34.871483+0000","last_peered":"2024-03-24T12:27:34.871483+0000","last_clean":"2024-03-24T12:27:34.871483+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:27:34.871483+0000","last_undegraded":"2024-03-24T12:27:34.871483+0000","last_fullsized":"2024-03-24T12:27:34.871483+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.411477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.411477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.665294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494588+0000","last_change":"2024-03-24T12:27:12.636522+0000","last_active":"2024-03-24T12:27:35.494588+0000","last_peered":"2024-03-24T12:27:35.494588+0000","last_clean":"2024-03-24T12:27:35.494588+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:27:35.494588+0000","last_undegraded":"2024-03-24T12:27:35.494588+0000","last_fullsized":"2024-03-24T12:27:35.494588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.636423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.636423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.538773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":85,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871126+0000","last_change":"2024-03-24T12:26:05.316692+0000","last_active":"2024-03-24T12:27:34.871126+0000","last_peered":"2024-03-24T12:27:34.871126+0000","last_clean":"2024-03-24T12:27:34.871126+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:27:34.871126+0000","last_undegraded":"2024-03-24T12:27:34.871126+0000","last_fullsized":"2024-03-24T12:27:34.871126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871534+0000","last_change":"2024-03-24T12:27:10.145724+0000","last_active":"2024-03-24T12:27:34.871534+0000","last_peered":"2024-03-24T12:27:34.871534+0000","last_clean":"2024-03-24T12:27:34.871534+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:27:34.871534+0000","last_undegraded":"2024-03-24T12:27:34.871534+0000","last_fullsized":"2024-03-24T12:27:34.871534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:10.145613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.145613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:22.594263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871322+0000","last_change":"2024-03-24T12:27:21.649282+0000","last_active":"2024-03-24T12:27:34.871322+0000","last_peered":"2024-03-24T12:27:34.871322+0000","last_clean":"2024-03-24T12:27:34.871322+0000","last_became_active":"2024-03-24T12:26:07.334798+0000","last_became_peered":"2024-03-24T12:26:07.334798+0000","last_unstale":"2024-03-24T12:27:34.871322+0000","last_undegraded":"2024-03-24T12:27:34.871322+0000","last_fullsized":"2024-03-24T12:27:34.871322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:21.649181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.649181+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.389315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494051+0000","last_change":"2024-03-24T12:26:03.162618+0000","last_active":"2024-03-24T12:27:35.494051+0000","last_peered":"2024-03-24T12:27:35.494051+0000","last_clean":"2024-03-24T12:27:35.494051+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:27:35.494051+0000","last_undegraded":"2024-03-24T12:27:35.494051+0000","last_fullsized":"2024-03-24T12:27:35.494051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870710+0000","last_change":"2024-03-24T12:27:22.098052+0000","last_active":"2024-03-24T12:27:34.870710+0000","last_peered":"2024-03-24T12:27:34.870710+0000","last_clean":"2024-03-24T12:27:34.870710+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:27:34.870710+0000","last_undegraded":"2024-03-24T12:27:34.870710+0000","last_fullsized":"2024-03-24T12:27:34.870710+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:22.097943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:22.097943+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.541070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":424,"ondisk_log_size":424,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":87,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037784,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28436,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716716,"statfs":{"total":95995035648,"available":95965917184,"internally_reserved":0,"allocated":1454080,"data_stored":964966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332126,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32572,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712580,"statfs":{"total":95995035648,"available":95961681920,"internally_reserved":0,"allocated":1429504,"data_stored":964378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":374988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32596,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712556,"statfs":{"total":95995035648,"available":95961657344,"internally_reserved":0,"allocated":1454080,"data_stored":964310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":373470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117036,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":374914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920808,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":373740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626467,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":847872,"data_stored":374844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:27:43.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:27:43.896 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:44.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:43 smithi190 ceph-mon[33748]: 3.8 scrub starts 2024-03-24T12:27:44.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:43 smithi190 ceph-mon[33748]: 3.8 scrub ok 2024-03-24T12:27:44.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:43 smithi190 ceph-mon[33748]: 4.f deep-scrub starts 2024-03-24T12:27:44.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:43 smithi190 ceph-mon[33748]: 4.f deep-scrub ok 2024-03-24T12:27:44.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:43 smithi029 ceph-mon[41500]: 3.8 scrub starts 2024-03-24T12:27:44.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:43 smithi029 ceph-mon[41500]: 3.8 scrub ok 2024-03-24T12:27:44.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:43 smithi029 ceph-mon[41500]: 4.f deep-scrub starts 2024-03-24T12:27:44.251 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:43 smithi029 ceph-mon[41500]: 4.f deep-scrub ok 2024-03-24T12:27:44.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:43 smithi029 ceph-mon[29919]: 3.8 scrub starts 2024-03-24T12:27:44.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:43 smithi029 ceph-mon[29919]: 3.8 scrub ok 2024-03-24T12:27:44.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:43 smithi029 ceph-mon[29919]: 4.f deep-scrub starts 2024-03-24T12:27:44.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:43 smithi029 ceph-mon[29919]: 4.f deep-scrub ok 2024-03-24T12:27:44.536 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 podman[66295]: 2024-03-24 12:27:44.426639936 +0000 UTC m=+0.143482971 container create 86dfe5a4032f20268d81ffe775b26556ee313288affa70e896aff605e96a0a86 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a, io.buildah.version=1.32.0, io.openshift.tags=base rhel8, vcs-type=git, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.openshift.expose-services=, distribution-scope=public, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, version=8.5) 2024-03-24T12:27:44.894 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 podman[66295]: 2024-03-24 12:27:44.646420898 +0000 UTC m=+0.363263901 container init 86dfe5a4032f20268d81ffe775b26556ee313288affa70e896aff605e96a0a86 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a, description=Ceph Grafana Container, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, com.redhat.component=ubi8-container, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, distribution-scope=public, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, vcs-type=git, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, io.openshift.expose-services=) 2024-03-24T12:27:44.894 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 podman[66295]: 2024-03-24 12:27:44.652073944 +0000 UTC m=+0.368916933 container start 86dfe5a4032f20268d81ffe775b26556ee313288affa70e896aff605e96a0a86 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-04-29T11:48:11.055897, vcs-type=git, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, release=239.1651231664, io.buildah.version=1.32.0, distribution-scope=public, io.openshift.tags=base rhel8, version=8.5, architecture=x86_64, vendor=Red Hat, Inc.) 2024-03-24T12:27:44.894 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 bash[66295]: 86dfe5a4032f20268d81ffe775b26556ee313288affa70e896aff605e96a0a86 2024-03-24T12:27:44.894 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 systemd[1]: Started Ceph grafana.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:27:44.894 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.80264838Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802861246Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802881413Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802888967Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802896233Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802903826Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802909687Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802915498Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802921384Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802927127Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802932646Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802938217Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802944227Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-24T12:27:44.895 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=settings t=2024-03-24T12:27:44.802949957Z level=info msg="App mode production" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=sqlstore t=2024-03-24T12:27:44.802996861Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=sqlstore t=2024-03-24T12:27:44.803013884Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.803867415Z level=info msg="Starting DB migrations" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.803950353Z level=info msg="Executing migration" id="create migration_log table" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.807212425Z level=info msg="Executing migration" id="create user table" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.810070327Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.810534324Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.813603796Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.816422546Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.819227706Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.822002332Z level=info msg="Executing migration" id="create user table v2" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.824863404Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.827726404Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-24T12:27:44.896 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.830598177Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.833322911Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.836198379Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.838772278Z level=info msg="Executing migration" id="Update user table charset" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.839082904Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.841892847Z level=info msg="Executing migration" id="Add missing user data" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.842258666Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.84264775Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.843064285Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.845375545Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.847966186Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.850704441Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.85116965Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.853763973Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-24T12:27:44.897 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.856444176Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-24T12:27:44.898 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.858953196Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-24T12:27:44.898 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.861559944Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-24T12:27:44.898 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.864195802Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-24T12:27:44.898 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.866748149Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-24T12:27:44.898 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.86927334Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-24T12:27:44.898 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.870119671Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.870954935Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.871811609Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.872707739Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.873507459Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.873878237Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.874359831Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.875136242Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.875504798Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.875990989Z level=info msg="Executing migration" id="create star table" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.876374848Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.877282625Z level=info msg="Executing migration" id="create org table v1" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.878866342Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.879325144Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-24T12:27:44.899 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.879727404Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.880585445Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.881441589Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.88187748Z level=info msg="Executing migration" id="Update org table charset" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.882707245Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.887001869Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.889807139Z level=info msg="Executing migration" id="create dashboard table" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.890292421Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.890748874Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.891868786Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.892323043Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.895832659Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.896525429Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.89754073Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-24T12:27:44.900 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.898318205Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-24T12:27:45.145 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-mon[33748]: 4.12 scrub starts 2024-03-24T12:27:45.146 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-mon[33748]: 4.12 scrub ok 2024-03-24T12:27:45.146 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-mon[33748]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:45.146 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.146 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.146 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.146 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.146 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-mon[33748]: Deploying daemon node-exporter.a on smithi029 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.900749601Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.903296853Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.903677161Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.904169277Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.906793722Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.907419513Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.910055126Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.910590453Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.913011355Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.913541056Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.91622879Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-24T12:27:45.147 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.916786697Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.918802073Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.919183829Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.919706678Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.92254885Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.92504336Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.927644821Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.928002486Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.930896096Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.933420797Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.935936585Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.936582235Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.937368402Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-24T12:27:45.148 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.938395784Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.939361465Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.93993803Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.94080736Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.941716834Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.942415465Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.942918151Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.943716495Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.944562766Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.945375626Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.945795869Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.946749189Z level=info msg="Executing migration" id="create data_source table" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.947396499Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-24T12:27:45.149 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.947795911Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-24T12:27:45.150 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.948245942Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.949083076Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.949648054Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.95066316Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.951496576Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.951918946Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.95279123Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.95381088Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.955314835Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.956014812Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.956865539Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-24T12:27:45.151 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.957671296Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-24T12:27:45.152 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.958582015Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-24T12:27:45.152 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.958918985Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.959776251Z level=info msg="Executing migration" id="Add uid column" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.960605864Z level=info msg="Executing migration" id="Update uid value" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.961479905Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.962537576Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.963172927Z level=info msg="Executing migration" id="create api_key table" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.964002677Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.964419225Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.967241565Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.969992738Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.972868923Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.97564266Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-24T12:27:45.153 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.978353662Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-24T12:27:45.154 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.981268413Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-24T12:27:45.154 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.984120005Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-24T12:27:45.154 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.986881269Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-24T12:27:45.154 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.989643742Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-24T12:27:45.154 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.992506639Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-24T12:27:45.154 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.995116846Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-24T12:27:45.154 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.995485191Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-24T12:27:45.154 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.998215704Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:44.998800192Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.001037356Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.003871942Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.00666595Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.00931279Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.011998816Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.014886162Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.017470468Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.020180217Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.022875889Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.025725195Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.026067662Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-24T12:27:45.155 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.028797972Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.031610715Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.03415801Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.036821059Z level=info msg="Executing migration" id="create quota table v1" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.039372636Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.040077673Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.040847615Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.041863165Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.042562078Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.043414056Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.044250401Z level=info msg="Executing migration" id="create session table" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.045133247Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.045921844Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.046810526Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-24T12:27:45.156 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.047567976Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.048547699Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.049409529Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.050107613Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.051055777Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.051783866Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.052654337Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.053444623Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.054120946Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.054420897Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.055338532Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.056173039Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.05706615Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.058046759Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-24T12:27:45.157 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.058803909Z level=info msg="Executing migration" id="create alert table v1" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.059634459Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.060632506Z level=info msg="Executing migration" id="add index alert state" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.061368869Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.062283491Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.063231682Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.063963225Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.064960671Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.066535288Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.06738395Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.068242375Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.069171122Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.071975156Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-24T12:27:45.158 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.074723631Z level=info msg="Executing migration" id="Add column is_default" 2024-03-24T12:27:45.159 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.077651994Z level=info msg="Executing migration" id="Add column frequency" 2024-03-24T12:27:45.159 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.080588769Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-24T12:27:45.159 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.083240724Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-24T12:27:45.159 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.08603656Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-24T12:27:45.159 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.088914991Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-24T12:27:45.159 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.091680839Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-24T12:27:45.159 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.094335704Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-24T12:27:45.159 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.097243399Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-24T12:27:45.160 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.100115414Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-24T12:27:45.160 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.103052827Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-24T12:27:45.160 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.105864312Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-24T12:27:45.160 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.108620968Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-24T12:27:45.160 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.111566598Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-24T12:27:45.160 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.114277209Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-24T12:27:45.160 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.117167328Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-24T12:27:45.160 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.120080485Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-24T12:27:45.161 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.122731103Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-24T12:27:45.161 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.125277747Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-24T12:27:45.161 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.128056087Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-24T12:27:45.161 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.130387111Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-24T12:27:45.161 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.132961663Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-24T12:27:45.161 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.135745854Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-24T12:27:45.161 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.138316274Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-24T12:27:45.161 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.140949284Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-24T12:27:45.161 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.143732599Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-24T12:27:45.162 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.146364313Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-24T12:27:45.162 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.148848546Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-24T12:27:45.162 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.151558714Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-24T12:27:45.162 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.154060514Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-24T12:27:45.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[41500]: 4.12 scrub starts 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[41500]: 4.12 scrub ok 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[41500]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[41500]: Deploying daemon node-exporter.a on smithi029 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[29919]: 4.12 scrub starts 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[29919]: 4.12 scrub ok 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[29919]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:45.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:45.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:44 smithi029 ceph-mon[29919]: Deploying daemon node-exporter.a on smithi029 2024-03-24T12:27:45.397 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.156751197Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-24T12:27:45.397 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.159469397Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-24T12:27:45.397 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.162102504Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-24T12:27:45.397 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.16474845Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-24T12:27:45.397 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.165510763Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-24T12:27:45.397 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.166394919Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-24T12:27:45.397 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.167974732Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-24T12:27:45.397 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.168939021Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-24T12:27:45.397 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.169663746Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-24T12:27:45.397 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.170561005Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-24T12:27:45.398 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.171330803Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-24T12:27:45.398 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.172304364Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-24T12:27:45.398 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.173034593Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-24T12:27:45.398 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.173890401Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-24T12:27:45.398 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.174733929Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-24T12:27:45.398 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.175584913Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-24T12:27:45.398 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.176524014Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-24T12:27:45.398 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.177282878Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-24T12:27:45.399 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.178179397Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-24T12:27:45.399 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.17898422Z level=info msg="Executing migration" id="Move region to single row" 2024-03-24T12:27:45.399 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.179976806Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-24T12:27:45.399 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.180732072Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-24T12:27:45.399 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.181732269Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-24T12:27:45.399 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.182507699Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-24T12:27:45.399 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.18335145Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-24T12:27:45.399 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.184418073Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-24T12:27:45.399 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.185096405Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-24T12:27:45.399 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.186071693Z level=info msg="Executing migration" id="create test_data table" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.186811388Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.187816739Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.188569031Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.189537655Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.190346997Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.191303557Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.192022365Z level=info msg="Executing migration" id="create team table" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.193025979Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.193905299Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.194650296Z level=info msg="Executing migration" id="create team member table" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.195635688Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.198275184Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-24T12:27:45.400 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.20102413Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.204048086Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.206771663Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.209521899Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.212351853Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.215164378Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.217888961Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.220788454Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.223570818Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.22658152Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.229185781Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.232093199Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.232596209Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.235546825Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-24T12:27:45.401 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.238321369Z level=info msg="Executing migration" id="create tag table" 2024-03-24T12:27:45.402 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.241113283Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-24T12:27:45.402 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.241538887Z level=info msg="Executing migration" id="create login attempt table" 2024-03-24T12:27:45.402 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.244279257Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-24T12:27:45.402 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.246983275Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-24T12:27:45.402 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.247657825Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-24T12:27:45.402 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.249511745Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-24T12:27:45.402 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.250364881Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-24T12:27:45.402 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.251265099Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-24T12:27:45.402 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.25197422Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-24T12:27:45.402 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.252372095Z level=info msg="Executing migration" id="create user auth table" 2024-03-24T12:27:45.403 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.253164257Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-24T12:27:45.403 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.253599147Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-24T12:27:45.403 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.253899283Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-24T12:27:45.404 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.254651874Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-24T12:27:45.404 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.255430457Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-24T12:27:45.404 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.256226884Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-24T12:27:45.404 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.257178784Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-24T12:27:45.404 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.257921685Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-24T12:27:45.404 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.258828725Z level=info msg="Executing migration" id="create server_lock table" 2024-03-24T12:27:45.404 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.259481082Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-24T12:27:45.405 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.260336786Z level=info msg="Executing migration" id="create user auth token table" 2024-03-24T12:27:45.405 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.261078827Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-24T12:27:45.405 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.262088794Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.262526917Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.263374161Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.264183266Z level=info msg="Executing migration" id="create cache_data table" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.265066252Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.265524671Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.266398554Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.267260274Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.267566651Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.267832488Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.26830694Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.269178913Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.269766945Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-24T12:27:45.406 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.270078422Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.270476444Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.270883925Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.2713262Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.271746801Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.272554603Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.273029345Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.273335007Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.273719407Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.274162085Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.274612742Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.274892381Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.275349602Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.275726877Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.276162011Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-24T12:27:45.407 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.276620733Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.277458512Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.277904972Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.278335251Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.281723866Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.285086915Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.285558393Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.285988768Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.286852785Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.287303132Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.287731875Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.28820154Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.28863313Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-24T12:27:45.408 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.288907012Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.289761553Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.290609411Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.29145414Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.291882577Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.292340896Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.293175912Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.294004914Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.294482131Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.295313092Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.296218231Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.296557345Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.297111013Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.297943534Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-24T12:27:45.409 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.298516751Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.298868312Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.299800499Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.300742615Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.301578851Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.30241825Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.303261787Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.303568622Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.303959557Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.30478889Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.305119291Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.305990636Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.306974444Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-24T12:27:45.410 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.307996052Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.308408778Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.309254986Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.310168603Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.313196114Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.31370938Z level=info msg="Executing migration" id="create alert_image table" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.316665062Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.319431227Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.319915714Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.321977489Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.324792654Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.324893024Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.327708084Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.330567155Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-24T12:27:45.411 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.331024736Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.331157288Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.331272093Z level=info msg="alerts found to migrate" alerts=0 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.333856865Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.336712421Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.339477328Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.342233023Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.344857953Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.347629775Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.350455742Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.350931776Z level=info msg="Executing migration" id="create data_keys table" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.35369396Z level=info msg="Executing migration" id="create secrets table" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.356585471Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-24T12:27:45.412 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.360856408Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-24T12:27:45.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.3617496Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-24T12:27:45.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.364513952Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-24T12:27:45.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.368902516Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-24T12:27:45.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.373124821Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-24T12:27:45.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.373527976Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-24T12:27:45.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.376169794Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-24T12:27:45.413 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.378982723Z level=info msg="Executing migration" id="create permission table" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.381582039Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.382038798Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.384523554Z level=info msg="Executing migration" id="create role table" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.387259597Z level=info msg="Executing migration" id="add column display_name" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.389994906Z level=info msg="Executing migration" id="add column group_name" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.390937775Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.393592838Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.39613671Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.397043614Z level=info msg="Executing migration" id="create team role table" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.397633879Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.398527185Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.399412779Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-24T12:27:45.414 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.400104815Z level=info msg="Executing migration" id="create user role table" 2024-03-24T12:27:45.415 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.400640578Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-24T12:27:45.415 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.401455679Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-24T12:27:45.415 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.402276362Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-24T12:27:45.415 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.403090563Z level=info msg="Executing migration" id="create builtin role table" 2024-03-24T12:27:45.415 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.403653187Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-24T12:27:45.885 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.4064597Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-24T12:27:45.885 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.409245713Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-24T12:27:45.885 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.411889849Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-24T12:27:45.885 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.41237701Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-24T12:27:45.885 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.415029343Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-24T12:27:45.885 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.417523933Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-24T12:27:45.885 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.419971368Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-24T12:27:45.885 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.420369981Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-24T12:27:45.885 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.423132337Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-24T12:27:45.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.42564309Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-24T12:27:45.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.428344732Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-24T12:27:45.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.428828817Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-24T12:27:45.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.431494064Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-24T12:27:45.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.434204177Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-24T12:27:45.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.436883226Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-24T12:27:45.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.437411368Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-24T12:27:45.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.440167202Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-24T12:27:45.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.440958206Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-24T12:27:45.886 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.44279652Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-24T12:27:45.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.443705136Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-24T12:27:45.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.444709857Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-24T12:27:45.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.445562464Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-24T12:27:45.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.446580854Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-24T12:27:45.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.447261734Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-24T12:27:45.887 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.447749664Z level=info msg="Executing migration" id="add correlation config column" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.448723347Z level=info msg="Executing migration" id="create entity_events table" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.449547835Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.450322758Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.450425559Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.450745712Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.450846346Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.451665898Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.452426159Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.453371035Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.45394019Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.454762034Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.455560276Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T12:27:45.888 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.456561581Z level=info msg="Executing migration" id="Drop public config table" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.457173263Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.458045184Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.459018604Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.459757371Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.460198048Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.463171681Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.464156022Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.465156748Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.465469974Z level=info msg="Executing migration" id="add share column" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.466459829Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.467478486Z level=info msg="Executing migration" id="create file table" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.46819977Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.469005578Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-24T12:27:45.889 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.469909427Z level=info msg="Executing migration" id="create file_meta table" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.470760527Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.471627991Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.472382531Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.472923297Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.475887601Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.478608076Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.479729849Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.482560602Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.485290599Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.488023615Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.488368259Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.491135557Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-24T12:27:45.890 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.492341435Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-24T12:27:45.891 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.494013141Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-24T12:27:45.891 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.496814423Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-24T12:27:45.891 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.506622014Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-24T12:27:45.891 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.509535451Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-24T12:27:45.891 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.512322425Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-24T12:27:45.891 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.515740154Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-24T12:27:45.891 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.516090665Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-24T12:27:45.891 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.518994149Z level=info msg="Executing migration" id="create folder table" 2024-03-24T12:27:45.891 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.521871601Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-24T12:27:45.891 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.524464044Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-24T12:27:45.892 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.524927411Z level=info msg="Executing migration" id="Update folder title length" 2024-03-24T12:27:45.892 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.527447273Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-24T12:27:45.892 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=migrator t=2024-03-24T12:27:45.530238387Z level=info msg="migrations completed" performed=485 skipped=0 duration=726.316083ms 2024-03-24T12:27:45.892 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=sqlstore t=2024-03-24T12:27:45.530650756Z level=info msg="Created default organization" 2024-03-24T12:27:45.892 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=plugin.loader t=2024-03-24T12:27:45.589309624Z level=info msg="Plugin registered" pluginID=input 2024-03-24T12:27:45.892 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=plugin.loader t=2024-03-24T12:27:45.616848994Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-24T12:27:45.892 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=plugin.loader t=2024-03-24T12:27:45.616889549Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-24T12:27:45.892 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=secrets t=2024-03-24T12:27:45.616921987Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-24T12:27:45.892 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=query_data t=2024-03-24T12:27:45.619079113Z level=info msg="Query Service initialization" 2024-03-24T12:27:45.892 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=live.push_http t=2024-03-24T12:27:45.621507316Z level=info msg="Live Push Gateway initialization" 2024-03-24T12:27:45.959 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[41500]: 5.1b scrub starts 2024-03-24T12:27:45.959 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[41500]: 5.1b scrub ok 2024-03-24T12:27:46.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-mon[33748]: 5.1b scrub starts 2024-03-24T12:27:46.224 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-mon[33748]: 5.1b scrub ok 2024-03-24T12:27:46.225 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:27:46.225 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:27:46.225 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-mon[33748]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T12:27:46.225 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:27:46.225 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:45 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:27:46.344 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[29919]: 5.1b scrub starts 2024-03-24T12:27:46.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[29919]: 5.1b scrub ok 2024-03-24T12:27:46.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:27:46.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:27:46.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[29919]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T12:27:46.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:27:46.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:27:46.345 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:27:46.345 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:27:46.345 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[41500]: osdmap e93: 8 total, 8 up, 8 in 2024-03-24T12:27:46.345 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:27:46.346 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:45 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:27:46.371 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:27:46.373 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:27:46.634 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=infra.usagestats.collector t=2024-03-24T12:27:46.227744185Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-24T12:27:46.634 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=provisioning.datasources t=2024-03-24T12:27:46.228822329Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-24T12:27:46.634 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=provisioning.datasources t=2024-03-24T12:27:46.239739309Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-24T12:27:46.634 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=provisioning.alerting t=2024-03-24T12:27:46.2503376Z level=info msg="starting to provision alerting" 2024-03-24T12:27:46.635 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=provisioning.alerting t=2024-03-24T12:27:46.25035584Z level=info msg="finished to provision alerting" 2024-03-24T12:27:46.635 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=grafanaStorageLogger t=2024-03-24T12:27:46.250501774Z level=info msg="storage starting" 2024-03-24T12:27:46.635 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=ngalert.state.manager t=2024-03-24T12:27:46.25077745Z level=info msg="Warming state cache for startup" 2024-03-24T12:27:46.635 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=ngalert.state.manager t=2024-03-24T12:27:46.256731938Z level=info msg="State cache has been initialized" states=0 duration=5.952488ms 2024-03-24T12:27:46.635 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=ticker t=2024-03-24T12:27:46.25703518Z level=info msg=starting first_tick=2024-03-24T12:27:50Z 2024-03-24T12:27:46.635 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=ngalert.multiorg.alertmanager t=2024-03-24T12:27:46.257675877Z level=info msg="starting MultiOrg Alertmanager" 2024-03-24T12:27:46.635 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=http.server t=2024-03-24T12:27:46.258217778Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-24T12:27:46.635 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=sqlstore.transactions t=2024-03-24T12:27:46.268215954Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T12:27:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-mon[33748]: 5.3 scrub starts 2024-03-24T12:27:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-mon[33748]: 5.3 scrub ok 2024-03-24T12:27:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-mon[33748]: 5.4 scrub starts 2024-03-24T12:27:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-mon[33748]: 5.4 scrub ok 2024-03-24T12:27:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-mon[33748]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-mon[33748]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:47.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:47.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:47.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[41500]: 5.3 scrub starts 2024-03-24T12:27:47.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[41500]: 5.3 scrub ok 2024-03-24T12:27:47.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[41500]: 5.4 scrub starts 2024-03-24T12:27:47.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[41500]: 5.4 scrub ok 2024-03-24T12:27:47.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[41500]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:47.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[41500]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:47.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:47.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:47.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[29919]: 5.3 scrub starts 2024-03-24T12:27:47.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[29919]: 5.3 scrub ok 2024-03-24T12:27:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[29919]: 5.4 scrub starts 2024-03-24T12:27:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[29919]: 5.4 scrub ok 2024-03-24T12:27:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[29919]: from='client.14748 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:27:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[29919]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:47.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:27:47.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:27:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:27:47.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:27:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:27:48.049 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-03-24T12:27:44.959249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1026,"num_read_kb":872,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":632,"ondisk_log_size":632,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":237052,"kb_used_data":8444,"kb_used_omap":12,"kb_used_meta":228531,"kb_avail":749724164,"statfs":{"total":767960285184,"available":767717543936,"internally_reserved":0,"allocated":8646656,"data_stored":4765610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":234016305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.090136"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870600+0000","last_change":"2024-03-24T12:26:09.350829+0000","last_active":"2024-03-24T12:27:34.870600+0000","last_peered":"2024-03-24T12:27:34.870600+0000","last_clean":"2024-03-24T12:27:34.870600+0000","last_became_active":"2024-03-24T12:26:09.350521+0000","last_became_peered":"2024-03-24T12:26:09.350521+0000","last_unstale":"2024-03-24T12:27:34.870600+0000","last_undegraded":"2024-03-24T12:27:34.870600+0000","last_fullsized":"2024-03-24T12:27:34.870600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870389+0000","last_change":"2024-03-24T12:26:07.335814+0000","last_active":"2024-03-24T12:27:34.870389+0000","last_peered":"2024-03-24T12:27:34.870389+0000","last_clean":"2024-03-24T12:27:34.870389+0000","last_became_active":"2024-03-24T12:26:07.335517+0000","last_became_peered":"2024-03-24T12:26:07.335517+0000","last_unstale":"2024-03-24T12:27:34.870389+0000","last_undegraded":"2024-03-24T12:27:34.870389+0000","last_fullsized":"2024-03-24T12:27:34.870389+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044306+0000","last_change":"2024-03-24T12:26:03.171110+0000","last_active":"2024-03-24T12:27:35.044306+0000","last_peered":"2024-03-24T12:27:35.044306+0000","last_clean":"2024-03-24T12:27:35.044306+0000","last_became_active":"2024-03-24T12:26:03.170822+0000","last_became_peered":"2024-03-24T12:26:03.170822+0000","last_unstale":"2024-03-24T12:27:35.044306+0000","last_undegraded":"2024-03-24T12:27:35.044306+0000","last_fullsized":"2024-03-24T12:27:35.044306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870319+0000","last_change":"2024-03-24T12:27:24.141489+0000","last_active":"2024-03-24T12:27:34.870319+0000","last_peered":"2024-03-24T12:27:34.870319+0000","last_clean":"2024-03-24T12:27:34.870319+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:27:34.870319+0000","last_undegraded":"2024-03-24T12:27:34.870319+0000","last_fullsized":"2024-03-24T12:27:34.870319+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:24.141325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.141325+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.796727+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494713+0000","last_change":"2024-03-24T12:27:35.494737+0000","last_active":"2024-03-24T12:27:35.494713+0000","last_peered":"2024-03-24T12:27:35.494713+0000","last_clean":"2024-03-24T12:27:35.494713+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:27:35.494713+0000","last_undegraded":"2024-03-24T12:27:35.494713+0000","last_fullsized":"2024-03-24T12:27:35.494713+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:35.494635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:35.494635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:48.670706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870321+0000","last_change":"2024-03-24T12:27:34.234548+0000","last_active":"2024-03-24T12:27:34.870321+0000","last_peered":"2024-03-24T12:27:34.870321+0000","last_clean":"2024-03-24T12:27:34.870321+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:27:34.870321+0000","last_undegraded":"2024-03-24T12:27:34.870321+0000","last_fullsized":"2024-03-24T12:27:34.870321+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:34.234417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.234417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.166756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:37.470875+0000","last_change":"2024-03-24T12:27:37.470895+0000","last_active":"2024-03-24T12:27:37.470875+0000","last_peered":"2024-03-24T12:27:37.470875+0000","last_clean":"2024-03-24T12:27:37.470875+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:27:37.470875+0000","last_undegraded":"2024-03-24T12:27:37.470875+0000","last_fullsized":"2024-03-24T12:27:37.470875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:37.470809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:37.470809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:01.645277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"90'22","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493873+0000","last_change":"2024-03-24T12:27:25.396689+0000","last_active":"2024-03-24T12:27:35.493873+0000","last_peered":"2024-03-24T12:27:35.493873+0000","last_clean":"2024-03-24T12:27:35.493873+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:27:35.493873+0000","last_undegraded":"2024-03-24T12:27:35.493873+0000","last_fullsized":"2024-03-24T12:27:35.493873+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.396514+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.161411+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043813+0000","last_change":"2024-03-24T12:27:23.289755+0000","last_active":"2024-03-24T12:27:35.043813+0000","last_peered":"2024-03-24T12:27:35.043813+0000","last_clean":"2024-03-24T12:27:35.043813+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:27:35.043813+0000","last_undegraded":"2024-03-24T12:27:35.043813+0000","last_fullsized":"2024-03-24T12:27:35.043813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.289666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.684236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569461+0000","last_change":"2024-03-24T12:26:07.331880+0000","last_active":"2024-03-24T12:27:35.569461+0000","last_peered":"2024-03-24T12:27:35.569461+0000","last_clean":"2024-03-24T12:27:35.569461+0000","last_became_active":"2024-03-24T12:26:07.331562+0000","last_became_peered":"2024-03-24T12:26:07.331562+0000","last_unstale":"2024-03-24T12:27:35.569461+0000","last_undegraded":"2024-03-24T12:27:35.569461+0000","last_fullsized":"2024-03-24T12:27:35.569461+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570257+0000","last_change":"2024-03-24T12:27:20.255011+0000","last_active":"2024-03-24T12:27:35.570257+0000","last_peered":"2024-03-24T12:27:35.570257+0000","last_clean":"2024-03-24T12:27:35.570257+0000","last_became_active":"2024-03-24T12:26:03.161544+0000","last_became_peered":"2024-03-24T12:26:03.161544+0000","last_unstale":"2024-03-24T12:27:35.570257+0000","last_undegraded":"2024-03-24T12:27:35.570257+0000","last_fullsized":"2024-03-24T12:27:35.570257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:20.254898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.254898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.151491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870711+0000","last_change":"2024-03-24T12:26:05.318208+0000","last_active":"2024-03-24T12:27:34.870711+0000","last_peered":"2024-03-24T12:27:34.870711+0000","last_clean":"2024-03-24T12:27:34.870711+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:27:34.870711+0000","last_undegraded":"2024-03-24T12:27:34.870711+0000","last_fullsized":"2024-03-24T12:27:34.870711+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569689+0000","last_change":"2024-03-24T12:27:21.260071+0000","last_active":"2024-03-24T12:27:35.569689+0000","last_peered":"2024-03-24T12:27:35.569689+0000","last_clean":"2024-03-24T12:27:35.569689+0000","last_became_active":"2024-03-24T12:26:09.350721+0000","last_became_peered":"2024-03-24T12:26:09.350721+0000","last_unstale":"2024-03-24T12:27:35.569689+0000","last_undegraded":"2024-03-24T12:27:35.569689+0000","last_fullsized":"2024-03-24T12:27:35.569689+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.259962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.801612+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391054+0000","last_change":"2024-03-24T12:27:17.365794+0000","last_active":"2024-03-24T12:27:35.391054+0000","last_peered":"2024-03-24T12:27:35.391054+0000","last_clean":"2024-03-24T12:27:35.391054+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:27:35.391054+0000","last_undegraded":"2024-03-24T12:27:35.391054+0000","last_fullsized":"2024-03-24T12:27:35.391054+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.365696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.873148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570142+0000","last_change":"2024-03-24T12:27:10.251781+0000","last_active":"2024-03-24T12:27:35.570142+0000","last_peered":"2024-03-24T12:27:35.570142+0000","last_clean":"2024-03-24T12:27:35.570142+0000","last_became_active":"2024-03-24T12:26:03.159997+0000","last_became_peered":"2024-03-24T12:26:03.159997+0000","last_unstale":"2024-03-24T12:27:35.570142+0000","last_undegraded":"2024-03-24T12:27:35.570142+0000","last_fullsized":"2024-03-24T12:27:35.570142+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.251686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.902316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":75,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:38.276134+0000","last_change":"2024-03-24T12:27:38.276194+0000","last_active":"2024-03-24T12:27:38.276134+0000","last_peered":"2024-03-24T12:27:38.276134+0000","last_clean":"2024-03-24T12:27:38.276134+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:27:38.276134+0000","last_undegraded":"2024-03-24T12:27:38.276134+0000","last_fullsized":"2024-03-24T12:27:38.276134+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T12:27:38.276034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.276034+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:06.748494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871176+0000","last_change":"2024-03-24T12:27:25.164561+0000","last_active":"2024-03-24T12:27:34.871176+0000","last_peered":"2024-03-24T12:27:34.871176+0000","last_clean":"2024-03-24T12:27:34.871176+0000","last_became_active":"2024-03-24T12:26:09.643671+0000","last_became_peered":"2024-03-24T12:26:09.643671+0000","last_unstale":"2024-03-24T12:27:34.871176+0000","last_undegraded":"2024-03-24T12:27:34.871176+0000","last_fullsized":"2024-03-24T12:27:34.871176+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:25.164460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.164460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.284324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871818+0000","last_change":"2024-03-24T12:27:28.397274+0000","last_active":"2024-03-24T12:27:34.871818+0000","last_peered":"2024-03-24T12:27:34.871818+0000","last_clean":"2024-03-24T12:27:34.871818+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:27:34.871818+0000","last_undegraded":"2024-03-24T12:27:34.871818+0000","last_fullsized":"2024-03-24T12:27:34.871818+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.397179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.397179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.594543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044310+0000","last_change":"2024-03-24T12:27:28.361014+0000","last_active":"2024-03-24T12:27:35.044310+0000","last_peered":"2024-03-24T12:27:35.044310+0000","last_clean":"2024-03-24T12:27:35.044310+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:27:35.044310+0000","last_undegraded":"2024-03-24T12:27:35.044310+0000","last_fullsized":"2024-03-24T12:27:35.044310+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.360926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.835122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870865+0000","last_change":"2024-03-24T12:27:33.582805+0000","last_active":"2024-03-24T12:27:34.870865+0000","last_peered":"2024-03-24T12:27:34.870865+0000","last_clean":"2024-03-24T12:27:34.870865+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:27:34.870865+0000","last_undegraded":"2024-03-24T12:27:34.870865+0000","last_fullsized":"2024-03-24T12:27:34.870865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:27:33.582711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:33.582711+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.312118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494584+0000","last_change":"2024-03-24T12:27:30.441572+0000","last_active":"2024-03-24T12:27:35.494584+0000","last_peered":"2024-03-24T12:27:35.494584+0000","last_clean":"2024-03-24T12:27:35.494584+0000","last_became_active":"2024-03-24T12:26:09.643738+0000","last_became_peered":"2024-03-24T12:26:09.643738+0000","last_unstale":"2024-03-24T12:27:35.494584+0000","last_undegraded":"2024-03-24T12:27:35.494584+0000","last_fullsized":"2024-03-24T12:27:35.494584+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:30.441497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:30.441497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.867059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:39.295925+0000","last_change":"2024-03-24T12:27:39.295946+0000","last_active":"2024-03-24T12:27:39.295925+0000","last_peered":"2024-03-24T12:27:39.295925+0000","last_clean":"2024-03-24T12:27:39.295925+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:27:39.295925+0000","last_undegraded":"2024-03-24T12:27:39.295925+0000","last_fullsized":"2024-03-24T12:27:39.295925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:27:39.295864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:07.017784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870775+0000","last_change":"2024-03-24T12:26:03.160687+0000","last_active":"2024-03-24T12:27:34.870775+0000","last_peered":"2024-03-24T12:27:34.870775+0000","last_clean":"2024-03-24T12:27:34.870775+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:27:34.870775+0000","last_undegraded":"2024-03-24T12:27:34.870775+0000","last_fullsized":"2024-03-24T12:27:34.870775+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":88,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494411+0000","last_change":"2024-03-24T12:27:32.464080+0000","last_active":"2024-03-24T12:27:35.494411+0000","last_peered":"2024-03-24T12:27:35.494411+0000","last_clean":"2024-03-24T12:27:35.494411+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:27:35.494411+0000","last_undegraded":"2024-03-24T12:27:35.494411+0000","last_fullsized":"2024-03-24T12:27:35.494411+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:32.463972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.463972+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:58.687386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":58,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391678+0000","last_change":"2024-03-24T12:27:34.409721+0000","last_active":"2024-03-24T12:27:35.391678+0000","last_peered":"2024-03-24T12:27:35.391678+0000","last_clean":"2024-03-24T12:27:35.391678+0000","last_became_active":"2024-03-24T12:26:09.343786+0000","last_became_peered":"2024-03-24T12:26:09.343786+0000","last_unstale":"2024-03-24T12:27:35.391678+0000","last_undegraded":"2024-03-24T12:27:35.391678+0000","last_fullsized":"2024-03-24T12:27:35.391678+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:27:34.409577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.409577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:55.581370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":92,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:36.723422+0000","last_change":"2024-03-24T12:27:36.723422+0000","last_active":"2024-03-24T12:27:36.723422+0000","last_peered":"2024-03-24T12:27:36.723422+0000","last_clean":"2024-03-24T12:27:36.723422+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:27:36.723422+0000","last_undegraded":"2024-03-24T12:27:36.723422+0000","last_fullsized":"2024-03-24T12:27:36.723422+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:36.723265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:36.723265+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.467176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870414+0000","last_change":"2024-03-24T12:27:27.409032+0000","last_active":"2024-03-24T12:27:34.870414+0000","last_peered":"2024-03-24T12:27:34.870414+0000","last_clean":"2024-03-24T12:27:34.870414+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:27:34.870414+0000","last_undegraded":"2024-03-24T12:27:34.870414+0000","last_fullsized":"2024-03-24T12:27:34.870414+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:27.408937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.408937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.920390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044300+0000","last_change":"2024-03-24T12:27:11.269339+0000","last_active":"2024-03-24T12:27:35.044300+0000","last_peered":"2024-03-24T12:27:35.044300+0000","last_clean":"2024-03-24T12:27:35.044300+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:27:35.044300+0000","last_undegraded":"2024-03-24T12:27:35.044300+0000","last_fullsized":"2024-03-24T12:27:35.044300+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:11.269181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:11.269181+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:34.970841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871884+0000","last_change":"2024-03-24T12:27:33.377434+0000","last_active":"2024-03-24T12:27:34.871884+0000","last_peered":"2024-03-24T12:27:34.871884+0000","last_clean":"2024-03-24T12:27:34.871884+0000","last_became_active":"2024-03-24T12:26:09.348355+0000","last_became_peered":"2024-03-24T12:26:09.348355+0000","last_unstale":"2024-03-24T12:27:34.871884+0000","last_undegraded":"2024-03-24T12:27:34.871884+0000","last_fullsized":"2024-03-24T12:27:34.871884+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:33.377351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:33.377351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.631210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044456+0000","last_change":"2024-03-24T12:26:07.331932+0000","last_active":"2024-03-24T12:27:35.044456+0000","last_peered":"2024-03-24T12:27:35.044456+0000","last_clean":"2024-03-24T12:27:35.044456+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:27:35.044456+0000","last_undegraded":"2024-03-24T12:27:35.044456+0000","last_fullsized":"2024-03-24T12:27:35.044456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493361+0000","last_change":"2024-03-24T12:27:12.437790+0000","last_active":"2024-03-24T12:27:35.493361+0000","last_peered":"2024-03-24T12:27:35.493361+0000","last_clean":"2024-03-24T12:27:35.493361+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:27:35.493361+0000","last_undegraded":"2024-03-24T12:27:35.493361+0000","last_fullsized":"2024-03-24T12:27:35.493361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.437708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:25.841012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":84,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.495063+0000","last_change":"2024-03-24T12:27:26.763428+0000","last_active":"2024-03-24T12:27:35.495063+0000","last_peered":"2024-03-24T12:27:35.495063+0000","last_clean":"2024-03-24T12:27:35.495063+0000","last_became_active":"2024-03-24T12:26:05.317587+0000","last_became_peered":"2024-03-24T12:26:05.317587+0000","last_unstale":"2024-03-24T12:27:35.495063+0000","last_undegraded":"2024-03-24T12:27:35.495063+0000","last_fullsized":"2024-03-24T12:27:35.495063+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:26.763317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.763317+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.728888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391387+0000","last_change":"2024-03-24T12:26:09.345188+0000","last_active":"2024-03-24T12:27:35.391387+0000","last_peered":"2024-03-24T12:27:35.391387+0000","last_clean":"2024-03-24T12:27:35.391387+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:27:35.391387+0000","last_undegraded":"2024-03-24T12:27:35.391387+0000","last_fullsized":"2024-03-24T12:27:35.391387+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:37.908175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494107+0000","last_change":"2024-03-24T12:27:34.747936+0000","last_active":"2024-03-24T12:27:35.494107+0000","last_peered":"2024-03-24T12:27:35.494107+0000","last_clean":"2024-03-24T12:27:35.494107+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:27:35.494107+0000","last_undegraded":"2024-03-24T12:27:35.494107+0000","last_fullsized":"2024-03-24T12:27:35.494107+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:34.747834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.747834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:50.783426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044307+0000","last_change":"2024-03-24T12:26:03.171104+0000","last_active":"2024-03-24T12:27:35.044307+0000","last_peered":"2024-03-24T12:27:35.044307+0000","last_clean":"2024-03-24T12:27:35.044307+0000","last_became_active":"2024-03-24T12:26:03.170395+0000","last_became_peered":"2024-03-24T12:26:03.170395+0000","last_unstale":"2024-03-24T12:27:35.044307+0000","last_undegraded":"2024-03-24T12:27:35.044307+0000","last_fullsized":"2024-03-24T12:27:35.044307+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":71,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493391+0000","last_change":"2024-03-24T12:27:15.467117+0000","last_active":"2024-03-24T12:27:35.493391+0000","last_peered":"2024-03-24T12:27:35.493391+0000","last_clean":"2024-03-24T12:27:35.493391+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:27:35.493391+0000","last_undegraded":"2024-03-24T12:27:35.493391+0000","last_fullsized":"2024-03-24T12:27:35.493391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:27:15.466966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:15.466966+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:23.327159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:38.510313+0000","last_change":"2024-03-24T12:27:38.510336+0000","last_active":"2024-03-24T12:27:38.510313+0000","last_peered":"2024-03-24T12:27:38.510313+0000","last_clean":"2024-03-24T12:27:38.510313+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:27:38.510313+0000","last_undegraded":"2024-03-24T12:27:38.510313+0000","last_fullsized":"2024-03-24T12:27:38.510313+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:38.510237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.510237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.100244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391030+0000","last_change":"2024-03-24T12:27:21.335557+0000","last_active":"2024-03-24T12:27:35.391030+0000","last_peered":"2024-03-24T12:27:35.391030+0000","last_clean":"2024-03-24T12:27:35.391030+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:27:35.391030+0000","last_undegraded":"2024-03-24T12:27:35.391030+0000","last_fullsized":"2024-03-24T12:27:35.391030+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:21.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.789829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":75,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570411+0000","last_change":"2024-03-24T12:27:34.193568+0000","last_active":"2024-03-24T12:27:35.570411+0000","last_peered":"2024-03-24T12:27:35.570411+0000","last_clean":"2024-03-24T12:27:35.570411+0000","last_became_active":"2024-03-24T12:26:03.165083+0000","last_became_peered":"2024-03-24T12:26:03.165083+0000","last_unstale":"2024-03-24T12:27:35.570411+0000","last_undegraded":"2024-03-24T12:27:35.570411+0000","last_fullsized":"2024-03-24T12:27:35.570411+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:34.193480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.193480+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:50.648254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":84,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493768+0000","last_change":"2024-03-24T12:27:16.497904+0000","last_active":"2024-03-24T12:27:35.493768+0000","last_peered":"2024-03-24T12:27:35.493768+0000","last_clean":"2024-03-24T12:27:35.493768+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:27:35.493768+0000","last_undegraded":"2024-03-24T12:27:35.493768+0000","last_fullsized":"2024-03-24T12:27:35.493768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:16.497755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.497755+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.894323+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043666+0000","last_change":"2024-03-24T12:26:09.351371+0000","last_active":"2024-03-24T12:27:35.043666+0000","last_peered":"2024-03-24T12:27:35.043666+0000","last_clean":"2024-03-24T12:27:35.043666+0000","last_became_active":"2024-03-24T12:26:09.350656+0000","last_became_peered":"2024-03-24T12:26:09.350656+0000","last_unstale":"2024-03-24T12:27:35.043666+0000","last_undegraded":"2024-03-24T12:27:35.043666+0000","last_fullsized":"2024-03-24T12:27:35.043666+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.492934+0000","last_change":"2024-03-24T12:27:20.458514+0000","last_active":"2024-03-24T12:27:35.492934+0000","last_peered":"2024-03-24T12:27:35.492934+0000","last_clean":"2024-03-24T12:27:35.492934+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:27:35.492934+0000","last_undegraded":"2024-03-24T12:27:35.492934+0000","last_fullsized":"2024-03-24T12:27:35.492934+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.458419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:24.491860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391814+0000","last_change":"2024-03-24T12:27:14.329516+0000","last_active":"2024-03-24T12:27:35.391814+0000","last_peered":"2024-03-24T12:27:35.391814+0000","last_clean":"2024-03-24T12:27:35.391814+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:27:35.391814+0000","last_undegraded":"2024-03-24T12:27:35.391814+0000","last_fullsized":"2024-03-24T12:27:35.391814+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:14.329357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:14.329357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:21.223400+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":51,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569825+0000","last_change":"2024-03-24T12:26:16.882174+0000","last_active":"2024-03-24T12:27:35.569825+0000","last_peered":"2024-03-24T12:27:35.569825+0000","last_clean":"2024-03-24T12:27:35.569825+0000","last_became_active":"2024-03-24T12:26:16.874170+0000","last_became_peered":"2024-03-24T12:26:16.874170+0000","last_unstale":"2024-03-24T12:27:35.569825+0000","last_undegraded":"2024-03-24T12:27:35.569825+0000","last_fullsized":"2024-03-24T12:27:35.569825+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493631+0000","last_change":"2024-03-24T12:27:26.431474+0000","last_active":"2024-03-24T12:27:35.493631+0000","last_peered":"2024-03-24T12:27:35.493631+0000","last_clean":"2024-03-24T12:27:35.493631+0000","last_became_active":"2024-03-24T12:26:09.343296+0000","last_became_peered":"2024-03-24T12:26:09.343296+0000","last_unstale":"2024-03-24T12:27:35.493631+0000","last_undegraded":"2024-03-24T12:27:35.493631+0000","last_fullsized":"2024-03-24T12:27:35.493631+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.431382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.685011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871005+0000","last_change":"2024-03-24T12:27:29.397575+0000","last_active":"2024-03-24T12:27:34.871005+0000","last_peered":"2024-03-24T12:27:34.871005+0000","last_clean":"2024-03-24T12:27:34.871005+0000","last_became_active":"2024-03-24T12:26:07.330811+0000","last_became_peered":"2024-03-24T12:26:07.330811+0000","last_unstale":"2024-03-24T12:27:34.871005+0000","last_undegraded":"2024-03-24T12:27:34.871005+0000","last_fullsized":"2024-03-24T12:27:34.871005+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:29.397463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:29.397463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:52.953466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870888+0000","last_change":"2024-03-24T12:27:09.137207+0000","last_active":"2024-03-24T12:27:34.870888+0000","last_peered":"2024-03-24T12:27:34.870888+0000","last_clean":"2024-03-24T12:27:34.870888+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:27:34.870888+0000","last_undegraded":"2024-03-24T12:27:34.870888+0000","last_fullsized":"2024-03-24T12:27:34.870888+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:09.137104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:09.137104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.329586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"88'16","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871349+0000","last_change":"2024-03-24T12:27:17.084272+0000","last_active":"2024-03-24T12:27:34.871349+0000","last_peered":"2024-03-24T12:27:34.871349+0000","last_clean":"2024-03-24T12:27:34.871349+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:27:34.871349+0000","last_undegraded":"2024-03-24T12:27:34.871349+0000","last_fullsized":"2024-03-24T12:27:34.871349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.084148+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.541132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":50,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043972+0000","last_change":"2024-03-24T12:26:09.349851+0000","last_active":"2024-03-24T12:27:35.043972+0000","last_peered":"2024-03-24T12:27:35.043972+0000","last_clean":"2024-03-24T12:27:35.043972+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:27:35.043972+0000","last_undegraded":"2024-03-24T12:27:35.043972+0000","last_fullsized":"2024-03-24T12:27:35.043972+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569527+0000","last_change":"2024-03-24T12:26:07.330555+0000","last_active":"2024-03-24T12:27:35.569527+0000","last_peered":"2024-03-24T12:27:35.569527+0000","last_clean":"2024-03-24T12:27:35.569527+0000","last_became_active":"2024-03-24T12:26:07.329487+0000","last_became_peered":"2024-03-24T12:26:07.329487+0000","last_unstale":"2024-03-24T12:27:35.569527+0000","last_undegraded":"2024-03-24T12:27:35.569527+0000","last_fullsized":"2024-03-24T12:27:35.569527+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391484+0000","last_change":"2024-03-24T12:26:03.163577+0000","last_active":"2024-03-24T12:27:35.391484+0000","last_peered":"2024-03-24T12:27:35.391484+0000","last_clean":"2024-03-24T12:27:35.391484+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:27:35.391484+0000","last_undegraded":"2024-03-24T12:27:35.391484+0000","last_fullsized":"2024-03-24T12:27:35.391484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494049+0000","last_change":"2024-03-24T12:27:21.451811+0000","last_active":"2024-03-24T12:27:35.494049+0000","last_peered":"2024-03-24T12:27:35.494049+0000","last_clean":"2024-03-24T12:27:35.494049+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:27:35.494049+0000","last_undegraded":"2024-03-24T12:27:35.494049+0000","last_fullsized":"2024-03-24T12:27:35.494049+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:21.451717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.451717+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.819749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:37.276934+0000","last_change":"2024-03-24T12:27:37.276957+0000","last_active":"2024-03-24T12:27:37.276934+0000","last_peered":"2024-03-24T12:27:37.276934+0000","last_clean":"2024-03-24T12:27:37.276934+0000","last_became_active":"2024-03-24T12:26:09.346062+0000","last_became_peered":"2024-03-24T12:26:09.346062+0000","last_unstale":"2024-03-24T12:27:37.276934+0000","last_undegraded":"2024-03-24T12:27:37.276934+0000","last_fullsized":"2024-03-24T12:27:37.276934+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:37.276875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:37.276875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.339012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493061+0000","last_change":"2024-03-24T12:26:07.338058+0000","last_active":"2024-03-24T12:27:35.493061+0000","last_peered":"2024-03-24T12:27:35.493061+0000","last_clean":"2024-03-24T12:27:35.493061+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:27:35.493061+0000","last_undegraded":"2024-03-24T12:27:35.493061+0000","last_fullsized":"2024-03-24T12:27:35.493061+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043881+0000","last_change":"2024-03-24T12:27:07.317120+0000","last_active":"2024-03-24T12:27:35.043881+0000","last_peered":"2024-03-24T12:27:35.043881+0000","last_clean":"2024-03-24T12:27:35.043881+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:27:35.043881+0000","last_undegraded":"2024-03-24T12:27:35.043881+0000","last_fullsized":"2024-03-24T12:27:35.043881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:07.317021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:07.317021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.086017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'13","reported_seq":85,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570422+0000","last_change":"2024-03-24T12:27:16.242881+0000","last_active":"2024-03-24T12:27:35.570422+0000","last_peered":"2024-03-24T12:27:35.570422+0000","last_clean":"2024-03-24T12:27:35.570422+0000","last_became_active":"2024-03-24T12:26:05.358213+0000","last_became_peered":"2024-03-24T12:26:05.358213+0000","last_unstale":"2024-03-24T12:27:35.570422+0000","last_undegraded":"2024-03-24T12:27:35.570422+0000","last_fullsized":"2024-03-24T12:27:35.570422+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.242776+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.762237+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871090+0000","last_change":"2024-03-24T12:27:27.661082+0000","last_active":"2024-03-24T12:27:34.871090+0000","last_peered":"2024-03-24T12:27:34.871090+0000","last_clean":"2024-03-24T12:27:34.871090+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:27:34.871090+0000","last_undegraded":"2024-03-24T12:27:34.871090+0000","last_fullsized":"2024-03-24T12:27:34.871090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.660932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.997979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:39.472978+0000","last_change":"2024-03-24T12:27:39.473015+0000","last_active":"2024-03-24T12:27:39.472978+0000","last_peered":"2024-03-24T12:27:39.472978+0000","last_clean":"2024-03-24T12:27:39.472978+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:27:39.472978+0000","last_undegraded":"2024-03-24T12:27:39.472978+0000","last_fullsized":"2024-03-24T12:27:39.472978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:39.472900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:39.472900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:04.877211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391468+0000","last_change":"2024-03-24T12:27:06.422517+0000","last_active":"2024-03-24T12:27:35.391468+0000","last_peered":"2024-03-24T12:27:35.391468+0000","last_clean":"2024-03-24T12:27:35.391468+0000","last_became_active":"2024-03-24T12:26:03.166149+0000","last_became_peered":"2024-03-24T12:26:03.166149+0000","last_unstale":"2024-03-24T12:27:35.391468+0000","last_undegraded":"2024-03-24T12:27:35.391468+0000","last_fullsized":"2024-03-24T12:27:35.391468+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:06.422387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.422387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:12.269750+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044352+0000","last_change":"2024-03-24T12:27:30.384123+0000","last_active":"2024-03-24T12:27:35.044352+0000","last_peered":"2024-03-24T12:27:35.044352+0000","last_clean":"2024-03-24T12:27:35.044352+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:27:35.044352+0000","last_undegraded":"2024-03-24T12:27:35.044352+0000","last_fullsized":"2024-03-24T12:27:35.044352+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:30.384035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:30.384035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:48.792180+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:42.333908+0000","last_change":"2024-03-24T12:27:42.333953+0000","last_active":"2024-03-24T12:27:42.333908+0000","last_peered":"2024-03-24T12:27:42.333908+0000","last_clean":"2024-03-24T12:27:42.333908+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:27:42.333908+0000","last_undegraded":"2024-03-24T12:27:42.333908+0000","last_fullsized":"2024-03-24T12:27:42.333908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:42.333813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:42.333813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:01.854155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569557+0000","last_change":"2024-03-24T12:26:07.329432+0000","last_active":"2024-03-24T12:27:35.569557+0000","last_peered":"2024-03-24T12:27:35.569557+0000","last_clean":"2024-03-24T12:27:35.569557+0000","last_became_active":"2024-03-24T12:26:07.329117+0000","last_became_peered":"2024-03-24T12:26:07.329117+0000","last_unstale":"2024-03-24T12:27:35.569557+0000","last_undegraded":"2024-03-24T12:27:35.569557+0000","last_fullsized":"2024-03-24T12:27:35.569557+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494270+0000","last_change":"2024-03-24T12:27:06.584912+0000","last_active":"2024-03-24T12:27:35.494270+0000","last_peered":"2024-03-24T12:27:35.494270+0000","last_clean":"2024-03-24T12:27:35.494270+0000","last_became_active":"2024-03-24T12:26:03.161414+0000","last_became_peered":"2024-03-24T12:26:03.161414+0000","last_unstale":"2024-03-24T12:27:35.494270+0000","last_undegraded":"2024-03-24T12:27:35.494270+0000","last_fullsized":"2024-03-24T12:27:35.494270+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:06.584809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.584809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:16.866693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":66,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494581+0000","last_change":"2024-03-24T12:26:05.357244+0000","last_active":"2024-03-24T12:27:35.494581+0000","last_peered":"2024-03-24T12:27:35.494581+0000","last_clean":"2024-03-24T12:27:35.494581+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:27:35.494581+0000","last_undegraded":"2024-03-24T12:27:35.494581+0000","last_fullsized":"2024-03-24T12:27:35.494581+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570086+0000","last_change":"2024-03-24T12:26:09.643942+0000","last_active":"2024-03-24T12:27:35.570086+0000","last_peered":"2024-03-24T12:27:35.570086+0000","last_clean":"2024-03-24T12:27:35.570086+0000","last_became_active":"2024-03-24T12:26:09.643602+0000","last_became_peered":"2024-03-24T12:26:09.643602+0000","last_unstale":"2024-03-24T12:27:35.570086+0000","last_undegraded":"2024-03-24T12:27:35.570086+0000","last_fullsized":"2024-03-24T12:27:35.570086+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":73,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.392046+0000","last_change":"2024-03-24T12:26:03.161676+0000","last_active":"2024-03-24T12:27:35.392046+0000","last_peered":"2024-03-24T12:27:35.392046+0000","last_clean":"2024-03-24T12:27:35.392046+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:27:35.392046+0000","last_undegraded":"2024-03-24T12:27:35.392046+0000","last_fullsized":"2024-03-24T12:27:35.392046+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":90,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569814+0000","last_change":"2024-03-24T12:27:17.288726+0000","last_active":"2024-03-24T12:27:35.569814+0000","last_peered":"2024-03-24T12:27:35.569814+0000","last_clean":"2024-03-24T12:27:35.569814+0000","last_became_active":"2024-03-24T12:26:05.358217+0000","last_became_peered":"2024-03-24T12:26:05.358217+0000","last_unstale":"2024-03-24T12:27:35.569814+0000","last_undegraded":"2024-03-24T12:27:35.569814+0000","last_fullsized":"2024-03-24T12:27:35.569814+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:27:17.288616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.288616+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:31.276817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":92,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391251+0000","last_change":"2024-03-24T12:27:20.354464+0000","last_active":"2024-03-24T12:27:35.391251+0000","last_peered":"2024-03-24T12:27:35.391251+0000","last_clean":"2024-03-24T12:27:35.391251+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:27:35.391251+0000","last_undegraded":"2024-03-24T12:27:35.391251+0000","last_fullsized":"2024-03-24T12:27:35.391251+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:20.354367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.354367+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.013310+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493952+0000","last_change":"2024-03-24T12:26:09.350769+0000","last_active":"2024-03-24T12:27:35.493952+0000","last_peered":"2024-03-24T12:27:35.493952+0000","last_clean":"2024-03-24T12:27:35.493952+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:27:35.493952+0000","last_undegraded":"2024-03-24T12:27:35.493952+0000","last_fullsized":"2024-03-24T12:27:35.493952+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:38.324576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391982+0000","last_change":"2024-03-24T12:26:03.162567+0000","last_active":"2024-03-24T12:27:35.391982+0000","last_peered":"2024-03-24T12:27:35.391982+0000","last_clean":"2024-03-24T12:27:35.391982+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:27:35.391982+0000","last_undegraded":"2024-03-24T12:27:35.391982+0000","last_fullsized":"2024-03-24T12:27:35.391982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":88,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870620+0000","last_change":"2024-03-24T12:27:23.131685+0000","last_active":"2024-03-24T12:27:34.870620+0000","last_peered":"2024-03-24T12:27:34.870620+0000","last_clean":"2024-03-24T12:27:34.870620+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:27:34.870620+0000","last_undegraded":"2024-03-24T12:27:34.870620+0000","last_fullsized":"2024-03-24T12:27:34.870620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:23.131573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.131573+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.851864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.581442+0000","last_change":"2024-03-24T12:27:35.581463+0000","last_active":"2024-03-24T12:27:35.581442+0000","last_peered":"2024-03-24T12:27:35.581442+0000","last_clean":"2024-03-24T12:27:35.581442+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:27:35.581442+0000","last_undegraded":"2024-03-24T12:27:35.581442+0000","last_fullsized":"2024-03-24T12:27:35.581442+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:35.581370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:35.581370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.191248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391391+0000","last_change":"2024-03-24T12:26:09.339988+0000","last_active":"2024-03-24T12:27:35.391391+0000","last_peered":"2024-03-24T12:27:35.391391+0000","last_clean":"2024-03-24T12:27:35.391391+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:27:35.391391+0000","last_undegraded":"2024-03-24T12:27:35.391391+0000","last_fullsized":"2024-03-24T12:27:35.391391+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.803044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870725+0000","last_change":"2024-03-24T12:27:32.548089+0000","last_active":"2024-03-24T12:27:34.870725+0000","last_peered":"2024-03-24T12:27:34.870725+0000","last_clean":"2024-03-24T12:27:34.870725+0000","last_became_active":"2024-03-24T12:26:07.331416+0000","last_became_peered":"2024-03-24T12:26:07.331416+0000","last_unstale":"2024-03-24T12:27:34.870725+0000","last_undegraded":"2024-03-24T12:27:34.870725+0000","last_fullsized":"2024-03-24T12:27:34.870725+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.547951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.547951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.882159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:37.366726+0000","last_change":"2024-03-24T12:27:37.366770+0000","last_active":"2024-03-24T12:27:37.366726+0000","last_peered":"2024-03-24T12:27:37.366726+0000","last_clean":"2024-03-24T12:27:37.366726+0000","last_became_active":"2024-03-24T12:26:03.165214+0000","last_became_peered":"2024-03-24T12:26:03.165214+0000","last_unstale":"2024-03-24T12:27:37.366726+0000","last_undegraded":"2024-03-24T12:27:37.366726+0000","last_fullsized":"2024-03-24T12:27:37.366726+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:37.366618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:37.366618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.039598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":96,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494684+0000","last_change":"2024-03-24T12:27:15.668713+0000","last_active":"2024-03-24T12:27:35.494684+0000","last_peered":"2024-03-24T12:27:35.494684+0000","last_clean":"2024-03-24T12:27:35.494684+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:27:35.494684+0000","last_undegraded":"2024-03-24T12:27:35.494684+0000","last_fullsized":"2024-03-24T12:27:35.494684+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:27:15.668556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:15.668556+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:43.755371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043720+0000","last_change":"2024-03-24T12:26:09.350031+0000","last_active":"2024-03-24T12:27:35.043720+0000","last_peered":"2024-03-24T12:27:35.043720+0000","last_clean":"2024-03-24T12:27:35.043720+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:27:35.043720+0000","last_undegraded":"2024-03-24T12:27:35.043720+0000","last_fullsized":"2024-03-24T12:27:35.043720+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:38.596198+0000","last_change":"2024-03-24T12:27:38.596223+0000","last_active":"2024-03-24T12:27:38.596198+0000","last_peered":"2024-03-24T12:27:38.596198+0000","last_clean":"2024-03-24T12:27:38.596198+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:27:38.596198+0000","last_undegraded":"2024-03-24T12:27:38.596198+0000","last_fullsized":"2024-03-24T12:27:38.596198+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.596133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:52.862429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570168+0000","last_change":"2024-03-24T12:27:09.234408+0000","last_active":"2024-03-24T12:27:35.570168+0000","last_peered":"2024-03-24T12:27:35.570168+0000","last_clean":"2024-03-24T12:27:35.570168+0000","last_became_active":"2024-03-24T12:26:03.162562+0000","last_became_peered":"2024-03-24T12:26:03.162562+0000","last_unstale":"2024-03-24T12:27:35.570168+0000","last_undegraded":"2024-03-24T12:27:35.570168+0000","last_fullsized":"2024-03-24T12:27:35.570168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:27:09.234299+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:17.263964+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044198+0000","last_change":"2024-03-24T12:26:16.882489+0000","last_active":"2024-03-24T12:27:35.044198+0000","last_peered":"2024-03-24T12:27:35.044198+0000","last_clean":"2024-03-24T12:27:35.044198+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:27:35.044198+0000","last_undegraded":"2024-03-24T12:27:35.044198+0000","last_fullsized":"2024-03-24T12:27:35.044198+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494257+0000","last_change":"2024-03-24T12:27:24.439386+0000","last_active":"2024-03-24T12:27:35.494257+0000","last_peered":"2024-03-24T12:27:35.494257+0000","last_clean":"2024-03-24T12:27:35.494257+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:27:35.494257+0000","last_undegraded":"2024-03-24T12:27:35.494257+0000","last_fullsized":"2024-03-24T12:27:35.494257+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.439296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.495934+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569976+0000","last_change":"2024-03-24T12:26:07.318867+0000","last_active":"2024-03-24T12:27:35.569976+0000","last_peered":"2024-03-24T12:27:35.569976+0000","last_clean":"2024-03-24T12:27:35.569976+0000","last_became_active":"2024-03-24T12:26:03.165771+0000","last_became_peered":"2024-03-24T12:26:03.165771+0000","last_unstale":"2024-03-24T12:27:35.569976+0000","last_undegraded":"2024-03-24T12:27:35.569976+0000","last_fullsized":"2024-03-24T12:27:35.569976+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870916+0000","last_change":"2024-03-24T12:27:32.227062+0000","last_active":"2024-03-24T12:27:34.870916+0000","last_peered":"2024-03-24T12:27:34.870916+0000","last_clean":"2024-03-24T12:27:34.870916+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:27:34.870916+0000","last_undegraded":"2024-03-24T12:27:34.870916+0000","last_fullsized":"2024-03-24T12:27:34.870916+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.226982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.226982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.516101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"90'38","reported_seq":143,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871376+0000","last_change":"2024-03-24T12:27:31.375798+0000","last_active":"2024-03-24T12:27:34.871376+0000","last_peered":"2024-03-24T12:27:34.871376+0000","last_clean":"2024-03-24T12:27:34.871376+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:27:34.871376+0000","last_undegraded":"2024-03-24T12:27:34.871376+0000","last_fullsized":"2024-03-24T12:27:34.871376+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'38","last_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:27:31.375629+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:55.069683+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044171+0000","last_change":"2024-03-24T12:26:07.332154+0000","last_active":"2024-03-24T12:27:35.044171+0000","last_peered":"2024-03-24T12:27:35.044171+0000","last_clean":"2024-03-24T12:27:35.044171+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:27:35.044171+0000","last_undegraded":"2024-03-24T12:27:35.044171+0000","last_fullsized":"2024-03-24T12:27:35.044171+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.392103+0000","last_change":"2024-03-24T12:27:18.336245+0000","last_active":"2024-03-24T12:27:35.392103+0000","last_peered":"2024-03-24T12:27:35.392103+0000","last_clean":"2024-03-24T12:27:35.392103+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:27:35.392103+0000","last_undegraded":"2024-03-24T12:27:35.392103+0000","last_fullsized":"2024-03-24T12:27:35.392103+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:18.336147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:18.336147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.167902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870392+0000","last_change":"2024-03-24T12:27:16.330188+0000","last_active":"2024-03-24T12:27:34.870392+0000","last_peered":"2024-03-24T12:27:34.870392+0000","last_clean":"2024-03-24T12:27:34.870392+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:27:34.870392+0000","last_undegraded":"2024-03-24T12:27:34.870392+0000","last_fullsized":"2024-03-24T12:27:34.870392+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:16.330050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.330050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.775200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870388+0000","last_change":"2024-03-24T12:27:24.614983+0000","last_active":"2024-03-24T12:27:34.870388+0000","last_peered":"2024-03-24T12:27:34.870388+0000","last_clean":"2024-03-24T12:27:34.870388+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:27:34.870388+0000","last_undegraded":"2024-03-24T12:27:34.870388+0000","last_fullsized":"2024-03-24T12:27:34.870388+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.614896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.614896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871644+0000","last_change":"2024-03-24T12:27:06.412196+0000","last_active":"2024-03-24T12:27:34.871644+0000","last_peered":"2024-03-24T12:27:34.871644+0000","last_clean":"2024-03-24T12:27:34.871644+0000","last_became_active":"2024-03-24T12:26:05.315639+0000","last_became_peered":"2024-03-24T12:26:05.315639+0000","last_unstale":"2024-03-24T12:27:34.871644+0000","last_undegraded":"2024-03-24T12:27:34.871644+0000","last_fullsized":"2024-03-24T12:27:34.871644+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:27:06.412074+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:27:06.412074+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.412074+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:27.288814+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871108+0000","last_change":"2024-03-24T12:27:19.658489+0000","last_active":"2024-03-24T12:27:34.871108+0000","last_peered":"2024-03-24T12:27:34.871108+0000","last_clean":"2024-03-24T12:27:34.871108+0000","last_became_active":"2024-03-24T12:26:07.343010+0000","last_became_peered":"2024-03-24T12:26:07.343010+0000","last_unstale":"2024-03-24T12:27:34.871108+0000","last_undegraded":"2024-03-24T12:27:34.871108+0000","last_fullsized":"2024-03-24T12:27:34.871108+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:19.658367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:19.658367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.033169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570399+0000","last_change":"2024-03-24T12:27:12.274502+0000","last_active":"2024-03-24T12:27:35.570399+0000","last_peered":"2024-03-24T12:27:35.570399+0000","last_clean":"2024-03-24T12:27:35.570399+0000","last_became_active":"2024-03-24T12:26:09.347821+0000","last_became_peered":"2024-03-24T12:26:09.347821+0000","last_unstale":"2024-03-24T12:27:35.570399+0000","last_undegraded":"2024-03-24T12:27:35.570399+0000","last_fullsized":"2024-03-24T12:27:35.570399+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.274393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.274393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.457433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":322,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391519+0000","last_change":"2024-03-24T12:27:08.342835+0000","last_active":"2024-03-24T12:27:35.391519+0000","last_peered":"2024-03-24T12:27:35.391519+0000","last_clean":"2024-03-24T12:27:35.391519+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:27:35.391519+0000","last_undegraded":"2024-03-24T12:27:35.391519+0000","last_fullsized":"2024-03-24T12:27:35.391519+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:27:08.342708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:27:08.342708+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:25.314000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493810+0000","last_change":"2024-03-24T12:27:28.436490+0000","last_active":"2024-03-24T12:27:35.493810+0000","last_peered":"2024-03-24T12:27:35.493810+0000","last_clean":"2024-03-24T12:27:35.493810+0000","last_became_active":"2024-03-24T12:26:03.166467+0000","last_became_peered":"2024-03-24T12:26:03.166467+0000","last_unstale":"2024-03-24T12:27:35.493810+0000","last_undegraded":"2024-03-24T12:27:35.493810+0000","last_fullsized":"2024-03-24T12:27:35.493810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.436402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.436402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.490307+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":107,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494871+0000","last_change":"2024-03-24T12:27:21.712724+0000","last_active":"2024-03-24T12:27:35.494871+0000","last_peered":"2024-03-24T12:27:35.494871+0000","last_clean":"2024-03-24T12:27:35.494871+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:27:35.494871+0000","last_undegraded":"2024-03-24T12:27:35.494871+0000","last_fullsized":"2024-03-24T12:27:35.494871+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:27:21.712602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.712602+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.700708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391063+0000","last_change":"2024-03-24T12:26:07.337813+0000","last_active":"2024-03-24T12:27:35.391063+0000","last_peered":"2024-03-24T12:27:35.391063+0000","last_clean":"2024-03-24T12:27:35.391063+0000","last_became_active":"2024-03-24T12:26:07.337548+0000","last_became_peered":"2024-03-24T12:26:07.337548+0000","last_unstale":"2024-03-24T12:27:35.391063+0000","last_undegraded":"2024-03-24T12:27:35.391063+0000","last_fullsized":"2024-03-24T12:27:35.391063+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870685+0000","last_change":"2024-03-24T12:27:25.409449+0000","last_active":"2024-03-24T12:27:34.870685+0000","last_peered":"2024-03-24T12:27:34.870685+0000","last_clean":"2024-03-24T12:27:34.870685+0000","last_became_active":"2024-03-24T12:26:09.349435+0000","last_became_peered":"2024-03-24T12:26:09.349435+0000","last_unstale":"2024-03-24T12:27:34.870685+0000","last_undegraded":"2024-03-24T12:27:34.870685+0000","last_fullsized":"2024-03-24T12:27:34.870685+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:25.409346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.409346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.841693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043877+0000","last_change":"2024-03-24T12:27:27.333417+0000","last_active":"2024-03-24T12:27:35.043877+0000","last_peered":"2024-03-24T12:27:35.043877+0000","last_clean":"2024-03-24T12:27:35.043877+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:27:35.043877+0000","last_undegraded":"2024-03-24T12:27:35.043877+0000","last_fullsized":"2024-03-24T12:27:35.043877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:27.333274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.333274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.779264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":171,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:40.525206+0000","last_change":"2024-03-24T12:27:16.322773+0000","last_active":"2024-03-24T12:27:40.525206+0000","last_peered":"2024-03-24T12:27:40.525206+0000","last_clean":"2024-03-24T12:27:40.525206+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:27:40.525206+0000","last_undegraded":"2024-03-24T12:27:40.525206+0000","last_fullsized":"2024-03-24T12:27:40.525206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:27:16.322655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.322655+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:22.799348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":92,"num_read_kb":87,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":83,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:38.328522+0000","last_change":"2024-03-24T12:27:38.328555+0000","last_active":"2024-03-24T12:27:38.328522+0000","last_peered":"2024-03-24T12:27:38.328522+0000","last_clean":"2024-03-24T12:27:38.328522+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:27:38.328522+0000","last_undegraded":"2024-03-24T12:27:38.328522+0000","last_fullsized":"2024-03-24T12:27:38.328522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:38.328458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.328458+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.105741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569584+0000","last_change":"2024-03-24T12:27:23.276910+0000","last_active":"2024-03-24T12:27:35.569584+0000","last_peered":"2024-03-24T12:27:35.569584+0000","last_clean":"2024-03-24T12:27:35.569584+0000","last_became_active":"2024-03-24T12:26:07.326457+0000","last_became_peered":"2024-03-24T12:26:07.326457+0000","last_unstale":"2024-03-24T12:27:35.569584+0000","last_undegraded":"2024-03-24T12:27:35.569584+0000","last_fullsized":"2024-03-24T12:27:35.569584+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.276789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.276789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.022163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:41.410555+0000","last_change":"2024-03-24T12:27:41.410577+0000","last_active":"2024-03-24T12:27:41.410555+0000","last_peered":"2024-03-24T12:27:41.410555+0000","last_clean":"2024-03-24T12:27:41.410555+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:27:41.410555+0000","last_undegraded":"2024-03-24T12:27:41.410555+0000","last_fullsized":"2024-03-24T12:27:41.410555+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:41.410495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:41.410495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.007651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871261+0000","last_change":"2024-03-24T12:27:32.343982+0000","last_active":"2024-03-24T12:27:34.871261+0000","last_peered":"2024-03-24T12:27:34.871261+0000","last_clean":"2024-03-24T12:27:34.871261+0000","last_became_active":"2024-03-24T12:26:03.167244+0000","last_became_peered":"2024-03-24T12:26:03.167244+0000","last_unstale":"2024-03-24T12:27:34.871261+0000","last_undegraded":"2024-03-24T12:27:34.871261+0000","last_fullsized":"2024-03-24T12:27:34.871261+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.343880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.343880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:53.358101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":93,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044244+0000","last_change":"2024-03-24T12:26:05.357149+0000","last_active":"2024-03-24T12:27:35.044244+0000","last_peered":"2024-03-24T12:27:35.044244+0000","last_clean":"2024-03-24T12:27:35.044244+0000","last_became_active":"2024-03-24T12:26:05.356745+0000","last_became_peered":"2024-03-24T12:26:05.356745+0000","last_unstale":"2024-03-24T12:27:35.044244+0000","last_undegraded":"2024-03-24T12:27:35.044244+0000","last_fullsized":"2024-03-24T12:27:35.044244+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494114+0000","last_change":"2024-03-24T12:27:23.763463+0000","last_active":"2024-03-24T12:27:35.494114+0000","last_peered":"2024-03-24T12:27:35.494114+0000","last_clean":"2024-03-24T12:27:35.494114+0000","last_became_active":"2024-03-24T12:26:07.337303+0000","last_became_peered":"2024-03-24T12:26:07.337303+0000","last_unstale":"2024-03-24T12:27:35.494114+0000","last_undegraded":"2024-03-24T12:27:35.494114+0000","last_fullsized":"2024-03-24T12:27:35.494114+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.763326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:36.340669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.043666+0000","last_change":"2024-03-24T12:26:09.341274+0000","last_active":"2024-03-24T12:27:35.043666+0000","last_peered":"2024-03-24T12:27:35.043666+0000","last_clean":"2024-03-24T12:27:35.043666+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:27:35.043666+0000","last_undegraded":"2024-03-24T12:27:35.043666+0000","last_fullsized":"2024-03-24T12:27:35.043666+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570395+0000","last_change":"2024-03-24T12:27:26.215446+0000","last_active":"2024-03-24T12:27:35.570395+0000","last_peered":"2024-03-24T12:27:35.570395+0000","last_clean":"2024-03-24T12:27:35.570395+0000","last_became_active":"2024-03-24T12:26:03.168184+0000","last_became_peered":"2024-03-24T12:26:03.168184+0000","last_unstale":"2024-03-24T12:27:35.570395+0000","last_undegraded":"2024-03-24T12:27:35.570395+0000","last_fullsized":"2024-03-24T12:27:35.570395+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:26.215348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.215348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.879847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871050+0000","last_change":"2024-03-24T12:27:24.375037+0000","last_active":"2024-03-24T12:27:34.871050+0000","last_peered":"2024-03-24T12:27:34.871050+0000","last_clean":"2024-03-24T12:27:34.871050+0000","last_became_active":"2024-03-24T12:26:05.316739+0000","last_became_peered":"2024-03-24T12:26:05.316739+0000","last_unstale":"2024-03-24T12:27:34.871050+0000","last_undegraded":"2024-03-24T12:27:34.871050+0000","last_fullsized":"2024-03-24T12:27:34.871050+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:24.374935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.374935+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:47.083756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":81,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044174+0000","last_change":"2024-03-24T12:26:07.336444+0000","last_active":"2024-03-24T12:27:35.044174+0000","last_peered":"2024-03-24T12:27:35.044174+0000","last_clean":"2024-03-24T12:27:35.044174+0000","last_became_active":"2024-03-24T12:26:07.336080+0000","last_became_peered":"2024-03-24T12:26:07.336080+0000","last_unstale":"2024-03-24T12:27:35.044174+0000","last_undegraded":"2024-03-24T12:27:35.044174+0000","last_fullsized":"2024-03-24T12:27:35.044174+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.391746+0000","last_change":"2024-03-24T12:27:33.429270+0000","last_active":"2024-03-24T12:27:35.391746+0000","last_peered":"2024-03-24T12:27:35.391746+0000","last_clean":"2024-03-24T12:27:35.391746+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:27:35.391746+0000","last_undegraded":"2024-03-24T12:27:35.391746+0000","last_fullsized":"2024-03-24T12:27:35.391746+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:33.429173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:33.429173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.664982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":70,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.044198+0000","last_change":"2024-03-24T12:27:12.294371+0000","last_active":"2024-03-24T12:27:35.044198+0000","last_peered":"2024-03-24T12:27:35.044198+0000","last_clean":"2024-03-24T12:27:35.044198+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:27:35.044198+0000","last_undegraded":"2024-03-24T12:27:35.044198+0000","last_fullsized":"2024-03-24T12:27:35.044198+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:12.294292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.294292+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.602110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870646+0000","last_change":"2024-03-24T12:26:05.315771+0000","last_active":"2024-03-24T12:27:34.870646+0000","last_peered":"2024-03-24T12:27:34.870646+0000","last_clean":"2024-03-24T12:27:34.870646+0000","last_became_active":"2024-03-24T12:26:05.315454+0000","last_became_peered":"2024-03-24T12:26:05.315454+0000","last_unstale":"2024-03-24T12:27:34.870646+0000","last_undegraded":"2024-03-24T12:27:34.870646+0000","last_fullsized":"2024-03-24T12:27:34.870646+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.492894+0000","last_change":"2024-03-24T12:27:10.466243+0000","last_active":"2024-03-24T12:27:35.492894+0000","last_peered":"2024-03-24T12:27:35.492894+0000","last_clean":"2024-03-24T12:27:35.492894+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:27:35.492894+0000","last_undegraded":"2024-03-24T12:27:35.492894+0000","last_fullsized":"2024-03-24T12:27:35.492894+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:10.466147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.466147+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:12.236572+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870529+0000","last_change":"2024-03-24T12:26:09.352908+0000","last_active":"2024-03-24T12:27:34.870529+0000","last_peered":"2024-03-24T12:27:34.870529+0000","last_clean":"2024-03-24T12:27:34.870529+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:27:34.870529+0000","last_undegraded":"2024-03-24T12:27:34.870529+0000","last_fullsized":"2024-03-24T12:27:34.870529+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.176448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:36.482534+0000","last_change":"2024-03-24T12:27:36.482571+0000","last_active":"2024-03-24T12:27:36.482534+0000","last_peered":"2024-03-24T12:27:36.482534+0000","last_clean":"2024-03-24T12:27:36.482534+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:27:36.482534+0000","last_undegraded":"2024-03-24T12:27:36.482534+0000","last_fullsized":"2024-03-24T12:27:36.482534+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:36.482466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:36.482466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:03.061842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":97,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493506+0000","last_change":"2024-03-24T12:27:11.454130+0000","last_active":"2024-03-24T12:27:35.493506+0000","last_peered":"2024-03-24T12:27:35.493506+0000","last_clean":"2024-03-24T12:27:35.493506+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:27:35.493506+0000","last_undegraded":"2024-03-24T12:27:35.493506+0000","last_fullsized":"2024-03-24T12:27:35.493506+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T12:27:11.453973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:11.453973+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:18.945222+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":53,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870349+0000","last_change":"2024-03-24T12:26:07.336955+0000","last_active":"2024-03-24T12:27:34.870349+0000","last_peered":"2024-03-24T12:27:34.870349+0000","last_clean":"2024-03-24T12:27:34.870349+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:27:34.870349+0000","last_undegraded":"2024-03-24T12:27:34.870349+0000","last_fullsized":"2024-03-24T12:27:34.870349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:35.459896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.570232+0000","last_change":"2024-03-24T12:26:09.351354+0000","last_active":"2024-03-24T12:27:35.570232+0000","last_peered":"2024-03-24T12:27:35.570232+0000","last_clean":"2024-03-24T12:27:35.570232+0000","last_became_active":"2024-03-24T12:26:09.351061+0000","last_became_peered":"2024-03-24T12:26:09.351061+0000","last_unstale":"2024-03-24T12:27:35.570232+0000","last_undegraded":"2024-03-24T12:27:35.570232+0000","last_fullsized":"2024-03-24T12:27:35.570232+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871496+0000","last_change":"2024-03-24T12:27:20.101666+0000","last_active":"2024-03-24T12:27:34.871496+0000","last_peered":"2024-03-24T12:27:34.871496+0000","last_clean":"2024-03-24T12:27:34.871496+0000","last_became_active":"2024-03-24T12:26:07.328595+0000","last_became_peered":"2024-03-24T12:26:07.328595+0000","last_unstale":"2024-03-24T12:27:34.871496+0000","last_undegraded":"2024-03-24T12:27:34.871496+0000","last_fullsized":"2024-03-24T12:27:34.871496+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:20.101554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.101554+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:34.528554+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.493461+0000","last_change":"2024-03-24T12:26:03.163624+0000","last_active":"2024-03-24T12:27:35.493461+0000","last_peered":"2024-03-24T12:27:35.493461+0000","last_clean":"2024-03-24T12:27:35.493461+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:27:35.493461+0000","last_undegraded":"2024-03-24T12:27:35.493461+0000","last_fullsized":"2024-03-24T12:27:35.493461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871458+0000","last_change":"2024-03-24T12:26:16.885041+0000","last_active":"2024-03-24T12:27:34.871458+0000","last_peered":"2024-03-24T12:27:34.871458+0000","last_clean":"2024-03-24T12:27:34.871458+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:27:34.871458+0000","last_undegraded":"2024-03-24T12:27:34.871458+0000","last_fullsized":"2024-03-24T12:27:34.871458+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:40.439771+0000","last_change":"2024-03-24T12:27:40.439811+0000","last_active":"2024-03-24T12:27:40.439771+0000","last_peered":"2024-03-24T12:27:40.439771+0000","last_clean":"2024-03-24T12:27:40.439771+0000","last_became_active":"2024-03-24T12:26:09.643739+0000","last_became_peered":"2024-03-24T12:26:09.643739+0000","last_unstale":"2024-03-24T12:27:40.439771+0000","last_undegraded":"2024-03-24T12:27:40.439771+0000","last_fullsized":"2024-03-24T12:27:40.439771+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:40.439695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:40.439695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:04.958905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.569735+0000","last_change":"2024-03-24T12:27:08.227925+0000","last_active":"2024-03-24T12:27:35.569735+0000","last_peered":"2024-03-24T12:27:35.569735+0000","last_clean":"2024-03-24T12:27:35.569735+0000","last_became_active":"2024-03-24T12:26:07.332766+0000","last_became_peered":"2024-03-24T12:26:07.332766+0000","last_unstale":"2024-03-24T12:27:35.569735+0000","last_undegraded":"2024-03-24T12:27:35.569735+0000","last_fullsized":"2024-03-24T12:27:35.569735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:08.227812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:08.227812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:14.184705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870384+0000","last_change":"2024-03-24T12:27:31.177660+0000","last_active":"2024-03-24T12:27:34.870384+0000","last_peered":"2024-03-24T12:27:34.870384+0000","last_clean":"2024-03-24T12:27:34.870384+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:27:34.870384+0000","last_undegraded":"2024-03-24T12:27:34.870384+0000","last_fullsized":"2024-03-24T12:27:34.870384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:31.177529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:31.177529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.777930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871082+0000","last_change":"2024-03-24T12:26:05.315880+0000","last_active":"2024-03-24T12:27:34.871082+0000","last_peered":"2024-03-24T12:27:34.871082+0000","last_clean":"2024-03-24T12:27:34.871082+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:27:34.871082+0000","last_undegraded":"2024-03-24T12:27:34.871082+0000","last_fullsized":"2024-03-24T12:27:34.871082+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":49,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871619+0000","last_change":"2024-03-24T12:26:09.348104+0000","last_active":"2024-03-24T12:27:34.871619+0000","last_peered":"2024-03-24T12:27:34.871619+0000","last_clean":"2024-03-24T12:27:34.871619+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:27:34.871619+0000","last_undegraded":"2024-03-24T12:27:34.871619+0000","last_fullsized":"2024-03-24T12:27:34.871619+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:27:36.236528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871483+0000","last_change":"2024-03-24T12:27:23.411568+0000","last_active":"2024-03-24T12:27:34.871483+0000","last_peered":"2024-03-24T12:27:34.871483+0000","last_clean":"2024-03-24T12:27:34.871483+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:27:34.871483+0000","last_undegraded":"2024-03-24T12:27:34.871483+0000","last_fullsized":"2024-03-24T12:27:34.871483+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.411477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.411477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.665294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":69,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494588+0000","last_change":"2024-03-24T12:27:12.636522+0000","last_active":"2024-03-24T12:27:35.494588+0000","last_peered":"2024-03-24T12:27:35.494588+0000","last_clean":"2024-03-24T12:27:35.494588+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:27:35.494588+0000","last_undegraded":"2024-03-24T12:27:35.494588+0000","last_fullsized":"2024-03-24T12:27:35.494588+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.636423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.636423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.538773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":85,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871126+0000","last_change":"2024-03-24T12:26:05.316692+0000","last_active":"2024-03-24T12:27:34.871126+0000","last_peered":"2024-03-24T12:27:34.871126+0000","last_clean":"2024-03-24T12:27:34.871126+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:27:34.871126+0000","last_undegraded":"2024-03-24T12:27:34.871126+0000","last_fullsized":"2024-03-24T12:27:34.871126+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":57,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871534+0000","last_change":"2024-03-24T12:27:10.145724+0000","last_active":"2024-03-24T12:27:34.871534+0000","last_peered":"2024-03-24T12:27:34.871534+0000","last_clean":"2024-03-24T12:27:34.871534+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:27:34.871534+0000","last_undegraded":"2024-03-24T12:27:34.871534+0000","last_fullsized":"2024-03-24T12:27:34.871534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:10.145613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.145613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:22.594263+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":89,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.871322+0000","last_change":"2024-03-24T12:27:21.649282+0000","last_active":"2024-03-24T12:27:34.871322+0000","last_peered":"2024-03-24T12:27:34.871322+0000","last_clean":"2024-03-24T12:27:34.871322+0000","last_became_active":"2024-03-24T12:26:07.334798+0000","last_became_peered":"2024-03-24T12:26:07.334798+0000","last_unstale":"2024-03-24T12:27:34.871322+0000","last_undegraded":"2024-03-24T12:27:34.871322+0000","last_fullsized":"2024-03-24T12:27:34.871322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:21.649181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.649181+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.389315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":61,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:35.494051+0000","last_change":"2024-03-24T12:26:03.162618+0000","last_active":"2024-03-24T12:27:35.494051+0000","last_peered":"2024-03-24T12:27:35.494051+0000","last_clean":"2024-03-24T12:27:35.494051+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:27:35.494051+0000","last_undegraded":"2024-03-24T12:27:35.494051+0000","last_fullsized":"2024-03-24T12:27:35.494051+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":80,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-24T12:27:34.870710+0000","last_change":"2024-03-24T12:27:22.098052+0000","last_active":"2024-03-24T12:27:34.870710+0000","last_peered":"2024-03-24T12:27:34.870710+0000","last_clean":"2024-03-24T12:27:34.870710+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:27:34.870710+0000","last_undegraded":"2024-03-24T12:27:34.870710+0000","last_fullsized":"2024-03-24T12:27:34.870710+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:22.097943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:22.097943+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.541070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":424,"ondisk_log_size":424,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":87,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037784,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28436,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716716,"statfs":{"total":95995035648,"available":95965917184,"internally_reserved":0,"allocated":1454080,"data_stored":964966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332126,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32572,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712580,"statfs":{"total":95995035648,"available":95961681920,"internally_reserved":0,"allocated":1429504,"data_stored":964378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":374988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32596,"kb_used_data":1420,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712556,"statfs":{"total":95995035648,"available":95961657344,"internally_reserved":0,"allocated":1454080,"data_stored":964310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411379,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31992,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713160,"statfs":{"total":95995035648,"available":95962275840,"internally_reserved":0,"allocated":835584,"data_stored":373470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117036,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27872,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717280,"statfs":{"total":95995035648,"available":95966494720,"internally_reserved":0,"allocated":876544,"data_stored":374914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920808,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27856,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717296,"statfs":{"total":95995035648,"available":95966511104,"internally_reserved":0,"allocated":860160,"data_stored":373740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626467,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27844,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717308,"statfs":{"total":95995035648,"available":95966523392,"internally_reserved":0,"allocated":847872,"data_stored":374844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:27:48.055 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T12:27:48.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump 2024-03-24T12:27:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:47 smithi190 ceph-mon[33748]: 5.15 scrub starts 2024-03-24T12:27:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:47 smithi190 ceph-mon[33748]: 5.15 scrub ok 2024-03-24T12:27:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:47 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:27:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:47 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:27:48.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:47 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:48.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-mon[29919]: 5.15 scrub starts 2024-03-24T12:27:48.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-mon[29919]: 5.15 scrub ok 2024-03-24T12:27:48.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:27:48.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:27:48.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:48.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-mon[41500]: 5.15 scrub starts 2024-03-24T12:27:48.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-mon[41500]: 5.15 scrub ok 2024-03-24T12:27:48.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:27:48.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:27:48.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:47 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:48.937 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:49.247 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:48 smithi029 ceph-mon[41500]: 5.16 scrub starts 2024-03-24T12:27:49.247 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:48 smithi029 ceph-mon[41500]: 5.16 scrub ok 2024-03-24T12:27:49.247 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:48 smithi029 ceph-mon[41500]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:49.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:48 smithi029 ceph-mon[29919]: 5.16 scrub starts 2024-03-24T12:27:49.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:48 smithi029 ceph-mon[29919]: 5.16 scrub ok 2024-03-24T12:27:49.247 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:48 smithi029 ceph-mon[29919]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:49.248 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:49 smithi029 bash[81145]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T12:27:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:48 smithi190 ceph-mon[33748]: 5.16 scrub starts 2024-03-24T12:27:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:48 smithi190 ceph-mon[33748]: 5.16 scrub ok 2024-03-24T12:27:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:48 smithi190 ceph-mon[33748]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:49.669 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:49 smithi029 bash[81145]: Writing manifest to image destination 2024-03-24T12:27:50.207 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:49 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:27:50.207 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:49 smithi029 ceph-mon[29919]: 3.13 scrub ok 2024-03-24T12:27:50.207 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:49 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:27:50.207 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:49 smithi029 ceph-mon[41500]: 3.13 scrub ok 2024-03-24T12:27:50.207 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 podman[81145]: 2024-03-24 12:27:49.976438073 +0000 UTC m=+3.741472656 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T12:27:50.207 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 podman[81145]: 2024-03-24 12:27:50.210049253 +0000 UTC m=+3.975083829 container create e5bbcb3d32819411e67d5d0f52ebb1244199e1c7dcd8f0c1fee6b342ca4f715f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:27:50.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:49 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:27:50.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:49 smithi190 ceph-mon[33748]: 3.13 scrub ok 2024-03-24T12:27:51.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-mon[29919]: 3.b scrub starts 2024-03-24T12:27:51.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-mon[29919]: 3.b scrub ok 2024-03-24T12:27:51.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-mon[29919]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:51.087 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-mon[41500]: 3.b scrub starts 2024-03-24T12:27:51.087 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-mon[41500]: 3.b scrub ok 2024-03-24T12:27:51.087 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-mon[41500]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:51.088 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 podman[81145]: 2024-03-24 12:27:50.987700507 +0000 UTC m=+4.752735098 container init e5bbcb3d32819411e67d5d0f52ebb1244199e1c7dcd8f0c1fee6b342ca4f715f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:27:51.088 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 podman[81145]: 2024-03-24 12:27:50.993981617 +0000 UTC m=+4.759016194 container start e5bbcb3d32819411e67d5d0f52ebb1244199e1c7dcd8f0c1fee6b342ca4f715f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:27:51.088 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.998Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T12:27:51.088 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.998Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T12:27:51.088 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.998Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.998Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T12:27:51.089 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T12:27:51.090 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T12:27:51.091 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:50 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:50.999Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:51.000Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a[81330]: ts=2024-03-24T12:27:51.000Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:51 smithi029 bash[81145]: e5bbcb3d32819411e67d5d0f52ebb1244199e1c7dcd8f0c1fee6b342ca4f715f 2024-03-24T12:27:51.092 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:27:51 smithi029 systemd[1]: Started Ceph node-exporter.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:27:51.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:50 smithi190 ceph-mon[33748]: 3.b scrub starts 2024-03-24T12:27:51.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:50 smithi190 ceph-mon[33748]: 3.b scrub ok 2024-03-24T12:27:51.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:50 smithi190 ceph-mon[33748]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 232 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 93 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:fsid bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:created 2024-03-24T12:20:18.712998+0000 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:modified 2024-03-24T12:27:44.953047+0000 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:crush_version 18 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:full_ratio 0.95 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:backfillfull_ratio 0.9 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:nearfull_ratio 0.85 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:require_min_compat_client luminous 2024-03-24T12:27:51.891 INFO:teuthology.orchestra.run.smithi029.stdout:min_compat_client luminous 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:require_osd_release squid 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:stretch_mode_enabled false 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 71 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:max_osd 8 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0 up in weight 1 up_from 8 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.29:6801/2514818464 v2:172.21.15.29:6802/2514818464 exists,up 0aa1c49b-7887-4f74-b03c-08cc0adec90b 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.29:6805/2923396572 v2:172.21.15.29:6806/2923396572 exists,up 99d8ef8f-eb59-4882-ab51-b398e6b53bcc 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2 up in weight 1 up_from 18 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.29:6809/126032915 v2:172.21.15.29:6810/126032915 exists,up c3f83505-25ee-4217-b60a-081c36092e92 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3 up in weight 1 up_from 27 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.29:6813/1080171824 v2:172.21.15.29:6814/1080171824 exists,up f5e479a8-459a-4315-a808-711a7b84bb96 2024-03-24T12:27:51.892 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4 up in weight 1 up_from 33 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6800/3690258869 v2:172.21.15.190:6801/3690258869 exists,up 5e420304-2798-4d11-9688-d0d89482008c 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5 up in weight 1 up_from 42 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6804/1327220488 v2:172.21.15.190:6805/1327220488 exists,up 1213431a-652c-4cbf-b32a-36335430c666 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6 up in weight 1 up_from 51 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6808/1295064976 v2:172.21.15.190:6809/1295064976 exists,up 0712e511-0257-44f3-a82f-ee09ab957c0c 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7 up in weight 1 up_from 60 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6812/110963949 v2:172.21.15.190:6813/110963949 exists,up 7e36bde4-14d0-4e7e-9a82-2bb97859dbda 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 4.b [1,2] 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 4.f [1,2] 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:pg_upmap_items 4.12 [1,5] 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2633022206 expires 2024-03-25T12:26:46.364982+0000 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/2612078930 expires 2024-03-25T12:26:46.364982+0000 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2365028933 expires 2024-03-25T12:26:46.364982+0000 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/4113609697 expires 2024-03-25T12:26:46.364982+0000 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/4130171883 expires 2024-03-25T12:21:15.151901+0000 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/3697529912 expires 2024-03-25T12:26:46.364982+0000 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/2929430696 expires 2024-03-25T12:21:15.151901+0000 2024-03-24T12:27:51.893 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/327626505 expires 2024-03-25T12:20:46.948976+0000 2024-03-24T12:27:51.894 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/3715448582 expires 2024-03-25T12:20:46.948976+0000 2024-03-24T12:27:51.894 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/861146182 expires 2024-03-25T12:26:46.364982+0000 2024-03-24T12:27:51.894 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/2109454792 expires 2024-03-25T12:20:46.948976+0000 2024-03-24T12:27:51.894 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1631806715 expires 2024-03-25T12:20:46.948976+0000 2024-03-24T12:27:51.894 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:6800/852212806 expires 2024-03-25T12:21:15.151901+0000 2024-03-24T12:27:51.894 INFO:teuthology.orchestra.run.smithi029.stdout:blocklist 172.21.15.29:0/1823252077 expires 2024-03-25T12:21:15.151901+0000 2024-03-24T12:27:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[41500]: 4.17 scrub starts 2024-03-24T12:27:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[41500]: 4.17 scrub ok 2024-03-24T12:27:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[41500]: Deploying daemon node-exporter.b on smithi190 2024-03-24T12:27:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[41500]: 3.e scrub starts 2024-03-24T12:27:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[41500]: 3.e scrub ok 2024-03-24T12:27:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3818732407' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T12:27:52.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[29919]: 4.17 scrub starts 2024-03-24T12:27:52.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[29919]: 4.17 scrub ok 2024-03-24T12:27:52.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:52.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:52.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:52.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[29919]: Deploying daemon node-exporter.b on smithi190 2024-03-24T12:27:52.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[29919]: 3.e scrub starts 2024-03-24T12:27:52.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[29919]: 3.e scrub ok 2024-03-24T12:27:52.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:51 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3818732407' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T12:27:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:51 smithi190 ceph-mon[33748]: 4.17 scrub starts 2024-03-24T12:27:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:51 smithi190 ceph-mon[33748]: 4.17 scrub ok 2024-03-24T12:27:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:27:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:51 smithi190 ceph-mon[33748]: Deploying daemon node-exporter.b on smithi190 2024-03-24T12:27:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:51 smithi190 ceph-mon[33748]: 3.e scrub starts 2024-03-24T12:27:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:51 smithi190 ceph-mon[33748]: 3.e scrub ok 2024-03-24T12:27:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:51 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3818732407' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T12:27:52.885 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.29:6801/2514818464 v2:172.21.15.29:6802/2514818464 exists,up 0aa1c49b-7887-4f74-b03c-08cc0adec90b', 'osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.29:6805/2923396572 v2:172.21.15.29:6806/2923396572 exists,up 99d8ef8f-eb59-4882-ab51-b398e6b53bcc', 'osd.2 up in weight 1 up_from 18 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.29:6809/126032915 v2:172.21.15.29:6810/126032915 exists,up c3f83505-25ee-4217-b60a-081c36092e92', 'osd.3 up in weight 1 up_from 27 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.29:6813/1080171824 v2:172.21.15.29:6814/1080171824 exists,up f5e479a8-459a-4315-a808-711a7b84bb96', 'osd.4 up in weight 1 up_from 33 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6800/3690258869 v2:172.21.15.190:6801/3690258869 exists,up 5e420304-2798-4d11-9688-d0d89482008c', 'osd.5 up in weight 1 up_from 42 up_thru 76 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6804/1327220488 v2:172.21.15.190:6805/1327220488 exists,up 1213431a-652c-4cbf-b32a-36335430c666', 'osd.6 up in weight 1 up_from 51 up_thru 70 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6808/1295064976 v2:172.21.15.190:6809/1295064976 exists,up 0712e511-0257-44f3-a82f-ee09ab957c0c', 'osd.7 up in weight 1 up_from 60 up_thru 72 down_at 0 last_clean_interval [0,0) v2:172.21.15.190:6812/110963949 v2:172.21.15.190:6813/110963949 exists,up 7e36bde4-14d0-4e7e-9a82-2bb97859dbda'] 2024-03-24T12:27:52.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon mon.a config show 2024-03-24T12:27:52.886 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:52 smithi190 bash[66583]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-24T12:27:53.142 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:53.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:27:53.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:27:53.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:27:53.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:27:53.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[41500]: 5.14 deep-scrub starts 2024-03-24T12:27:53.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[41500]: 5.14 deep-scrub ok 2024-03-24T12:27:53.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[41500]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:53.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:27:53.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:27:53.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:27:53.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:27:53.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[29919]: 5.14 deep-scrub starts 2024-03-24T12:27:53.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[29919]: 5.14 deep-scrub ok 2024-03-24T12:27:53.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:52 smithi029 ceph-mon[29919]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:52 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:27:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:52 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:27:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:52 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:27:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:52 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:27:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:52 smithi190 ceph-mon[33748]: 5.14 deep-scrub starts 2024-03-24T12:27:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:52 smithi190 ceph-mon[33748]: 5.14 deep-scrub ok 2024-03-24T12:27:53.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:52 smithi190 ceph-mon[33748]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:53.385 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:53 smithi190 bash[66583]: Getting image source signatures 2024-03-24T12:27:53.385 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:53 smithi190 bash[66583]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-24T12:27:53.385 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:53 smithi190 bash[66583]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-24T12:27:53.385 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:53 smithi190 bash[66583]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-24T12:27:54.275 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[41500]: 6.a scrub starts 2024-03-24T12:27:54.275 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[41500]: 6.a scrub ok 2024-03-24T12:27:54.275 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[41500]: 6.1a scrub starts 2024-03-24T12:27:54.275 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[41500]: 6.1a scrub ok 2024-03-24T12:27:54.275 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[41500]: 6.b scrub starts 2024-03-24T12:27:54.275 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[41500]: 6.b scrub ok 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[41500]: 6.10 deep-scrub starts 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[41500]: 6.10 deep-scrub ok 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[29919]: 6.a scrub starts 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[29919]: 6.a scrub ok 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[29919]: 6.1a scrub starts 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[29919]: 6.1a scrub ok 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[29919]: 6.b scrub starts 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[29919]: 6.b scrub ok 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[29919]: 6.10 deep-scrub starts 2024-03-24T12:27:54.276 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[29919]: 6.10 deep-scrub ok 2024-03-24T12:27:54.277 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:27:54.277 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:53 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:27:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:53 smithi190 ceph-mon[33748]: 6.a scrub starts 2024-03-24T12:27:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:53 smithi190 ceph-mon[33748]: 6.a scrub ok 2024-03-24T12:27:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:53 smithi190 ceph-mon[33748]: 6.1a scrub starts 2024-03-24T12:27:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:53 smithi190 ceph-mon[33748]: 6.1a scrub ok 2024-03-24T12:27:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:53 smithi190 ceph-mon[33748]: 6.b scrub starts 2024-03-24T12:27:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:53 smithi190 ceph-mon[33748]: 6.b scrub ok 2024-03-24T12:27:54.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:53 smithi190 ceph-mon[33748]: 6.10 deep-scrub starts 2024-03-24T12:27:54.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:53 smithi190 ceph-mon[33748]: 6.10 deep-scrub ok 2024-03-24T12:27:54.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:53 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:27:54.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:53 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:27:54.895 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:27:54.895 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "mon.a", 2024-03-24T12:27:54.895 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster": "ceph", 2024-03-24T12:27:54.895 INFO:teuthology.orchestra.run.smithi029.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-24T12:27:54.895 INFO:teuthology.orchestra.run.smithi029.stdout: "admin_socket_mode": "", 2024-03-24T12:27:54.895 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-24T12:27:54.895 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_client_required": "cephx, none", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_cluster_required": "cephx", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_debug": "true", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_service_required": "cephx", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_supported": "", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_aio": "true", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_aio_poll_ms": "250", 2024-03-24T12:27:54.896 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_aio_reap_max": "16", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_async_discard": "false", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_block_size": "4096", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_debug_aio": "false", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_debug_inflight_ios": "false", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_enable_discard": "false", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_flock_retry": "3", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_inject_crash": "0", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-24T12:27:54.897 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_ioring": "false", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_ioring_hipri": "false", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bdev_type": "", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_alloc_size": "1048576", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_allocator": "hybrid", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_buffered_io": "true", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_check_for_zeros": "false", 2024-03-24T12:27:54.898 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_compact_log_sync": "false", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_max_log_runway": "4194304", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_max_prefetch": "1048576", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_min_flush_size": "524288", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_min_log_runway": "1048576", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_replay_recovery": "false", 2024-03-24T12:27:54.899 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluefs_sync_write": "false", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_allocation_from_file": "true", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_allocator": "hybrid", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-24T12:27:54.900 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_create": "true", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_db_create": "false", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_db_path": "", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_db_size": "0", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_path": "", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_preallocate_file": "false", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_size": "107374182400", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_wal_create": "false", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_wal_path": "", 2024-03-24T12:27:54.901 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_block_wal_size": "100663296", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bluefs": "true", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_autotune": "true", 2024-03-24T12:27:54.902 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_size": "0", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cache_type": "2q", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_clone_cow": "true", 2024-03-24T12:27:54.903 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_mode": "none", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_csum_type": "crc32c", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-24T12:27:54.904 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_freelist": "false", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-24T12:27:54.905 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_small_allocations": "0", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_default_buffered_read": "true", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_default_buffered_write": "false", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-24T12:27:54.906 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fail_eio": "false", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_mount": "false", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_umount": "false", 2024-03-24T12:27:54.907 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_ignore_data_csum": "false", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-24T12:27:54.908 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_log_op_age": "5.000000", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_alloc_size": "0", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_blob_size": "0", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_max_deferred_txc": "32", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_min_alloc_size": "0", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_nid_prealloc": "1024", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-24T12:27:54.909 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_retry_disk_reads": "3", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_rocksdb_cf": "true", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-24T12:27:54.910 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_spdk_mem": "512", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_tracing": "false", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-24T12:27:54.911 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "bluestore_zero_block_detection": "false", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-24T12:27:54.912 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_cluster_require_signatures": "false", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_cluster_require_version": "2", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_require_signatures": "false", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_require_version": "2", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_service_require_signatures": "false", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_service_require_version": "2", 2024-03-24T12:27:54.913 INFO:teuthology.orchestra.run.smithi029.stdout: "cephx_sign_messages": "true", 2024-03-24T12:27:54.914 INFO:teuthology.orchestra.run.smithi029.stdout: "chdir": "", 2024-03-24T12:27:54.914 INFO:teuthology.orchestra.run.smithi029.stdout: "client_acl_type": "", 2024-03-24T12:27:54.914 INFO:teuthology.orchestra.run.smithi029.stdout: "client_asio_thread_count": "2", 2024-03-24T12:27:54.914 INFO:teuthology.orchestra.run.smithi029.stdout: "client_cache_mid": "0.750000", 2024-03-24T12:27:54.920 INFO:teuthology.orchestra.run.smithi029.stdout: "client_cache_size": "16384", 2024-03-24T12:27:54.920 INFO:teuthology.orchestra.run.smithi029.stdout: "client_caps_release_delay": "5", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_check_pool_perm": "true", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_debug_force_sync_read": "false", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_debug_getattr_caps": "false", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_debug_inject_tick_delay": "0", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_die_on_failed_remount": "false", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_dirsize_rbytes": "true", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_force_lazyio": "false", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_fs": "", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_inject_release_failure": "false", 2024-03-24T12:27:54.921 INFO:teuthology.orchestra.run.smithi029.stdout: "client_max_inline_size": "4096", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_mds_namespace": "", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_metadata": "", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_mount_gid": "-1", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_mount_timeout": "300", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_mount_uid": "-1", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_mountpoint": "/", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_notify_timeout": "10", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc": "true", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc_max_dirty": "104857600", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc_max_objects": "1000", 2024-03-24T12:27:54.922 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc_size": "209715200", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_oc_target_dirty": "8388608", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_permissions": "true", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_quota": "true", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_quota_df": "true", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_readahead_max_bytes": "0", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_readahead_max_periods": "4", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_readahead_min": "131072", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_reconnect_stale": "false", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_shutdown_timeout": "30", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_snapdir": ".snap", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_tick_interval": "1", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_trace": "", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_try_dentry_invalidate": "false", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_use_faked_inos": "false", 2024-03-24T12:27:54.923 INFO:teuthology.orchestra.run.smithi029.stdout: "client_use_random_mds": "false", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_graylog": "false", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_graylog_port": "12201", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_monitors": "default=true", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_syslog": "false", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "clog_to_syslog_level": "info", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "-", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_network": "", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_network_interface": "", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "compressor_zlib_isal": "false", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "compressor_zlib_level": "5", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "compressor_zlib_winsize": "-15", 2024-03-24T12:27:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: "compressor_zstd_level": "1", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_alien_op_num_threads": "6", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "crimson_seastar_cpu_cores": "", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_location": "", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_location_hook": "", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_location_hook_timeout": "10", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "daemonize": "false", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "daos_pool": "tank", 2024-03-24T12:27:54.925 INFO:teuthology.orchestra.run.smithi029.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_alienstore": "0/5", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_allow_any_pool_priority": "false", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_asok": "1/5", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_asok_assert_abort": "false", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_asserts_on_shutdown": "true", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_auth": "1/5", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_bdev": "1/3", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_bluefs": "1/5", 2024-03-24T12:27:54.926 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_bluestore": "1/5", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_buffer": "0/1", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_ceph_exporter": "1/5", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_cephfs_mirror": "0/5", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_cephsqlite": "0/5", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_client": "0/5", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_compressor": "1/5", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_context": "0/1", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_crimson_interrupt": "0/5", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_crush": "1/1", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_crypto": "1/5", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_cyanstore": "0/5", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_deliberately_leak_memory": "false", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_disable_randomized_ping": "false", 2024-03-24T12:27:54.927 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_dpdk": "1/5", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_eventtrace": "1/5", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_filer": "0/1", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_filestore": "1/3", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_finisher": "1/1", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_fuse": "1/5", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_heartbeat_testing_span": "0", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_heartbeatmap": "1/5", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_javaclient": "1/5", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_journal": "1/3", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_journaler": "0/5", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_kstore": "1/5", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_lockdep": "0/1", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mclock": "1/5", 2024-03-24T12:27:54.928 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds": "1/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_balancer": "1/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_locker": "1/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_log": "1/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_log_expire": "1/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mds_migrator": "1/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_memstore": "1/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mgr": "2/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mgrc": "1/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_mon": "20/20", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_monc": "0/10", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_ms": "1/1", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_none": "0/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_objclass": "0/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_objectcacher": "0/5", 2024-03-24T12:27:54.929 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_objecter": "0/1", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_optracker": "0/5", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_osd": "1/5", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_paxos": "20/20", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_perfcounter": "1/5", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_prioritycache": "1/5", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rados": "0/5", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rbd": "0/5", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rbd_mirror": "0/5", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rbd_pwl": "0/5", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rbd_replay": "0/5", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_refs": "0/0", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_reserver": "1/1", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw": "1/5", 2024-03-24T12:27:54.930 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_access": "1/5", 2024-03-24T12:27:54.931 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_datacache": "1/5", 2024-03-24T12:27:54.931 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_dbstore": "1/5", 2024-03-24T12:27:54.931 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_flight": "1/5", 2024-03-24T12:27:54.931 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-24T12:27:54.931 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rgw_sync": "1/5", 2024-03-24T12:27:54.931 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_rocksdb": "4/5", 2024-03-24T12:27:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore": "0/5", 2024-03-24T12:27:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_backref": "0/5", 2024-03-24T12:27:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_cache": "0/5", 2024-03-24T12:27:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_cleaner": "0/5", 2024-03-24T12:27:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_device": "0/5", 2024-03-24T12:27:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_epm": "0/5", 2024-03-24T12:27:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-24T12:27:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_journal": "0/5", 2024-03-24T12:27:54.932 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_lba": "0/5", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_odata": "0/5", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_omap": "0/5", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_onode": "0/5", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_t": "0/5", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_seastore_tm": "0/5", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_striper": "0/1", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_test": "0/5", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_throttle": "1/1", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_timer": "0/1", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_tp": "0/5", 2024-03-24T12:27:54.933 INFO:teuthology.orchestra.run.smithi029.stdout: "debug_trace": "1/5", 2024-03-24T12:27:54.935 INFO:teuthology.orchestra.run.smithi029.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-24T12:27:54.935 INFO:teuthology.orchestra.run.smithi029.stdout: "device_failure_prediction_mode": "none", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "err_to_graylog": "false", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "err_to_journald": "false", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "err_to_stderr": "true", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "err_to_syslog": "false", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "event_tracing": "false", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_addr": "0.0.0.0", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_http_port": "9926", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_prio_limit": "5", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_sort_metrics": "true", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "exporter_stats_period": "5", 2024-03-24T12:27:54.936 INFO:teuthology.orchestra.run.smithi029.stdout: "fake_statfs_for_testing": "0", 2024-03-24T12:27:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "fatal_signal_handlers": "true", 2024-03-24T12:27:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "filer_max_purge_ops": "10", 2024-03-24T12:27:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "filer_max_truncate_ops": "128", 2024-03-24T12:27:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_apply_finisher_threads": "1", 2024-03-24T12:27:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_blackhole": "false", 2024-03-24T12:27:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_btrfs_clone_range": "true", 2024-03-24T12:27:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_btrfs_snap": "true", 2024-03-24T12:27:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_caller_concurrency": "10", 2024-03-24T12:27:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_collect_device_partition_information": "true", 2024-03-24T12:27:54.937 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_commit_timeout": "600.000000", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_debug_inject_read_err": "false", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_debug_omap_check": "false", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_debug_verify_split": "false", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_dump_file": "", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fadvise": "true", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fail_eio": "true", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fd_cache_shards": "16", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fd_cache_size": "128", 2024-03-24T12:27:54.938 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fiemap": "false", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fiemap_threshold": "4096", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_inject_stall": "0", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_journal_parallel": "false", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_journal_trailing": "false", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_journal_writeahead": "false", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_kill_at": "0", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattrs": "0", 2024-03-24T12:27:54.939 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_xattr_value_size": "0", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_merge_threshold": "-10", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_odsync_write": "false", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_omap_backend": "rocksdb", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_omap_backend_path": "", 2024-03-24T12:27:54.940 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_op_thread_timeout": "60", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_op_threads": "2", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_punch_hole": "false", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-24T12:27:54.941 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_queue_max_ops": "50", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_seek_data_hole": "false", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_sloppy_crc": "false", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_splice": "false", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_split_multiple": "2", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_split_rand_factor": "20", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_update_to": "1000", 2024-03-24T12:27:54.942 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_enable": "true", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_xfs_extsize": "false", 2024-03-24T12:27:54.943 INFO:teuthology.orchestra.run.smithi029.stdout: "filestore_zfs_snap": "false", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fio_dir": "/tmp/fio", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_allow_other": "true", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_atomic_o_trunc": "true", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_big_writes": "true", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_debug": "false", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_default_permissions": "false", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_disable_pagecache": "false", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_max_write": "0", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_multithreaded": "true", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_require_active_mds": "true", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_set_user_groups": "true", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_splice_move": "true", 2024-03-24T12:27:54.944 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_splice_read": "true", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_splice_write": "true", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "fuse_use_invalidate_cb": "true", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "gss_target_name": "ceph", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_file": "", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_inject_failure": "0", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_interval": "5", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "host": "smithi029", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-24T12:27:54.945 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "inject_early_sigterm": "false", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "jaeger_agent_port": "6799", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "jaeger_tracing_enable": "false", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_aio": "true", 2024-03-24T12:27:54.946 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_align_min_size": "65536", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_block_align": "true", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_block_size": "4096", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_dio": "true", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_discard": "false", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_force_aio": "false", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_ignore_corruption": "false", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_max_write_bytes": "10485760", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_max_write_entries": "100", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_replay_from": "0", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_write_header_frequency": "0", 2024-03-24T12:27:54.947 INFO:teuthology.orchestra.run.smithi029.stdout: "journal_zero_on_create": "false", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "journaler_prefetch_periods": "10", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "journaler_prezero_periods": "5", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "journaler_write_head_interval": "15", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "keyfile": "", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_backend": "rocksdb", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_default_stripe_size": "65536", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_fsck_on_mount": "false", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_max_bytes": "67108864", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_max_ops": "512", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_nid_prealloc": "1024", 2024-03-24T12:27:54.948 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_onode_map_size": "1024", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_sync_submit_transaction": "false", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "kstore_sync_transaction": "false", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "librados_thread_count": "2", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "lockdep": "false", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "lockdep_force_backtrace": "false", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_coarse_timestamps": "true", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_flush_on_exit": "false", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_graylog_host": "127.0.0.1", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_graylog_port": "12201", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_max_new": "1000", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_max_recent": "10000", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_stderr_prefix": "", 2024-03-24T12:27:54.949 INFO:teuthology.orchestra.run.smithi029.stdout: "log_stop_at_utilization": "0.970000", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_file": "true", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_graylog": "false", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_journald": "false", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_stderr": "false", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "log_to_syslog": "false", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "max_rotating_auth_attempts": "10", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_action_on_write_error": "1", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_alternate_name_max": "8192", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_asio_thread_count": "2", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_export_pin": "true", 2024-03-24T12:27:54.950 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_fragment_dirs": "true", 2024-03-24T12:27:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-24T12:27:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_fragment_interval": "5", 2024-03-24T12:27:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-24T12:27:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-24T12:27:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_interval": "10", 2024-03-24T12:27:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_max": "-1", 2024-03-24T12:27:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_max_until": "-1", 2024-03-24T12:27:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_merge_size": "50", 2024-03-24T12:27:54.951 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_midchunk": "0.300000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_min_start": "0.200000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_minchunk": "0.001000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_mode": "0", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_need_max": "1.200000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_need_min": "0.800000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_overload_epochs": "2", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_split_bits": "3", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_split_size": "10000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-24T12:27:54.952 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_target_decay": "10.000000", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_beacon_grace": "15.000000", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_beacon_interval": "4.000000", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_mid": "0.700000", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_release_free_interval": "10", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_reservation": "0.050000", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_trim_interval": "1", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cache_trim_threshold": "262144", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-24T12:27:54.953 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_client_prealloc_inos": "1000", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_connect_bootstrapping": "false", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_damage_table_max_entries": "10000", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_debug_auth_pins": "false", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_debug_frag": "false", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_debug_scatterstat": "false", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_debug_subtrees": "false", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_decay_halflife": "5.000000", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_default_dir_hash": "2", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_defer_session_stale": "true", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_deny_all_reconnect": "false", 2024-03-24T12:27:54.954 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dir_keys_per_op": "16384", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dir_max_commit_size": "10", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dir_max_entries": "0", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dir_prefetch": "true", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dump_cache_on_map": "false", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_early_reply": "true", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_enable_op_tracker": "true", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_enforce_unique_name": "true", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-24T12:27:54.955 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_export_ephemeral_random": "true", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_health_summarize_threshold": "10", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_health_dummy": "false", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-24T12:27:54.956 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_migrator_session_race": "false", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_join_fs": "", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_journal_format": "1", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_create_at": "0", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_export_at": "0", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_import_at": "0", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_journal_at": "0", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_journal_expire_at": "0", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_journal_replay_at": "0", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_link_at": "0", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_mdstable_at": "0", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_openc_at": "0", 2024-03-24T12:27:54.957 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_rename_at": "0", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_shutdown_at": "0", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_event_large_threshold": "524288", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_events_per_segment": "1024", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_max_events": "-1", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_max_segments": "128", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_pause": "false", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_segment_size": "0", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_trim_threshold": "128", 2024-03-24T12:27:54.958 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_log_warn_factor": "2.000000", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_caps_per_client": "1048576", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_completed_flushes": "100000", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_completed_requests": "100000", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_export_size": "20971520", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_file_recover": "32", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_purge_files": "64", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_purge_ops": "8192", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_max_snaps_per_dir": "100", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_metrics_update_interval": "2", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_min_caps_per_client": "100", 2024-03-24T12:27:54.959 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_min_caps_working_set": "10000", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_numa_node": "-1", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_complaint_time": "30.000000", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_history_duration": "600", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_history_size": "20", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_history_slow_op_size": "20", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_op_log_threshold": "5", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_ping_grace": "15", 2024-03-24T12:27:54.960 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_ping_interval": "5", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_max_caps": "30000", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_recall_warning_threshold": "262144", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_replay_interval": "1.000000", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_root_ino_gid": "0", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_root_ino_uid": "0", 2024-03-24T12:27:54.961 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_shutdown_check": "0", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_skip_ino": "0", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_snap_max_uid": "4294967294", 2024-03-24T12:27:54.962 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_snap_min_uid": "0", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_snap_rstat": "false", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_standby_replay_damaged": "false", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_symlink_recovery": "true", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_thrash_exports": "0", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_thrash_fragments": "0", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_tick_interval": "5.000000", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_valgrind_exit": "false", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_verify_backtrace": "1", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_verify_scatter": "false", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mds_wipe_sessions": "false", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "mempool_debug": "false", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-24T12:27:54.963 INFO:teuthology.orchestra.run.smithi029.stdout: "memstore_device_bytes": "1073741824", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "memstore_page_set": "false", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "memstore_page_size": "65536", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_client_bytes": "134217728", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_client_messages": "512", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_disabled_modules": "", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_max_pg_num_change": "128", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_mds_bytes": "134217728", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_mds_messages": "128", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-24T12:27:54.964 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_mon_bytes": "134217728", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_mon_messages": "128", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_osd_bytes": "536870912", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_osd_messages": "8192", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_pool": "true", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_standby_modules": "true", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_stats_period": "5", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_stats_threshold": "5", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_tick_period": "2", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_allow_pool_delete": "true", 2024-03-24T12:27:54.965 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_allow_pool_size_one": "false", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_auth_validate_all_caps": "true", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_bytes": "104857600", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_directed_command_retry": "2", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_hunt_parallel": "3", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_log_interval": "1.000000", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_ping_interval": "10.000000", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-24T12:27:54.966 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_client_target_rank": "-1", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_file_level": "debug", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_file": "true", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_journald": "true", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-24T12:27:54.967 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_compact_on_bootstrap": "false", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_compact_on_start": "false", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_compact_on_trim": "true", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_cpu_threads": "4", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_crush_min_required_version": "hammer", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_daemon_bytes": "419430400", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_data_avail_crit": "5", 2024-03-24T12:27:54.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_data_avail_warn": "5", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_data_size_warn": "16106127360", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_dump_json": "false", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_dump_transactions": "false", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_extra_checks": "false", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_no_require_reef": "false", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_no_require_squid": "false", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-24T12:27:54.969 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_down_mkfs_grace": "60", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_election_default_strategy": "1", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_election_timeout": "5.000000", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_elector_ping_divisor": "2", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_enable_op_tracker": "true", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_fake_pool_delete": "false", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_force_quorum_join": "false", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_globalid_prealloc": "10000", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_detail_to_clog": "true", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_log_update_period": "5", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_max_detail": "50", 2024-03-24T12:27:54.970 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_to_clog": "true", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_to_clog_interval": "600", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_host": "[v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0]", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_host_override": "", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_initial_members": "", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_lease": "5.000000", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-24T12:27:54.971 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_log_full_interval": "50", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_log_max": "10000", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_log_max_summary": "50", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_log_epochs": "500", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_osd": "10000", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_pg_per_osd": "10000", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_pool_pg_num": "65536", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mds_force_trim_to": "0", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mds_skip_sanity": "false", 2024-03-24T12:27:54.972 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_memory_autotune": "true", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_memory_target": "2147483648", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_beacon_grace": "30", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_digest_period": "5", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_inactive_grace": "60", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_min_osdmap_epochs": "50", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_complaint_time": "30", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_history_duration": "600", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_history_size": "20", 2024-03-24T12:27:54.973 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_history_slow_op_size": "20", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_op_log_threshold": "5", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_auto_mark_in": "false", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_cache_size": "500", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-24T12:27:54.974 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_down_out_interval": "600", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_force_trim_to": "0", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-24T12:27:54.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_min_down_reporters": "2", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_report_timeout": "900", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-24T12:27:54.976 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_stuck_threshold": "60", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_probe_timeout": "2.000000", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_reweight_max_change": "0.050000", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_reweight_max_osds": "4", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-24T12:27:54.977 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_scrub_interval": "86400", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_scrub_max_keys": "100", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_scrub_timeout": "300", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_session_timeout": "300", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_smart_report_timeout": "5", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_stat_smooth_intervals": "6", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_stretch_pool_min_size": "2", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_stretch_pool_size": "4", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-24T12:27:54.978 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_debug": "false", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_provider_kill_at": "0", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_requester_kill_at": "0", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_sync_timeout": "60.000000", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_target_pg_per_osd": "100", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_tick_interval": "5", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_timecheck_interval": "300.000000", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_use_min_delay_socket": "false", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_older_version_delay": "604800", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-24T12:27:54.979 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_misplaced": "false", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_older_version": "true", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-24T12:27:54.980 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "monmap": "", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "motr_tracing_enabled": "false", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_op_threads": "3", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-24T12:27:54.981 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_cm": "false", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_device_name": "", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_dscp": "96", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_local_gid": "", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_port_num": "1", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_sl": "3", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_support_srq": "true", 2024-03-24T12:27:54.982 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_rdma_type": "ib", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_async_reap_threshold": "5", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_before_connect": "false", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_ipv4": "true", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_ipv6": "false", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_msgr1": "false", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_msgr2": "true", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_port_max": "7568", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_port_min": "6800", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_retry_count": "3", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_bind_retry_delay": "5", 2024-03-24T12:27:54.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_blackhole_client": "false", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_blackhole_mds": "false", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_blackhole_mgr": "false", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_blackhole_mon": "false", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_blackhole_osd": "false", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_client_mode": "crc secure", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_cluster_mode": "crc secure", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_cluster_type": "", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_compress_secure": "false", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_connection_idle_timeout": "900", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_connection_ready_timeout": "10", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_crc_data": "true", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_crc_header": "true", 2024-03-24T12:27:54.984 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_die_on_bad_msg": "false", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_die_on_bug": "true", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_die_on_old_message": "true", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_die_on_skipped_message": "false", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_coremask": "0xF", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_enable_tso": "true", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-24T12:27:54.985 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_hugepages": "", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_lro": "true", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_memory_channel": "4", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_pmd": "", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_port_id": "0", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_dump_on_send": "false", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_initial_backoff": "0.200000", 2024-03-24T12:27:54.986 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_delay_max": "1.000000", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_delay_type": "", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_network_congestion": "0", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_inject_socket_failures": "0", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_learn_addr_from_peer": "true", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_max_accept_failures": "4", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_max_backoff": "15.000000", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_mon_client_mode": "secure crc", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_mon_service_mode": "secure crc", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_osd_compress_min_size": "1024", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_osd_compress_mode": "none", 2024-03-24T12:27:54.987 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_pq_min_cost": "65536", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_public_type": "", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_service_mode": "crc secure", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_tcp_listen_backlog": "512", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_tcp_nodelay": "true", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_tcp_rcvbuf": "0", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "ms_type": "async+posix", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "no_config_file": "false", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_completion_locks_per_session": "32", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-24T12:27:54.988 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_inflight_ops": "1024", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_tick_interval": "5.000000", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "objecter_timeout": "10.000000", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "objectstore_blackhole": "false", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "openssl_engine_opts": "", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_delay_time": "5.000000", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_hist_halflife": "1000", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_max_low_ops": "2", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_max_ops": "4", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-24T12:27:54.989 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_agent_slop": "0.020000", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_asio_thread_count": "2", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_async_recovery_min_cost": "100", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backfill_scan_max": "512", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backfill_scan_min": "64", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backoff_on_degraded": "false", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backoff_on_peering": "false", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_backoff_on_unfound": "true", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_beacon_report_interval": "300", 2024-03-24T12:27:54.990 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bench_duration": "30", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bench_max_block_size": "67108864", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_blkin_trace_all": "false", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_check_for_log_corruption": "false", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-24T12:27:54.991 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_class_update_on_start": "true", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_client_message_cap": "256", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_client_message_size_cap": "524288000", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_client_op_priority": "63", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_client_watch_timeout": "30", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_command_max_records": "256", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_compact_on_start": "false", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_crush_update_on_start": "true", 2024-03-24T12:27:54.992 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_crush_update_weight_set": "true", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_feed_pullee": "-1", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_misdirected_ops": "false", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_no_acting_change": "false", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_no_purge_strays": "false", 2024-03-24T12:27:54.993 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_op_order": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_shutdown": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_trim_objects": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_verify_snaps": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-24T12:27:54.994 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_keys": "1024", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_stride": "524288", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_default_notify_timeout": "30", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_delete_sleep": "0.000000", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_discard_disconnected_ops": "true", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_enable_op_tracker": "true", 2024-03-24T12:27:54.995 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_fast_info": "true", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_fast_shutdown": "true", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_function_tracing": "false", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_grace": "20", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_interval": "6", 2024-03-24T12:27:54.996 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_min_peers": "10", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_min_size": "2000", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_stale": "600", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_hit_set_max_size": "100000", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_hit_set_min_size": "1000", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_journal_size": "5120", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_kill_backfill_at": "0", 2024-03-24T12:27:54.997 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_map_cache_size": "50", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_map_dedup": "true", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_map_message_max": "40", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_map_share_max_epochs": "40", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_attr_name_len": "100", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_attr_size": "0", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_backfills": "1", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_markdown_count": "5", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_markdown_period": "600", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_object_name_len": "2048", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_object_namespace_len": "256", 2024-03-24T12:27:54.998 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_object_size": "134217728", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_pg_blocked_by": "16", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_pg_log_entries": "10000", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_pgls": "1024", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_push_cost": "8388608", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_push_objects": "10", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_scrubs": "3", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_trimming_pgs": "2", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_write_op_reply_len": "64", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_max_write_size": "90", 2024-03-24T12:27:54.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_profile": "balanced", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-24T12:27:55.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_base": "805306368", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_cache_min": "134217728", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_target": "4294967296", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_target_autotune": "false", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_min_pg_log_entries": "250", 2024-03-24T12:27:55.001 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_min_recovery_priority": "0", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mon_report_interval": "5", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_num_cache_shards": "32", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_num_op_tracker_shard": "32", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_numa_auto_affinity": "true", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_numa_node": "-1", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_numa_prefer_iface": "true", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_objecter_finishers": "1", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_objectstore": "bluestore", 2024-03-24T12:27:55.002 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_objectstore_fuse": "false", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_objectstore_tracing": "false", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_complaint_time": "30.000000", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_history_duration": "600", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_history_size": "20", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_history_slow_op_size": "20", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_log_threshold": "5", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_shards": "0", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_shards_hdd": "5", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_shards_ssd": "8", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-24T12:27:55.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_pq_min_cost": "65536", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_queue_cut_off": "high", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_op_thread_timeout": "15", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_open_classes_on_start": "true", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_os_flags": "0", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_peering_op_priority": "255", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_delete_cost": "1048576", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_delete_priority": "5", 2024-03-24T12:27:55.004 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-24T12:27:55.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-24T12:27:55.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-24T12:27:55.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_log_trim_max": "10000", 2024-03-24T12:27:55.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_log_trim_min": "100", 2024-03-24T12:27:55.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-24T12:27:55.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-24T12:27:55.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-24T12:27:55.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-24T12:27:55.006 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-24T12:27:55.006 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-24T12:27:55.006 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-24T12:27:55.006 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_crimson": "false", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-24T12:27:55.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_flags": "0", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_min_size": "0", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_pg_num": "32", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_pgp_num": "0", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_read_ratio": "70", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_size": "3", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_default_type": "replicated", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_push_per_object_cost": "1000", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-24T12:27:55.008 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recover_clone_overlap": "true", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_cost": "20971520", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_active": "0", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_max_single_start": "1", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_op_priority": "3", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_priority": "5", 2024-03-24T12:27:55.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_sleep": "0.000000", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_repair_during_recovery": "false", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_requested_scrub_priority": "120", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_auto_repair": "false", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_begin_hour": "0", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_begin_week_day": "0", 2024-03-24T12:27:55.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_chunk_max": "25", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_chunk_min": "5", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_cost": "52428800", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_during_recovery": "false", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_end_hour": "0", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_end_week_day": "0", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_event_cost": "4096", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_invalid_stats": "true", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_max_preemptions": "5", 2024-03-24T12:27:55.011 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_priority": "5", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_sleep": "0.000000", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_skip_data_digest": "false", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_smart_report_timeout": "5", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_cost": "1048576", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_priority": "5", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-24T12:27:55.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_target_transaction_size": "30", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-24T12:27:55.013 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_tracing": "false", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_use_stale_snap": "false", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "osdc_blkin_trace_all": "false", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_kill_at": "0", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_max_join_drift": "10", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_min": "500", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_min_wait": "0.050000", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_propose_interval": "1.000000", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_service_trim_max": "500", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_service_trim_min": "10", 2024-03-24T12:27:55.014 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_stash_full_interval": "25", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_trim_max": "500", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "paxos_trim_min": "250", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "perf": "true", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "pid_file": "", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "-", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrv": "", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "public_bind_addr": "-", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "public_network": "", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "public_network_interface": "", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "qat_compressor_enabled": "false", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "qat_compressor_session_max_number": "256", 2024-03-24T12:27:55.015 INFO:teuthology.orchestra.run.smithi029.stdout: "rados_mon_op_timeout": "0", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rados_osd_op_timeout": "0", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rados_tracing": "false", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_atime_update_interval": "60", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_balance_parent_reads": "false", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_balance_snap_reads": "false", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_blkin_trace_all": "false", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache": "true", 2024-03-24T12:27:55.016 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_policy": "writearound", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_size": "33554432", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_clone_copy_on_read": "false", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_compression_hint": "none", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_concurrent_management_ops": "10", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-24T12:27:55.017 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_clone_format": "auto", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_data_pool": "", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_features": "61", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_format": "2", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_map_options": "", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_order": "22", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_pool": "rbd", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_stripe_count": "0", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_default_stripe_unit": "0", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-24T12:27:55.018 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_enable_alloc_hint": "true", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_io_scheduler": "simple", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_order": "24", 2024-03-24T12:27:55.019 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_pool": "", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_journal_splay_width": "4", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_localize_parent_reads": "false", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_localize_snap_reads": "false", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-24T12:27:55.020 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-24T12:27:55.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_mtime_update_interval": "60", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_non_blocking_aio": "true", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_op_thread_timeout": "60", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_op_threads": "1", 2024-03-24T12:27:55.022 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_parent_cache_enabled": "false", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_plugins": "", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_bps_burst": "0", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_bps_limit": "0", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_exclude_ops": "0", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_iops_burst": "0", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_iops_limit": "0", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-24T12:27:55.023 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_read_from_replica_policy": "default", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-24T12:27:55.024 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_skip_partial_discard": "true", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_tracing": "false", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_validate_names": "true", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd_validate_pool": "true", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "restapi_base_url": "", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "restapi_log_level": "", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_acl_grants_max_num": "100", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_admin_entry": "admin", 2024-03-24T12:27:55.025 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_backend_store": "rados", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_barbican_url": "", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_beast_enable_async": "true", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_counters_cache": "false", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_cache_enabled": "true", 2024-03-24T12:27:55.026 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_cache_expiry_interval": "900", 2024-03-24T12:27:55.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_cache_lru_size": "10000", 2024-03-24T12:27:55.027 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_config_store": "rados", 2024-03-24T12:27:55.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_content_length_compat": "false", 2024-03-24T12:27:55.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_copy_obj_progress": "true", 2024-03-24T12:27:55.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_cors_rules_max_num": "100", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_cross_domain_policy": "", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_addr": "", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_password": "", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_kmip_username": "", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_require_ssl": "true", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-24T12:27:55.029 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-24T12:27:55.030 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_addr": "", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_auth": "token", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_namespace": "", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_prefix": "", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_token_file": "", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_curl_buffersize": "524288", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-24T12:27:55.031 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_curl_low_speed_time": "300", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_d4n_port": "6379", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-24T12:27:55.032 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_log_changes_size": "1000", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_log_num_shards": "128", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_log_window": "30", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-24T12:27:55.033 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-24T12:27:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-24T12:27:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-24T12:27:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-24T12:27:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-24T12:27:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-24T12:27:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-24T12:27:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-24T12:27:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-24T12:27:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dns_name": "", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dns_s3website_name": "", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_dynamic_resharding": "true", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_gc_threads": "true", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_lc_threads": "true", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_ops_log": "false", 2024-03-24T12:27:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_quota_threads": "true", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_static_website": "false", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enable_usage_log": "false", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_enforce_swift_acls": "true", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_exit_timeout_secs": "120", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_expose_bucket": "false", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_extended_http_attrs": "", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_filter": "none", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_frontends": "beast port=7480", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_deferred": "50", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_objs": "32", 2024-03-24T12:27:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_gc_processor_period": "3600", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_init_timeout": "300", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_kafka_connection_idle": "30", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-24T12:27:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_domain": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_password": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_password_path": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_project": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_tenant": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_token": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_token_path": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_admin_user": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_api_version": "2", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_barbican_domain": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_barbican_password": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_barbican_project": "", 2024-03-24T12:27:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_barbican_user": "", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_url": "", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_debug_interval": "-1", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_lock_max_time": "90", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_max_objs": "32", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_max_rules": "1000", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_max_worker": "3", 2024-03-24T12:27:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lc_thread_delay": "0", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_dnattr": "uid", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_searchfilter": "", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_log_http_headers": "", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_log_object_name_utc": "false", 2024-03-24T12:27:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_attr_name_len": "0", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_attr_size": "0", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_chunk_size": "4194304", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_listing_results": "1000", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_notify_retries": "10", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_put_param_size": "1048576", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_put_size": "5368709120", 2024-03-24T12:27:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_max_slo_entries": "1000", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_md_log_max_shards": "64", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_mp_lock_max_time": "600", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-24T12:27:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_max_gc": "300", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_num_async_rados_threads": "32", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_num_control_oids": "8", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_numa_node": "-1", 2024-03-24T12:27:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_objexp_chunk_size": "100", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_objexp_gc_interval": "600", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_op_thread_timeout": "600", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_op_tracing": "false", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_opa_token": "", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_opa_url": "", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_opa_verify_ssl": "true", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-24T12:27:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ops_log_rados": "false", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_ops_log_socket_path": "", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_period_push_interval": "2.000000", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_cache_lanes": "3", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_cache_partitions": "3", 2024-03-24T12:27:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_print_continue": "true", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_rados_tracing": "false", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_realm": "", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_realm_id": "", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_region": "", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-24T12:27:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_request_uri": "", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_batch_size": "64", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_max_aio": "128", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_num_logs": "16", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_reshard_thread_interval": "600", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_resolve_cname": "false", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_run_sync_thread": "true", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-24T12:27:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_scheduler_type": "throttler", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_script_uri": "", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_service_provider_name": "", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_client_id": "", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_client_secret": "", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_entry": "sts", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_key": "sts", 2024-03-24T12:27:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_min_session_duration": "900", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sts_token_introspection_url": "", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_account_in_url": "false", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_auth_entry": "auth", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_auth_url": "", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_custom_header": "", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_need_stats": "true", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_tenant_name": "", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_token_expiration": "86400", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_url": "", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_url_prefix": "swift", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-24T12:27:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_lease_period": "120", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_thread_pool_size": "512", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-24T12:27:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_comment": "", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_createby": "", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_encoding": "", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_flag": "false", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_origin": "", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_torrent_tracker": "", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_trust_forwarded_https": "false", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_usage_max_shards": "32", 2024-03-24T12:27:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_usage_max_user_shards": "1", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_use_opa_authz": "false", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_counters_cache": "false", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_max_buckets": "1000", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_policies_max_num": "100", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_user_unique_email": "true", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_verify_ssl": "true", 2024-03-24T12:27:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zone": "", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zone_id": "", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zonegroup": "", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zonegroup_id": "", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rgwlc_auto_session_clear": "true", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_block_size": "4096", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-24T12:27:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-24T12:27:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_size": "536870912", 2024-03-24T12:27:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-24T12:27:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-24T12:27:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-24T12:27:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-24T12:27:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-24T12:27:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_index_type": "binary_search", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_partition_filters": "false", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_perf": "false", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "run_dir": "/var/run/ceph", 2024-03-24T12:27:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_block_create": "true", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_cache_lru_size": "67108864", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_default_max_object_size": "16777216", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_device_size": "53687091200", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_journal_batch_capacity": "16", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_main_device_type": "SSD", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-24T12:27:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "seastore_segment_size": "67108864", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "set_keepcaps": "false", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "setgroup": "ceph", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "setuser": "ceph", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "setuser_match_path": "", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "thp": "false", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "threadpool_default_timeout": "60", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "throttler_perf_counter": "true" 2024-03-24T12:27:55.057 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:27:55.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:54 smithi029 ceph-mon[29919]: 6.d scrub starts 2024-03-24T12:27:55.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:54 smithi029 ceph-mon[29919]: 6.d scrub ok 2024-03-24T12:27:55.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:54 smithi029 ceph-mon[29919]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:55.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:54 smithi029 ceph-mon[41500]: 6.d scrub starts 2024-03-24T12:27:55.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:54 smithi029 ceph-mon[41500]: 6.d scrub ok 2024-03-24T12:27:55.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:54 smithi029 ceph-mon[41500]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:54 smithi190 ceph-mon[33748]: 6.d scrub starts 2024-03-24T12:27:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:54 smithi190 ceph-mon[33748]: 6.d scrub ok 2024-03-24T12:27:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:54 smithi190 ceph-mon[33748]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:27:55.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-24T12:27:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[29919]: 4.1a scrub starts 2024-03-24T12:27:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[29919]: 4.1a scrub ok 2024-03-24T12:27:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:27:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:27:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[29919]: 6.12 scrub starts 2024-03-24T12:27:56.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[29919]: 6.12 scrub ok 2024-03-24T12:27:56.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[41500]: 4.1a scrub starts 2024-03-24T12:27:56.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[41500]: 4.1a scrub ok 2024-03-24T12:27:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:27:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:27:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[41500]: 6.12 scrub starts 2024-03-24T12:27:56.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:55 smithi029 ceph-mon[41500]: 6.12 scrub ok 2024-03-24T12:27:56.170 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:56.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:55 smithi190 ceph-mon[33748]: 4.1a scrub starts 2024-03-24T12:27:56.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:55 smithi190 ceph-mon[33748]: 4.1a scrub ok 2024-03-24T12:27:56.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:55 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:27:56.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:55 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:27:56.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:55 smithi190 ceph-mon[33748]: 6.12 scrub starts 2024-03-24T12:27:56.286 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:55 smithi190 ceph-mon[33748]: 6.12 scrub ok 2024-03-24T12:27:57.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:56 smithi190 ceph-mon[33748]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-03-24T12:27:57.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:56 smithi029 ceph-mon[29919]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-03-24T12:27:57.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:56 smithi029 ceph-mon[41500]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 0 op/s 2024-03-24T12:27:57.784 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:27:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:27:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:27:57.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:27:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:27:58.041 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a: {} 2024-03-24T12:27:58.041 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:57 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:58.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:57 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:58.051 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b: {} 2024-03-24T12:27:58.061 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c: {} 2024-03-24T12:27:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T12:27:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T12:27:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T12:27:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:57 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:27:58.770 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-24T12:27:58.780 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T12:27:58.781 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T12:27:58.781 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-24T12:27:58.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-24T12:27:58.782 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-24T12:27:58.782 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:27:58.782 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:27:58.783 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-24T12:27:58.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:27:58.783 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-24T12:27:58.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:27:58.784 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-24T12:27:58.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:27:58.785 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-24T12:27:58.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:27:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:27:58.812 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:27:58.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:27:58.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:27:58.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:27:58.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:27:58.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:27:58.998 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:59 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1006872554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:27:58.999 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:27:59 smithi029 ceph-mon[41500]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:59.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:27:59.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:27:59.071 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:27:59.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:27:59.090 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:59.090 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:59.094 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:59.094 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:27:59.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1006872554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:27:59.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-mon[33748]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:59.135 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:58 smithi190 bash[66583]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-24T12:27:59.135 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:58 smithi190 bash[66583]: Writing manifest to image destination 2024-03-24T12:27:59.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:27:59.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:27:59.198 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:27:59.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:27:59.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:27:59.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:59 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1006872554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:27:59.307 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:27:59 smithi029 ceph-mon[29919]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:27:59.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:27:59.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:27:59.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:27:59.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:27:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:27:59.450 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:27:59.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:27:59.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:27:59.570 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 podman[66583]: 2024-03-24 12:27:59.221492276 +0000 UTC m=+6.663483538 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-24T12:27:59.570 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 podman[66583]: 2024-03-24 12:27:59.312144023 +0000 UTC m=+6.754135285 container create e5177deff2c0bb4af4de3cb5bf272d0f901250fff2d2fd84c1f13bfef08f73ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:27:59.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:27:59.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:27:59.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:27:59.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:27:59.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:27:59.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:27:59.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:27:59.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:27:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:27:59.831 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:27:59.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 podman[66583]: 2024-03-24 12:27:59.573299519 +0000 UTC m=+7.015290787 container init e5177deff2c0bb4af4de3cb5bf272d0f901250fff2d2fd84c1f13bfef08f73ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 podman[66583]: 2024-03-24 12:27:59.578545425 +0000 UTC m=+7.020536698 container start e5177deff2c0bb4af4de3cb5bf272d0f901250fff2d2fd84c1f13bfef08f73ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.582Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.582Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.582Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.582Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=arp 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-24T12:27:59.885 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-24T12:27:59.886 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-24T12:27:59.886 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-24T12:27:59.886 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-24T12:27:59.886 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-24T12:27:59.886 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-24T12:27:59.886 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=edac 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-24T12:27:59.887 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=os 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=stat 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-24T12:27:59.888 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=time 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=uname 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.583Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.584Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b[66697]: ts=2024-03-24T12:27:59.584Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 bash[66583]: e5177deff2c0bb4af4de3cb5bf272d0f901250fff2d2fd84c1f13bfef08f73ca 2024-03-24T12:27:59.889 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:27:59 smithi190 systemd[1]: Started Ceph node-exporter.b for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:27:59.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:27:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:27:59.958 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:27:59.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:00.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:00.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:00.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:00.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:00.209 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:00.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:00.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:00.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:00.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:00.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:00.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:00.461 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:00.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:00.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:00.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:00.587 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:00.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:00.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:00.711 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:00.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:00.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:00.838 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:00.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:00.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:00.964 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:00.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: 4.1 deep-scrub starts 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: 4.1 deep-scrub ok 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: 3.17 scrub starts 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: 3.17 scrub ok 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:28:00.974 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:00 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:28:01.037 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: 4.1 deep-scrub starts 2024-03-24T12:28:01.037 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: 4.1 deep-scrub ok 2024-03-24T12:28:01.037 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.037 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.037 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: 3.17 scrub starts 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: 3.17 scrub ok 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: 4.1 deep-scrub starts 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: 4.1 deep-scrub ok 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.038 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:28:01.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: 3.17 scrub starts 2024-03-24T12:28:01.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: 3.17 scrub ok 2024-03-24T12:28:01.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:28:01.039 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:00 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:28:01.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:01.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:01.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:01.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:01.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:01.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:01.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:01.343 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:01.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:01.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:01.470 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:01.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:01.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:01.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:01.598 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:01.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:01.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:01.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:01.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:01.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:01.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:01.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:01.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:01.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:01.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:01.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:01.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:01.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:01.982 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:01 smithi190 ceph-mon[33748]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:01.982 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:01 smithi190 ceph-mon[33748]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T12:28:01.982 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:01 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:01.982 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:01 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:02.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:02.104 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:02.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:02.128 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:01 smithi029 ceph-mon[41500]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:02.129 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:01 smithi029 ceph-mon[41500]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T12:28:02.129 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:01 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:02.129 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:01 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:02.129 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:01 smithi029 ceph-mon[29919]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:02.129 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:01 smithi029 ceph-mon[29919]: osdmap e94: 8 total, 8 up, 8 in 2024-03-24T12:28:02.129 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:01 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:02.129 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:01 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:02.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:02.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:02.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:02.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:02.359 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:02.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:02.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:02.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:02.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:02.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:02.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:02.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:02.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:02.612 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:02.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:02.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:02.740 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:02.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:02.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:02.867 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:02.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:02.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:02.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:02 smithi190 ceph-mon[33748]: 3.1a scrub starts 2024-03-24T12:28:02.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:02 smithi190 ceph-mon[33748]: 3.1a scrub ok 2024-03-24T12:28:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:02 smithi029 ceph-mon[41500]: 3.1a scrub starts 2024-03-24T12:28:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:02 smithi029 ceph-mon[41500]: 3.1a scrub ok 2024-03-24T12:28:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:02 smithi029 ceph-mon[29919]: 3.1a scrub starts 2024-03-24T12:28:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:02 smithi029 ceph-mon[29919]: 3.1a scrub ok 2024-03-24T12:28:02.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:02.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:02.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:03.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:03.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:03.120 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:03.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:03.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:03.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:03.246 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:03.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:03.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:03.370 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:03.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:03.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:03.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:03.496 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:03.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:03.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:03.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:03.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:03.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:03.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:03.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:03.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:03.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:03.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:03.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:03.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:03.879 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:03 smithi029 ceph-mon[29919]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:28:03.879 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:03 smithi029 ceph-mon[29919]: 4.d deep-scrub starts 2024-03-24T12:28:03.879 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:03 smithi029 ceph-mon[29919]: 4.d deep-scrub ok 2024-03-24T12:28:03.879 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:03 smithi029 ceph-mon[41500]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:28:03.879 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:03 smithi029 ceph-mon[41500]: 4.d deep-scrub starts 2024-03-24T12:28:03.879 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:03 smithi029 ceph-mon[41500]: 4.d deep-scrub ok 2024-03-24T12:28:03.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:03.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:03.999 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:03.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:04.022 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:28:04.042 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:28:04.061 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:28:04.081 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:28:04.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:04.100 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:28:04.116 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:03 smithi190 ceph-mon[33748]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:28:04.116 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:03 smithi190 ceph-mon[33748]: 4.d deep-scrub starts 2024-03-24T12:28:04.116 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:03 smithi190 ceph-mon[33748]: 4.d deep-scrub ok 2024-03-24T12:28:04.122 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:04.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:04.131 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:28:04.152 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:28:04.171 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:28:04.172 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:28:04.172 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:28:04.173 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:28:04.173 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:28:04.173 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:28:04.173 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:28:04.173 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:28:04.173 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:28:04.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:04.249 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:04.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:04.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:04.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:04.374 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:04.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:04.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:04.504 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:04.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:04.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:04.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:04.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:04.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:04.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:04.754 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:04.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:04.788 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'unique_pool_0' created 2024-03-24T12:28:04.832 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:04 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1649683244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:28:04.832 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:04 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1573542348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T12:28:04.832 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:04 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T12:28:04.832 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:04.832 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:04.832 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:04 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1649683244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:28:04.833 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:04 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1573542348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T12:28:04.833 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:04 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T12:28:04.833 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:04.833 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:04.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:04.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:04.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:04.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:05.010 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:05.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:05.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:05.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:04 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1649683244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:28:05.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:04 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1573542348' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T12:28:05.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:04 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-24T12:28:05.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:05.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:05.135 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:05.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:05.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:05.260 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:05.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:05.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:05.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:05.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:05.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:05.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:05.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-24T12:28:05.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:05.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:05.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:05.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:05.639 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:05.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:05.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:05.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:05.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:05.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:05.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:05.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:05.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:05.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:06.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:06.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:06.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:06.020 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:05 smithi029 ceph-mon[41500]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:28:06.020 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:05 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-24T12:28:06.020 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:05 smithi029 ceph-mon[41500]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T12:28:06.020 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:05 smithi029 ceph-mon[29919]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:28:06.021 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:05 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-24T12:28:06.021 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:05 smithi029 ceph-mon[29919]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T12:28:06.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:06.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:05 smithi190 ceph-mon[33748]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:28:06.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:05 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-24T12:28:06.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:05 smithi190 ceph-mon[33748]: osdmap e95: 8 total, 8 up, 8 in 2024-03-24T12:28:06.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:06.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:06.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:06.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:06.267 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:06.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:06.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:06.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:06.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:06.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:06.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:06.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:06.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:06.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:06.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:06.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:06.645 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:06.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:06.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:06.770 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:06.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:06.795 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:28:06.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:06.895 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:06.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:06.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:06 smithi029 ceph-mon[29919]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T12:28:06.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:06 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1308233087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:28:06.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:06 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:28:06.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:06 smithi029 ceph-mon[41500]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T12:28:06.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:06 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1308233087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:28:06.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:06 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:28:06.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:07.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:07.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:07.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:07.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:07.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:06 smithi190 ceph-mon[33748]: osdmap e96: 8 total, 8 up, 8 in 2024-03-24T12:28:07.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:06 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1308233087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:28:07.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:06 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:28:07.146 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:07.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:07.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:07.222 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:28:07.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:07.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:07.273 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:07.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:07.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:07.398 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:07.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:07.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:07.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:07.524 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:07.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:07.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:07.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:07.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:07.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:07.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:07.777 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:07.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:07.789 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:28:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:28:07.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:07.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:07.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:07.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:07.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:28:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:28:07.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[29919]: 5.1 scrub starts 2024-03-24T12:28:07.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[29919]: 5.1 scrub ok 2024-03-24T12:28:07.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[29919]: pgmap v56: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:07.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[29919]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:28:07.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:28:07.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:28:07.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[29919]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T12:28:07.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[41500]: 5.1 scrub starts 2024-03-24T12:28:07.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[41500]: 5.1 scrub ok 2024-03-24T12:28:07.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[41500]: pgmap v56: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:07.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[41500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:28:07.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:28:07.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:28:07.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:07 smithi029 ceph-mon[41500]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T12:28:08.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:08.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:08.028 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:08.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:08.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:08.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:07 smithi190 ceph-mon[33748]: 5.1 scrub starts 2024-03-24T12:28:08.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:07 smithi190 ceph-mon[33748]: 5.1 scrub ok 2024-03-24T12:28:08.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:07 smithi190 ceph-mon[33748]: pgmap v56: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 231 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:08.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:07 smithi190 ceph-mon[33748]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:28:08.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:07 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:28:08.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:07 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:28:08.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:07 smithi190 ceph-mon[33748]: osdmap e97: 8 total, 8 up, 8 in 2024-03-24T12:28:08.154 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:08.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:08.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:08.280 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:08.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:08.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:08.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:08.409 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:08.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:08.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:08.534 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:08.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:08.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:08.602 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:28:08.603 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:28:08.636 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:08.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:08.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:08.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:08.790 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:08.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:08.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:08.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:08.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:08.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:08.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:09.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:28:09.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:09.075 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:09.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:09.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:09.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:09.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:09.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:09.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:09.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:09.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:09.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:09.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:09.457 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:09.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:09.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:09.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:09.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:09.583 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:09.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:09.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:09.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:09.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:09.814 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:09.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:09.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:09.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:09.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:09.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:09.973 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:09.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:10.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:10.098 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:10.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:10.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:28:10.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:09 smithi190 ceph-mon[33748]: pgmap v58: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:10.162 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:09 smithi029 ceph-mon[41500]: pgmap v58: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:10.162 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:09 smithi029 ceph-mon[29919]: pgmap v58: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:10.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:10.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:10.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:10.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:10.353 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:10.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:10.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:10.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:10.479 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:10.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:10.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:10.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:10.609 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:10.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:10.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:10.738 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:10.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:10.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T12:28:10.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:10.870 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:10.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:10.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:10.996 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:10.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:11.097 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:11.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:11.122 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:11.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:11.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:11.252 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:11.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:11.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:11.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:11.379 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:11.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:11.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:11.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:11.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:11.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:11.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:11.635 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:11.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:11.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:11.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:11.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:11.761 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:11.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:11.808 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-24T12:28:11.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:11.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:11.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:11.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:11.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:12.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:12.018 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:12.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:28:12.084 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:28:12.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:12.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:11 smithi190 ceph-mon[33748]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:12.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:11 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1661987465' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:28:12.142 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:12.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:12.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:11 smithi029 ceph-mon[29919]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:12.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:11 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1661987465' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:28:12.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:11 smithi029 ceph-mon[41500]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:28:12.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:11 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1661987465' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:28:12.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:12.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:12.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:12.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:12.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:12.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:12.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:12.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:12.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:12.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:12.580 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:28:12.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:12.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:12.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:12.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:12.785 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:12.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:12.862 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:12 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1661987465' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:28:12.862 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:12 smithi029 ceph-mon[41500]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T12:28:12.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:12 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1661987465' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:28:12.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:12 smithi029 ceph-mon[29919]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T12:28:12.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:12.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:12.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:12.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:13.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:13.041 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:13.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:13.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:12 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1661987465' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:28:13.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:12 smithi190 ceph-mon[33748]: osdmap e98: 8 total, 8 up, 8 in 2024-03-24T12:28:13.143 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:13.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:13.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:13.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:13.300 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:13.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:13.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:13.428 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:13.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:13.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:13.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:13.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:13.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:13.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:13.597 DEBUG:teuthology.orchestra.run.smithi029:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-24T12:28:13.598 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-03-24T12:28:13.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:28:13.600 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T12:28:13.600 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T12:28:13.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:13.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:hints = 1 2024-03-24T12:28:13.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T12:28:13.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Object prefix: benchmark_data_smithi029_84282 2024-03-24T12:28:13.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:28:13.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 0 0 0 0 0 0 - 0 2024-03-24T12:28:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:13.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:13.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:13.843 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:13.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:13.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:13.988 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:13.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:13.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:14.071 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:13 smithi029 ceph-mon[41500]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T12:28:14.071 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:13 smithi029 ceph-mon[41500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:28:14.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:13 smithi029 ceph-mon[29919]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T12:28:14.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:13 smithi029 ceph-mon[29919]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:28:14.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:14.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:14.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:14.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:14.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:13 smithi190 ceph-mon[33748]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T12:28:14.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:13 smithi190 ceph-mon[33748]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:28:14.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:14.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:14.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:14.373 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:14.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:14.417 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:14.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:14.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:14.552 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:14.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:14.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:14.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:14.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 1 16 3262 3246 202.863 202.875 0.0067633 0.0049105 2024-03-24T12:28:14.686 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:14.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:14.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:14.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:14.823 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:14.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:14.924 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:14.967 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:14.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:14.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:15.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:15.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:15.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:15.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:15.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:15.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:15.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:15.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:15.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:15.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:15.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:15.530 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:15.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:15.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:15.670 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:15.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:15.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:15.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 2 16 6456 6440 201.216 199.625 0.00442298 0.00496102 2024-03-24T12:28:15.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:15.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:15.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:15.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:15.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:15.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:15.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:16.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:16.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:15 smithi029 ceph-mon[29919]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:28:16.073 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:15 smithi029 ceph-mon[41500]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:28:16.079 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:16.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:16.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:16.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:15 smithi190 ceph-mon[33748]: pgmap v62: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:28:16.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:16.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:16.228 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:16.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:16.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:16.368 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:16.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:16.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:16.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:16.501 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:16.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:16.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:16.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:16.637 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:16.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:16.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 3 16 8874 8858 184.515 151.125 0.00320229 0.00541113 2024-03-24T12:28:16.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:16.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:16.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:16.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:16.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:16.911 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:16.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:17.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:17.047 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:17.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:17.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:17.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:17.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:17.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:17.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:17.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:17.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:17.466 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:17.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:17.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:17.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:17.615 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:17.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:17.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 4 16 11790 11774 183.945 182.25 0.00629067 0.00542955 2024-03-24T12:28:17.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:17.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:17.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:17.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:17.858 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:17 smithi029 ceph-mon[41500]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:28:17.858 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:17.858 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:17.859 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:17 smithi029 ceph-mon[29919]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:28:17.859 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:17.859 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:17.859 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:28:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:28:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:28:17.881 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:17.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:17.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:28:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:28:17.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:17 smithi190 ceph-mon[33748]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:28:17.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:17.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:17.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:18.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:18.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:18.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:18.146 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:18.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:18.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:18.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:18.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:18.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:18.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:18.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:18.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:18.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:18.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:18.560 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:18.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:18.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:18.566 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:28:18.567 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:28:18.584 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:28:18.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:18.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 5 16 14168 14152 176.874 148.625 0.00500032 0.00564662 2024-03-24T12:28:18.703 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:18.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:18.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:18.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:18.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:18.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:18 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1544679248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:28:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:18 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:28:18.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:18.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:18 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1544679248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:28:18.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:18 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:28:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:18 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1544679248' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:28:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:18 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:28:18.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:18.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:18.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:19.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:19.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:19.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:19.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:19.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:19.242 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:19.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:19.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:19.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:19.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:19.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:19.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:19.520 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:19.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:19.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:19.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:19.662 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:19.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:19.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:19 smithi029 ceph-mon[29919]: pgmap v64: 148 pgs: 148 active+clean; 690 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T12:28:19.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:19 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:28:19.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:19 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:28:19.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:19 smithi029 ceph-mon[29919]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T12:28:19.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:19 smithi029 ceph-mon[41500]: pgmap v64: 148 pgs: 148 active+clean; 690 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T12:28:19.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:19 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:28:19.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:19 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:28:19.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:19 smithi029 ceph-mon[41500]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T12:28:19.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 6 16 16910 16894 175.955 171.375 0.00706579 0.00567477 2024-03-24T12:28:19.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:19.798 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:19.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:19.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:19 smithi190 ceph-mon[33748]: pgmap v64: 148 pgs: 148 active+clean; 690 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T12:28:19.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:19 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:28:19.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:19 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:28:19.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:19 smithi190 ceph-mon[33748]: osdmap e99: 8 total, 8 up, 8 in 2024-03-24T12:28:19.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:19.934 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:19.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:20.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:20.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:20.079 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:20.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:20.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:20.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:20.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:20.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:20.314 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:20.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T12:28:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:20.364 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:20.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:20.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:20.511 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:20.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:20.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:20.644 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:20.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:20.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 7 15 19337 19322 172.494 151.75 0.00334288 0.00579098 2024-03-24T12:28:20.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:20.779 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:20.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:20.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:20.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:20.938 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:20.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:21.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:21.086 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:21.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:21.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:21.212 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:21.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:21.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:21.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:21.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:21.346 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:21.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:21.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:21.486 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:21.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:21.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:21.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:21.620 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:21.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:21.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 8 16 21376 21360 166.853 127.375 0.00361346 0.00597524 2024-03-24T12:28:21.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:21.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:21.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:21.899 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:21.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:21.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:22.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:22.039 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:22.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:22.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:22.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:21 smithi190 ceph-mon[33748]: pgmap v66: 148 pgs: 148 active+clean; 690 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:28:22.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:22.168 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:22.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:22.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:21 smithi029 ceph-mon[29919]: pgmap v66: 148 pgs: 148 active+clean; 690 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:28:22.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:21 smithi029 ceph-mon[41500]: pgmap v66: 148 pgs: 148 active+clean; 690 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:28:22.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:22.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:22.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:22.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:22.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:22.441 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:22.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:22.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:22.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:22.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:22.567 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:22.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:22.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:22.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 9 16 23246 23230 161.299 116.875 0.00438241 0.0061349 2024-03-24T12:28:22.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:22.692 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:22.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:22.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:22.822 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:22.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:22.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:22.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:28:22.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:22.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:22.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:22.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:23.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:23.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:23.091 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:23.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:23.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:23.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:23.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:23.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:23.342 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:23.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:23.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:23.475 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:23.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:23.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:23.577 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:23.619 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:23.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:23.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 10 15 24751 24736 154.58 94.125 0.0100275 0.00646262 2024-03-24T12:28:23.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:23.751 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:23.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:23.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:23.883 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:23.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:23.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:23.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:24.024 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:24.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:24.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:24.158 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:24.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:24.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[29919]: pgmap v67: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-24T12:28:24.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:28:24.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:28:24.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[29919]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T12:28:24.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:28:24.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:28:24.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[41500]: pgmap v67: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-24T12:28:24.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:28:24.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:28:24.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[41500]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T12:28:24.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:28:24.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:23 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:28:24.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:24.299 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:24.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:24.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:24.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:23 smithi190 ceph-mon[33748]: pgmap v67: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 143 MiB/s wr, 2.29k op/s 2024-03-24T12:28:24.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:23 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:28:24.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:23 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:28:24.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:23 smithi190 ceph-mon[33748]: osdmap e100: 8 total, 8 up, 8 in 2024-03-24T12:28:24.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:23 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:28:24.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:23 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:28:24.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:24.434 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:24.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:24.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:24.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:24.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:24.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:24.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:24.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 11 16 26814 26798 152.242 128.875 0.00379655 0.00654524 2024-03-24T12:28:24.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:24.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:24.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:24.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:24.828 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:24.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:24.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:24.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:24.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:24.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:24.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:25.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:25.110 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:28:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:25.111 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:25.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:25.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:24 smithi029 ceph-mon[29919]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T12:28:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:24 smithi029 ceph-mon[41500]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T12:28:25.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:25.258 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:25.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:25.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:25.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:25.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:25.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:24 smithi190 ceph-mon[33748]: osdmap e101: 8 total, 8 up, 8 in 2024-03-24T12:28:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:25.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:25.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:25.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:25.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:25.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:25.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:25.651 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:25.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:25.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 12 16 28554 28538 148.618 108.75 0.00515854 0.0067227 2024-03-24T12:28:25.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:25.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:25.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:25.915 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:25.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:26.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:26.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:26.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:26.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:26.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:26.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:26.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:25 smithi190 ceph-mon[33748]: pgmap v70: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-03-24T12:28:26.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:26.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:26.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:26.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:26.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:25 smithi029 ceph-mon[29919]: pgmap v70: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-03-24T12:28:26.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:25 smithi029 ceph-mon[41500]: pgmap v70: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 124 MiB/s wr, 1.99k op/s 2024-03-24T12:28:26.440 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:26.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:26.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:26.542 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:26.581 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:26.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:26.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:26.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 13 16 30086 30070 144.55 95.75 0.00981125 0.00691214 2024-03-24T12:28:26.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:26.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:26.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:26.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:26.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:26.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:26.840 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:26.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:26.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:26.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:27.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:27.113 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:27.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:27.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:27.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:27.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:27.240 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:27.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:27.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:27.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:27.365 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:27.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:27.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:27.509 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:27.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:27.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:27.651 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:28:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:28:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:28:27.653 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:27.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:27.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:27.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 14 16 31500 31484 140.537 88.375 0.00960045 0.00710937 2024-03-24T12:28:27.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:27.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:27.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:27.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:28:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:28:27.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:27.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:27.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:28.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:28.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:28.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:28.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:27 smithi029 ceph-mon[41500]: pgmap v71: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 653 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-24T12:28:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:27 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:28:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:27 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:28:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:27 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2116773390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:28:28.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:27 smithi029 ceph-mon[29919]: pgmap v71: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 653 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-24T12:28:28.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:27 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:28:28.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:27 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:28:28.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:27 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2116773390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:28:28.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:28.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:28.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:28.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:28.331 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:28.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:28.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:27 smithi190 ceph-mon[33748]: pgmap v71: 148 pgs: 148 active+clean; 1.4 GiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 653 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-24T12:28:28.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:27 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:28:28.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:27 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:28:28.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:27 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2116773390' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:28:28.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:28.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:28.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:28.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:28.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:28.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:28.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:28.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:28.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 15 16 32883 32867 136.93 86.4375 0.00637785 0.00728399 2024-03-24T12:28:28.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:28.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:28.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:28.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:28.856 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:28.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:28.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:29.000 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:29.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:29.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:29.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:29.139 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:29.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:29.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:29.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:29.264 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:29.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:29.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:28 smithi190 ceph-mon[33748]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T12:28:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:28 smithi190 ceph-mon[33748]: pgmap v73: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-24T12:28:29.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:29.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:29.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:29.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:28 smithi029 ceph-mon[29919]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T12:28:29.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:28 smithi029 ceph-mon[29919]: pgmap v73: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-24T12:28:29.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:29.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:28 smithi029 ceph-mon[41500]: osdmap e102: 8 total, 8 up, 8 in 2024-03-24T12:28:29.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:28 smithi029 ceph-mon[41500]: pgmap v73: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-24T12:28:29.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:29.525 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:29.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:29.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:29.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:29.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:29.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:29.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 16 16 34173 34157 133.411 80.625 0.00599288 0.00748875 2024-03-24T12:28:29.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:29.804 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:29.805 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:29.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:29.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:29.930 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:29.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:30.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:30.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:30.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:30.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:30.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:30.183 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:30.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:30.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:30.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:30.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:30.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:30.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:30.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:30.461 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:30.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:30.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:30.537 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:28:30.537 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:28:30.537 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:28:30.537 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:28:30.537 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:28:30.537 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:28:30.537 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:28:30.537 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:28:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:28:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:28:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:28:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:28:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:28:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:28:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:28:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:28:30.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:30.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:30.589 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:30.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:30.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 17 15 35313 35298 129.758 71.3125 0.11374 0.00767858 2024-03-24T12:28:30.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:30.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:30.714 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:30.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:30.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:30.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:30.839 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:30.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:30.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:30.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:30.965 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:30.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:31.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:31.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:31.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:31.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:31.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:31.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:31.224 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:31.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:31.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:31.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:31.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:31.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:31.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:31.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:31.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:31.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:31.635 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:31.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:31.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:31.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 18 16 36381 36365 126.253 66.6875 0.00393497 0.00791505 2024-03-24T12:28:31.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:31.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:31.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:31.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:31.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:31.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:31.905 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:31.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:32.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:32.040 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:32.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:32.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:31 smithi190 ceph-mon[33748]: pgmap v74: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T12:28:32.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:32.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:32.166 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:32.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:32.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:31 smithi029 ceph-mon[41500]: pgmap v74: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T12:28:32.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:32.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:31 smithi029 ceph-mon[29919]: pgmap v74: 148 pgs: 148 active+clean; 1.9 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 682 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T12:28:32.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:32.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:32.311 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:32.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:32.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:32.447 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:32.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:32.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:32.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:32.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:32.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 19 16 37948 37932 124.763 97.9375 0.00382247 0.00800899 2024-03-24T12:28:32.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:32.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:32.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:32.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:32.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:32.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:32.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:32.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:32.969 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:32.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:33.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:33.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:33.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:33.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:33.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:33.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:33.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:33.385 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:33.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:33.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:33.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:33.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:33.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:33.520 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:33.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:33.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:33.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:33.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:33.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:28:33.676648+0000 min lat: 0.00114673 max lat: 0.335061 avg lat: 0.00808953 2024-03-24T12:28:33.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:28:33.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 20 16 39554 39538 123.543 100.375 0.00761098 0.00808953 2024-03-24T12:28:33.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:33.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:33.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:33.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:33.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:33.915 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:33.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:34.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:34.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:34.057 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:34.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:34.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:34.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:33 smithi029 ceph-mon[41500]: pgmap v75: 148 pgs: 148 active+clean; 2.1 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T12:28:34.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:33 smithi029 ceph-mon[29919]: pgmap v75: 148 pgs: 148 active+clean; 2.1 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T12:28:34.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:34.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:34.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:34.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:34.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:34.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:33 smithi190 ceph-mon[33748]: pgmap v75: 148 pgs: 148 active+clean; 2.1 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T12:28:34.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:34.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:34.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:34.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:34.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:34.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:34.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:34.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 21 16 40787 40771 121.329 77.0625 0.00807048 0.00823647 2024-03-24T12:28:34.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:34.715 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:34.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:34.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:34.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:34.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:34.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:34.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:34.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:34.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:34.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:35.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:35.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:35.121 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:35.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:35.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:35.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:35.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:35.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:35.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:35.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:35.490 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:35.535 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:35.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:35.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:35.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:35.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:35.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:35.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 22 16 42063 42047 119.439 79.75 0.00453669 0.00834437 2024-03-24T12:28:35.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:35.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:35.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:35.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:35.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:35.932 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:35.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:35.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:36.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:36.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:36.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:36.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:36.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:36.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:35 smithi029 ceph-mon[41500]: pgmap v76: 148 pgs: 148 active+clean; 2.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T12:28:36.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:35 smithi029 ceph-mon[29919]: pgmap v76: 148 pgs: 148 active+clean; 2.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T12:28:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:36.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:36.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:36.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:36.347 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:36.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:35 smithi190 ceph-mon[33748]: pgmap v76: 148 pgs: 148 active+clean; 2.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T12:28:36.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:36.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:36.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:36.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:36.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:36.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:36.610 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:36.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:36.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 23 16 43463 43447 118.05 87.5 0.00331745 0.00846588 2024-03-24T12:28:36.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:36.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:36.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:36.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:36.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:36.868 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:36.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:36.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:37.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:37.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:37.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:37.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:37.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:37.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:37.277 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:37.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:37.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:37.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:37.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:37.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:37.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:37.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:37.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:37.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:37.570 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:28:37.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:37.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 24 16 44923 44907 116.927 91.25 0.00549505 0.00854684 2024-03-24T12:28:37.680 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:37.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:37.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:37.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:37.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:37.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:37.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:28:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:28:37.895 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:28:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:28:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:28:37.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:37.967 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:37.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:37.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:38.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:38.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:38.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:38.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:38.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:37 smithi029 ceph-mon[41500]: pgmap v77: 148 pgs: 148 active+clean; 2.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T12:28:38.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:37 smithi029 ceph-mon[29919]: pgmap v77: 148 pgs: 148 active+clean; 2.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T12:28:38.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:38.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:38.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:38.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:38.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:38.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:38.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:38.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:37 smithi190 ceph-mon[33748]: pgmap v77: 148 pgs: 148 active+clean; 2.3 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-24T12:28:38.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:38.526 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:38.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:38.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:38.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:38.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:38.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:38.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 25 15 46601 46586 116.447 104.938 0.00850566 0.00858271 2024-03-24T12:28:38.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:38.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:38.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:38.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:38.939 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:38.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:38.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:39.017 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:28:39.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:39.094 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:39.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:39.141 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:38 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:39.141 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:38 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:39.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:39.243 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:39.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:39.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:39.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:39.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:38 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:39.399 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:39.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:39.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:39.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:28:39.542 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:39.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:39.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:39.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 26 16 48175 48159 115.749 98.3125 0.0138977 0.0086337 2024-03-24T12:28:39.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:39.703 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:39.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:39.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:39.831 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:39.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:39.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:39.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:39 smithi029 ceph-mon[29919]: pgmap v78: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 977 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-24T12:28:39.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:39.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:39.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:39.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:40.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:40.108 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:40.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:40.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:40.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:40.242 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:40.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:40.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:40.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:39 smithi029 ceph-mon[41500]: pgmap v78: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 977 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-24T12:28:40.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:40.374 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:40.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:39 smithi190 ceph-mon[33748]: pgmap v78: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 977 B/s rd, 80 MiB/s wr, 1.29k op/s 2024-03-24T12:28:40.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:40.505 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:40.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:40.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:40.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:40.653 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:40.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:40.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:40.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 27 16 49650 49634 114.876 92.1875 0.011814 0.00869969 2024-03-24T12:28:40.755 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:40.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:40.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:40.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:40.931 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:40.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:40.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:41.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:41.072 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:41.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:41.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:41.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:41.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:41.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:41.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:41.346 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:41.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:41.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:41.475 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:41.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:41.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:41.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:41.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:41.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:41.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:41.675 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 28 16 50976 50960 113.733 82.875 0.0218594 0.0087872 2024-03-24T12:28:41.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:41.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:41.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:41.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:41.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:42.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:42.048 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:42.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:42.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:42.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:42.192 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:41 smithi029 ceph-mon[29919]: pgmap v79: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:28:42.192 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:41 smithi029 ceph-mon[41500]: pgmap v79: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:28:42.205 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:42.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:42.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:42.336 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:42.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:42.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:41 smithi190 ceph-mon[33748]: pgmap v79: 148 pgs: 148 active+clean; 2.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:28:42.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:42.480 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:42.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:42.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:42.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:42.606 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:42.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:42.675 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 29 16 52143 52127 112.326 72.9375 0.00692982 0.00889707 2024-03-24T12:28:42.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:42.747 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:42.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:42.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:42.890 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:42.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:42.992 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:43.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:43.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:43.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:43.117 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:43.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:43.160 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:43.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:43.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:42 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4068754922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:28:43.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:42 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4068754922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:28:43.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:43.289 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:43.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:43.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:42 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4068754922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:28:43.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:43.429 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:43.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:43.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:43.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:43.545 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:28:43.560 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:28:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:43.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:43.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:43.675 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 30 15 53392 53377 111.186 78.125 0.00517493 0.00898844 2024-03-24T12:28:43.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:43.703 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:28:43.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:43.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:43.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:43.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:43.847 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:43.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:43.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:43.986 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:43.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:44.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:44.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:44.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:44.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:44.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:44.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:44.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:44.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[29919]: pgmap v80: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4068754922' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[29919]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[29919]: 4.7 scrub starts 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[29919]: 6.18 scrub starts 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[29919]: 4.7 scrub ok 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[29919]: 6.18 scrub ok 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[41500]: pgmap v80: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:28:44.359 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:28:44.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:28:44.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4068754922' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:28:44.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[41500]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T12:28:44.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[41500]: 4.7 scrub starts 2024-03-24T12:28:44.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[41500]: 6.18 scrub starts 2024-03-24T12:28:44.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[41500]: 4.7 scrub ok 2024-03-24T12:28:44.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:43 smithi029 ceph-mon[41500]: 6.18 scrub ok 2024-03-24T12:28:44.367 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:44.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:43 smithi190 ceph-mon[33748]: pgmap v80: 148 pgs: 148 active+clean; 2.9 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:28:44.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:43 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:28:44.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:43 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:28:44.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:43 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:28:44.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:43 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4068754922' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:28:44.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:43 smithi190 ceph-mon[33748]: osdmap e103: 8 total, 8 up, 8 in 2024-03-24T12:28:44.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:43 smithi190 ceph-mon[33748]: 4.7 scrub starts 2024-03-24T12:28:44.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:43 smithi190 ceph-mon[33748]: 6.18 scrub starts 2024-03-24T12:28:44.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:43 smithi190 ceph-mon[33748]: 4.7 scrub ok 2024-03-24T12:28:44.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:43 smithi190 ceph-mon[33748]: 6.18 scrub ok 2024-03-24T12:28:44.408 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:44.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:44.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:44.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:44.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:44.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:44.675 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 31 16 54971 54955 110.781 98.625 0.0148309 0.0090212 2024-03-24T12:28:44.684 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:44.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:44.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:44.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:44.840 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:44.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:44.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:44.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:44.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:44.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:44.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:45.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:45.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:45.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:45.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:44 smithi029 ceph-mon[41500]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:28:45.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:44 smithi029 ceph-mon[41500]: 3.10 scrub starts 2024-03-24T12:28:45.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:44 smithi029 ceph-mon[41500]: 3.10 scrub ok 2024-03-24T12:28:45.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:44 smithi029 ceph-mon[29919]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:28:45.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:44 smithi029 ceph-mon[29919]: 3.10 scrub starts 2024-03-24T12:28:45.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:44 smithi029 ceph-mon[29919]: 3.10 scrub ok 2024-03-24T12:28:45.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:45.250 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:45.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:45.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:45.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:45.382 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:45.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:45.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:45.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:44 smithi190 ceph-mon[33748]: from='client.24713 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:28:45.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:44 smithi190 ceph-mon[33748]: 3.10 scrub starts 2024-03-24T12:28:45.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:44 smithi190 ceph-mon[33748]: 3.10 scrub ok 2024-03-24T12:28:45.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:45.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:45.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:45.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:45.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:45.645 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:45.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:45.675 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 32 16 56386 56370 110.081 88.4375 0.00836457 0.00907841 2024-03-24T12:28:45.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:45.841 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:45.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:45.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:45.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:45.957 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-03-24T12:28:42.956387+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3398768990,"num_objects":52050,"num_object_clones":0,"num_object_copies":156150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52050,"num_whiteouts":0,"num_read":1086,"num_read_kb":932,"num_write":52467,"num_write_kb":3320989,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52507,"ondisk_log_size":52507,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":10193532,"kb_used_data":9853884,"kb_used_omap":12,"kb_used_meta":339571,"kb_avail":739767684,"statfs":{"total":767960285184,"available":757522108416,"internally_reserved":0,"allocated":10090377216,"data_stored":10087309578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":347721265},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,1,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1102184448,"num_objects":16818,"num_object_clones":0,"num_object_copies":50454,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16818,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":16818,"num_write_kb":1076352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.550054"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942107+0000","last_change":"2024-03-24T12:27:52.319647+0000","last_active":"2024-03-24T12:28:27.942107+0000","last_peered":"2024-03-24T12:28:27.942107+0000","last_clean":"2024-03-24T12:28:27.942107+0000","last_became_active":"2024-03-24T12:26:09.350521+0000","last_became_peered":"2024-03-24T12:26:09.350521+0000","last_unstale":"2024-03-24T12:28:27.942107+0000","last_undegraded":"2024-03-24T12:28:27.942107+0000","last_fullsized":"2024-03-24T12:28:27.942107+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:52.319515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:52.319515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:00.236194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.944455+0000","last_change":"2024-03-24T12:27:42.444679+0000","last_active":"2024-03-24T12:28:27.944455+0000","last_peered":"2024-03-24T12:28:27.944455+0000","last_clean":"2024-03-24T12:28:27.944455+0000","last_became_active":"2024-03-24T12:26:07.335517+0000","last_became_peered":"2024-03-24T12:26:07.335517+0000","last_unstale":"2024-03-24T12:28:27.944455+0000","last_undegraded":"2024-03-24T12:28:27.944455+0000","last_fullsized":"2024-03-24T12:28:27.944455+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:42.444590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:42.444590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:07.589864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.947651+0000","last_change":"2024-03-24T12:26:03.171110+0000","last_active":"2024-03-24T12:28:27.947651+0000","last_peered":"2024-03-24T12:28:27.947651+0000","last_clean":"2024-03-24T12:28:27.947651+0000","last_became_active":"2024-03-24T12:26:03.170822+0000","last_became_peered":"2024-03-24T12:26:03.170822+0000","last_unstale":"2024-03-24T12:28:27.947651+0000","last_undegraded":"2024-03-24T12:28:27.947651+0000","last_fullsized":"2024-03-24T12:28:27.947651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:26:02.143860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":102,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942027+0000","last_change":"2024-03-24T12:27:24.141489+0000","last_active":"2024-03-24T12:28:27.942027+0000","last_peered":"2024-03-24T12:28:27.942027+0000","last_clean":"2024-03-24T12:28:27.942027+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:28:27.942027+0000","last_undegraded":"2024-03-24T12:28:27.942027+0000","last_fullsized":"2024-03-24T12:28:27.942027+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:24.141325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.141325+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.796727+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940667+0000","last_change":"2024-03-24T12:27:35.494737+0000","last_active":"2024-03-24T12:28:27.940667+0000","last_peered":"2024-03-24T12:28:27.940667+0000","last_clean":"2024-03-24T12:28:27.940667+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:28:27.940667+0000","last_undegraded":"2024-03-24T12:28:27.940667+0000","last_fullsized":"2024-03-24T12:28:27.940667+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:35.494635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:35.494635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:48.670706+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942085+0000","last_change":"2024-03-24T12:27:34.234548+0000","last_active":"2024-03-24T12:28:27.942085+0000","last_peered":"2024-03-24T12:28:27.942085+0000","last_clean":"2024-03-24T12:28:27.942085+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:28:27.942085+0000","last_undegraded":"2024-03-24T12:28:27.942085+0000","last_fullsized":"2024-03-24T12:28:27.942085+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:34.234417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.234417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.166756+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940549+0000","last_change":"2024-03-24T12:27:37.470895+0000","last_active":"2024-03-24T12:28:27.940549+0000","last_peered":"2024-03-24T12:28:27.940549+0000","last_clean":"2024-03-24T12:28:27.940549+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:28:27.940549+0000","last_undegraded":"2024-03-24T12:28:27.940549+0000","last_fullsized":"2024-03-24T12:28:27.940549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:37.470809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:37.470809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:01.645277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"90'22","reported_seq":118,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940188+0000","last_change":"2024-03-24T12:27:25.396689+0000","last_active":"2024-03-24T12:28:27.940188+0000","last_peered":"2024-03-24T12:28:27.940188+0000","last_clean":"2024-03-24T12:28:27.940188+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:28:27.940188+0000","last_undegraded":"2024-03-24T12:28:27.940188+0000","last_fullsized":"2024-03-24T12:28:27.940188+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.396514+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:41.829902+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.952513+0000","last_change":"2024-03-24T12:27:23.289755+0000","last_active":"2024-03-24T12:28:27.952513+0000","last_peered":"2024-03-24T12:28:27.952513+0000","last_clean":"2024-03-24T12:28:27.952513+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:28:27.952513+0000","last_undegraded":"2024-03-24T12:28:27.952513+0000","last_fullsized":"2024-03-24T12:28:27.952513+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.289666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.684236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939281+0000","last_change":"2024-03-24T12:27:44.241377+0000","last_active":"2024-03-24T12:28:27.939281+0000","last_peered":"2024-03-24T12:28:27.939281+0000","last_clean":"2024-03-24T12:28:27.939281+0000","last_became_active":"2024-03-24T12:26:07.331562+0000","last_became_peered":"2024-03-24T12:26:07.331562+0000","last_unstale":"2024-03-24T12:28:27.939281+0000","last_undegraded":"2024-03-24T12:28:27.939281+0000","last_fullsized":"2024-03-24T12:28:27.939281+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:44.241291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:44.241291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:59.176872+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939337+0000","last_change":"2024-03-24T12:27:20.255011+0000","last_active":"2024-03-24T12:28:27.939337+0000","last_peered":"2024-03-24T12:28:27.939337+0000","last_clean":"2024-03-24T12:28:27.939337+0000","last_became_active":"2024-03-24T12:26:03.161544+0000","last_became_peered":"2024-03-24T12:26:03.161544+0000","last_unstale":"2024-03-24T12:28:27.939337+0000","last_undegraded":"2024-03-24T12:28:27.939337+0000","last_fullsized":"2024-03-24T12:28:27.939337+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:20.254898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.254898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.151491+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":102,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942270+0000","last_change":"2024-03-24T12:27:54.323150+0000","last_active":"2024-03-24T12:28:27.942270+0000","last_peered":"2024-03-24T12:28:27.942270+0000","last_clean":"2024-03-24T12:28:27.942270+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:28:27.942270+0000","last_undegraded":"2024-03-24T12:28:27.942270+0000","last_fullsized":"2024-03-24T12:28:27.942270+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:54.323025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:54.323025+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:22.851680+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939015+0000","last_change":"2024-03-24T12:27:21.260071+0000","last_active":"2024-03-24T12:28:27.939015+0000","last_peered":"2024-03-24T12:28:27.939015+0000","last_clean":"2024-03-24T12:28:27.939015+0000","last_became_active":"2024-03-24T12:26:09.350721+0000","last_became_peered":"2024-03-24T12:26:09.350721+0000","last_unstale":"2024-03-24T12:28:27.939015+0000","last_undegraded":"2024-03-24T12:28:27.939015+0000","last_fullsized":"2024-03-24T12:28:27.939015+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.259962+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.937913+0000","last_change":"2024-03-24T12:27:17.365794+0000","last_active":"2024-03-24T12:28:27.937913+0000","last_peered":"2024-03-24T12:28:27.937913+0000","last_clean":"2024-03-24T12:28:27.937913+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:28:27.937913+0000","last_undegraded":"2024-03-24T12:28:27.937913+0000","last_fullsized":"2024-03-24T12:28:27.937913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.365696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.245421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939170+0000","last_change":"2024-03-24T12:27:10.251781+0000","last_active":"2024-03-24T12:28:27.939170+0000","last_peered":"2024-03-24T12:28:27.939170+0000","last_clean":"2024-03-24T12:28:27.939170+0000","last_became_active":"2024-03-24T12:26:03.159997+0000","last_became_peered":"2024-03-24T12:26:03.159997+0000","last_unstale":"2024-03-24T12:28:27.939170+0000","last_undegraded":"2024-03-24T12:28:27.939170+0000","last_fullsized":"2024-03-24T12:28:27.939170+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.251686+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":96,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942665+0000","last_change":"2024-03-24T12:27:38.276194+0000","last_active":"2024-03-24T12:28:27.942665+0000","last_peered":"2024-03-24T12:28:27.942665+0000","last_clean":"2024-03-24T12:28:27.942665+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:28:27.942665+0000","last_undegraded":"2024-03-24T12:28:27.942665+0000","last_fullsized":"2024-03-24T12:28:27.942665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T12:27:38.276034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.276034+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:06.748494+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.943132+0000","last_change":"2024-03-24T12:27:25.164561+0000","last_active":"2024-03-24T12:28:27.943132+0000","last_peered":"2024-03-24T12:28:27.943132+0000","last_clean":"2024-03-24T12:28:27.943132+0000","last_became_active":"2024-03-24T12:26:09.643671+0000","last_became_peered":"2024-03-24T12:26:09.643671+0000","last_unstale":"2024-03-24T12:28:27.943132+0000","last_undegraded":"2024-03-24T12:28:27.943132+0000","last_fullsized":"2024-03-24T12:28:27.943132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:25.164460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.164460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.284324+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.946332+0000","last_change":"2024-03-24T12:27:28.397274+0000","last_active":"2024-03-24T12:28:27.946332+0000","last_peered":"2024-03-24T12:28:27.946332+0000","last_clean":"2024-03-24T12:28:27.946332+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:28:27.946332+0000","last_undegraded":"2024-03-24T12:28:27.946332+0000","last_fullsized":"2024-03-24T12:28:27.946332+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.397179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.397179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.594543+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.947143+0000","last_change":"2024-03-24T12:27:28.361014+0000","last_active":"2024-03-24T12:28:27.947143+0000","last_peered":"2024-03-24T12:28:27.947143+0000","last_clean":"2024-03-24T12:28:27.947143+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:28:27.947143+0000","last_undegraded":"2024-03-24T12:28:27.947143+0000","last_fullsized":"2024-03-24T12:28:27.947143+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.360926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.835122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":111,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.936506+0000","last_change":"2024-03-24T12:27:33.582805+0000","last_active":"2024-03-24T12:28:27.936506+0000","last_peered":"2024-03-24T12:28:27.936506+0000","last_clean":"2024-03-24T12:28:27.936506+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:28:27.936506+0000","last_undegraded":"2024-03-24T12:28:27.936506+0000","last_fullsized":"2024-03-24T12:28:27.936506+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:27:33.582711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:33.582711+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.312118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940797+0000","last_change":"2024-03-24T12:27:30.441572+0000","last_active":"2024-03-24T12:28:27.940797+0000","last_peered":"2024-03-24T12:28:27.940797+0000","last_clean":"2024-03-24T12:28:27.940797+0000","last_became_active":"2024-03-24T12:26:09.643738+0000","last_became_peered":"2024-03-24T12:26:09.643738+0000","last_unstale":"2024-03-24T12:28:27.940797+0000","last_undegraded":"2024-03-24T12:28:27.940797+0000","last_fullsized":"2024-03-24T12:28:27.940797+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:30.441497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:30.441497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.867059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.943200+0000","last_change":"2024-03-24T12:27:39.295946+0000","last_active":"2024-03-24T12:28:27.943200+0000","last_peered":"2024-03-24T12:28:27.943200+0000","last_clean":"2024-03-24T12:28:27.943200+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:28:27.943200+0000","last_undegraded":"2024-03-24T12:28:27.943200+0000","last_fullsized":"2024-03-24T12:28:27.943200+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:27:39.295864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:07.017784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942322+0000","last_change":"2024-03-24T12:28:01.400460+0000","last_active":"2024-03-24T12:28:27.942322+0000","last_peered":"2024-03-24T12:28:27.942322+0000","last_clean":"2024-03-24T12:28:27.942322+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:28:27.942322+0000","last_undegraded":"2024-03-24T12:28:27.942322+0000","last_fullsized":"2024-03-24T12:28:27.942322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:01.400372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:01.400372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:03.221522+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":109,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940179+0000","last_change":"2024-03-24T12:27:32.464080+0000","last_active":"2024-03-24T12:28:27.940179+0000","last_peered":"2024-03-24T12:28:27.940179+0000","last_clean":"2024-03-24T12:28:27.940179+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:28:27.940179+0000","last_undegraded":"2024-03-24T12:28:27.940179+0000","last_fullsized":"2024-03-24T12:28:27.940179+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:32.463972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.463972+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:58.687386+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":79,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938551+0000","last_change":"2024-03-24T12:27:34.409721+0000","last_active":"2024-03-24T12:28:27.938551+0000","last_peered":"2024-03-24T12:28:27.938551+0000","last_clean":"2024-03-24T12:28:27.938551+0000","last_became_active":"2024-03-24T12:26:09.343786+0000","last_became_peered":"2024-03-24T12:26:09.343786+0000","last_unstale":"2024-03-24T12:28:27.938551+0000","last_undegraded":"2024-03-24T12:28:27.938551+0000","last_fullsized":"2024-03-24T12:28:27.938551+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:27:34.409577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.409577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:55.581370+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":125,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:33.237908+0000","last_change":"2024-03-24T12:27:36.723422+0000","last_active":"2024-03-24T12:28:33.237908+0000","last_peered":"2024-03-24T12:28:33.237908+0000","last_clean":"2024-03-24T12:28:33.237908+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:28:33.237908+0000","last_undegraded":"2024-03-24T12:28:33.237908+0000","last_fullsized":"2024-03-24T12:28:33.237908+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:36.723265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:36.723265+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.467176+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":91,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.944543+0000","last_change":"2024-03-24T12:27:27.409032+0000","last_active":"2024-03-24T12:28:27.944543+0000","last_peered":"2024-03-24T12:28:27.944543+0000","last_clean":"2024-03-24T12:28:27.944543+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:28:27.944543+0000","last_undegraded":"2024-03-24T12:28:27.944543+0000","last_fullsized":"2024-03-24T12:28:27.944543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:27.408937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.408937+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.920390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":101,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.952874+0000","last_change":"2024-03-24T12:27:11.269339+0000","last_active":"2024-03-24T12:28:27.952874+0000","last_peered":"2024-03-24T12:28:27.952874+0000","last_clean":"2024-03-24T12:28:27.952874+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:28:27.952874+0000","last_undegraded":"2024-03-24T12:28:27.952874+0000","last_fullsized":"2024-03-24T12:28:27.952874+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:11.269181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:11.269181+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:34.970841+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.946566+0000","last_change":"2024-03-24T12:27:33.377434+0000","last_active":"2024-03-24T12:28:27.946566+0000","last_peered":"2024-03-24T12:28:27.946566+0000","last_clean":"2024-03-24T12:28:27.946566+0000","last_became_active":"2024-03-24T12:26:09.348355+0000","last_became_peered":"2024-03-24T12:26:09.348355+0000","last_unstale":"2024-03-24T12:28:27.946566+0000","last_undegraded":"2024-03-24T12:28:27.946566+0000","last_fullsized":"2024-03-24T12:28:27.946566+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:33.377351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:33.377351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.631210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":74,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.948685+0000","last_change":"2024-03-24T12:26:07.331932+0000","last_active":"2024-03-24T12:28:27.948685+0000","last_peered":"2024-03-24T12:28:27.948685+0000","last_clean":"2024-03-24T12:28:27.948685+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:28:27.948685+0000","last_undegraded":"2024-03-24T12:28:27.948685+0000","last_fullsized":"2024-03-24T12:28:27.948685+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:26:06.312270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939114+0000","last_change":"2024-03-24T12:27:12.437790+0000","last_active":"2024-03-24T12:28:27.939114+0000","last_peered":"2024-03-24T12:28:27.939114+0000","last_clean":"2024-03-24T12:28:27.939114+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:28:27.939114+0000","last_undegraded":"2024-03-24T12:28:27.939114+0000","last_fullsized":"2024-03-24T12:28:27.939114+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.437708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":105,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.936645+0000","last_change":"2024-03-24T12:27:26.763428+0000","last_active":"2024-03-24T12:28:27.936645+0000","last_peered":"2024-03-24T12:28:27.936645+0000","last_clean":"2024-03-24T12:28:27.936645+0000","last_became_active":"2024-03-24T12:26:05.317587+0000","last_became_peered":"2024-03-24T12:26:05.317587+0000","last_unstale":"2024-03-24T12:28:27.936645+0000","last_undegraded":"2024-03-24T12:28:27.936645+0000","last_fullsized":"2024-03-24T12:28:27.936645+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:26.763317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.763317+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.728888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.937815+0000","last_change":"2024-03-24T12:27:55.392212+0000","last_active":"2024-03-24T12:28:27.937815+0000","last_peered":"2024-03-24T12:28:27.937815+0000","last_clean":"2024-03-24T12:28:27.937815+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:28:27.937815+0000","last_undegraded":"2024-03-24T12:28:27.937815+0000","last_fullsized":"2024-03-24T12:28:27.937815+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:55.392098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:55.392098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:59.885596+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.936275+0000","last_change":"2024-03-24T12:27:34.747936+0000","last_active":"2024-03-24T12:28:27.936275+0000","last_peered":"2024-03-24T12:28:27.936275+0000","last_clean":"2024-03-24T12:28:27.936275+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:28:27.936275+0000","last_undegraded":"2024-03-24T12:28:27.936275+0000","last_fullsized":"2024-03-24T12:28:27.936275+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:34.747834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.747834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:50.783426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.951067+0000","last_change":"2024-03-24T12:27:59.975825+0000","last_active":"2024-03-24T12:28:27.951067+0000","last_peered":"2024-03-24T12:28:27.951067+0000","last_clean":"2024-03-24T12:28:27.951067+0000","last_became_active":"2024-03-24T12:26:03.170395+0000","last_became_peered":"2024-03-24T12:26:03.170395+0000","last_unstale":"2024-03-24T12:28:27.951067+0000","last_undegraded":"2024-03-24T12:28:27.951067+0000","last_fullsized":"2024-03-24T12:28:27.951067+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:59.975745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:59.975745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:25.093817+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":92,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939210+0000","last_change":"2024-03-24T12:27:15.467117+0000","last_active":"2024-03-24T12:28:27.939210+0000","last_peered":"2024-03-24T12:28:27.939210+0000","last_clean":"2024-03-24T12:28:27.939210+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:28:27.939210+0000","last_undegraded":"2024-03-24T12:28:27.939210+0000","last_fullsized":"2024-03-24T12:28:27.939210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:27:15.466966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:15.466966+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.332878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940883+0000","last_change":"2024-03-24T12:27:38.510336+0000","last_active":"2024-03-24T12:28:27.940883+0000","last_peered":"2024-03-24T12:28:27.940883+0000","last_clean":"2024-03-24T12:28:27.940883+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:28:27.940883+0000","last_undegraded":"2024-03-24T12:28:27.940883+0000","last_fullsized":"2024-03-24T12:28:27.940883+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:38.510237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.510237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.100244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938302+0000","last_change":"2024-03-24T12:27:21.335557+0000","last_active":"2024-03-24T12:28:27.938302+0000","last_peered":"2024-03-24T12:28:27.938302+0000","last_clean":"2024-03-24T12:28:27.938302+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:28:27.938302+0000","last_undegraded":"2024-03-24T12:28:27.938302+0000","last_fullsized":"2024-03-24T12:28:27.938302+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:21.335455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.335455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.789829+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":96,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939862+0000","last_change":"2024-03-24T12:27:34.193568+0000","last_active":"2024-03-24T12:28:27.939862+0000","last_peered":"2024-03-24T12:28:27.939862+0000","last_clean":"2024-03-24T12:28:27.939862+0000","last_became_active":"2024-03-24T12:26:03.165083+0000","last_became_peered":"2024-03-24T12:26:03.165083+0000","last_unstale":"2024-03-24T12:28:27.939862+0000","last_undegraded":"2024-03-24T12:28:27.939862+0000","last_fullsized":"2024-03-24T12:28:27.939862+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:34.193480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.193480+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:50.648254+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":105,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940042+0000","last_change":"2024-03-24T12:27:16.497904+0000","last_active":"2024-03-24T12:28:27.940042+0000","last_peered":"2024-03-24T12:28:27.940042+0000","last_clean":"2024-03-24T12:28:27.940042+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:28:27.940042+0000","last_undegraded":"2024-03-24T12:28:27.940042+0000","last_fullsized":"2024-03-24T12:28:27.940042+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:16.497755+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.497755+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.894323+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.952251+0000","last_change":"2024-03-24T12:27:53.059036+0000","last_active":"2024-03-24T12:28:27.952251+0000","last_peered":"2024-03-24T12:28:27.952251+0000","last_clean":"2024-03-24T12:28:27.952251+0000","last_became_active":"2024-03-24T12:26:09.350656+0000","last_became_peered":"2024-03-24T12:26:09.350656+0000","last_unstale":"2024-03-24T12:28:27.952251+0000","last_undegraded":"2024-03-24T12:28:27.952251+0000","last_fullsized":"2024-03-24T12:28:27.952251+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:27:53.058946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:08.012714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940528+0000","last_change":"2024-03-24T12:27:20.458514+0000","last_active":"2024-03-24T12:28:27.940528+0000","last_peered":"2024-03-24T12:28:27.940528+0000","last_clean":"2024-03-24T12:28:27.940528+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:28:27.940528+0000","last_undegraded":"2024-03-24T12:28:27.940528+0000","last_fullsized":"2024-03-24T12:28:27.940528+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.458419+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938585+0000","last_change":"2024-03-24T12:27:14.329516+0000","last_active":"2024-03-24T12:28:27.938585+0000","last_peered":"2024-03-24T12:28:27.938585+0000","last_clean":"2024-03-24T12:28:27.938585+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:28:27.938585+0000","last_undegraded":"2024-03-24T12:28:27.938585+0000","last_fullsized":"2024-03-24T12:28:27.938585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:14.329357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:14.329357+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":80,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939094+0000","last_change":"2024-03-24T12:27:43.194108+0000","last_active":"2024-03-24T12:28:27.939094+0000","last_peered":"2024-03-24T12:28:27.939094+0000","last_clean":"2024-03-24T12:28:27.939094+0000","last_became_active":"2024-03-24T12:26:16.874170+0000","last_became_peered":"2024-03-24T12:26:16.874170+0000","last_unstale":"2024-03-24T12:28:27.939094+0000","last_undegraded":"2024-03-24T12:28:27.939094+0000","last_fullsized":"2024-03-24T12:28:27.939094+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:43.194004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:43.194004+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:12.704658+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939760+0000","last_change":"2024-03-24T12:27:26.431474+0000","last_active":"2024-03-24T12:28:27.939760+0000","last_peered":"2024-03-24T12:28:27.939760+0000","last_clean":"2024-03-24T12:28:27.939760+0000","last_became_active":"2024-03-24T12:26:09.343296+0000","last_became_peered":"2024-03-24T12:26:09.343296+0000","last_unstale":"2024-03-24T12:28:27.939760+0000","last_undegraded":"2024-03-24T12:28:27.939760+0000","last_fullsized":"2024-03-24T12:28:27.939760+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.431382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.685011+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.945224+0000","last_change":"2024-03-24T12:27:29.397575+0000","last_active":"2024-03-24T12:28:27.945224+0000","last_peered":"2024-03-24T12:28:27.945224+0000","last_clean":"2024-03-24T12:28:27.945224+0000","last_became_active":"2024-03-24T12:26:07.330811+0000","last_became_peered":"2024-03-24T12:26:07.330811+0000","last_unstale":"2024-03-24T12:28:27.945224+0000","last_undegraded":"2024-03-24T12:28:27.945224+0000","last_fullsized":"2024-03-24T12:28:27.945224+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:29.397463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:29.397463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:52.953466+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942959+0000","last_change":"2024-03-24T12:27:09.137207+0000","last_active":"2024-03-24T12:28:27.942959+0000","last_peered":"2024-03-24T12:28:27.942959+0000","last_clean":"2024-03-24T12:28:27.942959+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:28:27.942959+0000","last_undegraded":"2024-03-24T12:28:27.942959+0000","last_fullsized":"2024-03-24T12:28:27.942959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:09.137104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:09.137104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.329586+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"88'16","reported_seq":110,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942892+0000","last_change":"2024-03-24T12:27:17.084272+0000","last_active":"2024-03-24T12:28:27.942892+0000","last_peered":"2024-03-24T12:28:27.942892+0000","last_clean":"2024-03-24T12:28:27.942892+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:28:27.942892+0000","last_undegraded":"2024-03-24T12:28:27.942892+0000","last_fullsized":"2024-03-24T12:28:27.942892+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.084148+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.541132+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":71,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.947458+0000","last_change":"2024-03-24T12:26:09.349851+0000","last_active":"2024-03-24T12:28:27.947458+0000","last_peered":"2024-03-24T12:28:27.947458+0000","last_clean":"2024-03-24T12:28:27.947458+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:28:27.947458+0000","last_undegraded":"2024-03-24T12:28:27.947458+0000","last_fullsized":"2024-03-24T12:28:27.947458+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.997763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":122,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:33.237678+0000","last_change":"2024-03-24T12:27:46.279895+0000","last_active":"2024-03-24T12:28:33.237678+0000","last_peered":"2024-03-24T12:28:33.237678+0000","last_clean":"2024-03-24T12:28:33.237678+0000","last_became_active":"2024-03-24T12:26:07.329487+0000","last_became_peered":"2024-03-24T12:26:07.329487+0000","last_unstale":"2024-03-24T12:28:33.237678+0000","last_undegraded":"2024-03-24T12:28:33.237678+0000","last_fullsized":"2024-03-24T12:28:33.237678+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:46.279789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:46.279789+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:04.673451+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.937883+0000","last_change":"2024-03-24T12:27:49.334495+0000","last_active":"2024-03-24T12:28:27.937883+0000","last_peered":"2024-03-24T12:28:27.937883+0000","last_clean":"2024-03-24T12:28:27.937883+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:28:27.937883+0000","last_undegraded":"2024-03-24T12:28:27.937883+0000","last_fullsized":"2024-03-24T12:28:27.937883+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:49.334399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:49.334399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:09.196771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":101,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939927+0000","last_change":"2024-03-24T12:27:21.451811+0000","last_active":"2024-03-24T12:28:27.939927+0000","last_peered":"2024-03-24T12:28:27.939927+0000","last_clean":"2024-03-24T12:28:27.939927+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:28:27.939927+0000","last_undegraded":"2024-03-24T12:28:27.939927+0000","last_fullsized":"2024-03-24T12:28:27.939927+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:21.451717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.451717+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.819749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942856+0000","last_change":"2024-03-24T12:27:37.276957+0000","last_active":"2024-03-24T12:28:27.942856+0000","last_peered":"2024-03-24T12:28:27.942856+0000","last_clean":"2024-03-24T12:28:27.942856+0000","last_became_active":"2024-03-24T12:26:09.346062+0000","last_became_peered":"2024-03-24T12:26:09.346062+0000","last_unstale":"2024-03-24T12:28:27.942856+0000","last_undegraded":"2024-03-24T12:28:27.942856+0000","last_fullsized":"2024-03-24T12:28:27.942856+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:37.276875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:37.276875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.339012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"93'9","reported_seq":123,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:33.237964+0000","last_change":"2024-03-24T12:27:51.379024+0000","last_active":"2024-03-24T12:28:33.237964+0000","last_peered":"2024-03-24T12:28:33.237964+0000","last_clean":"2024-03-24T12:28:33.237964+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:28:33.237964+0000","last_undegraded":"2024-03-24T12:28:33.237964+0000","last_fullsized":"2024-03-24T12:28:33.237964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'9","last_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:27:51.378939+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.971149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.947873+0000","last_change":"2024-03-24T12:27:07.317120+0000","last_active":"2024-03-24T12:28:27.947873+0000","last_peered":"2024-03-24T12:28:27.947873+0000","last_clean":"2024-03-24T12:28:27.947873+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:28:27.947873+0000","last_undegraded":"2024-03-24T12:28:27.947873+0000","last_fullsized":"2024-03-24T12:28:27.947873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:07.317021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:07.317021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.086017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'13","reported_seq":106,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939561+0000","last_change":"2024-03-24T12:27:16.242881+0000","last_active":"2024-03-24T12:28:27.939561+0000","last_peered":"2024-03-24T12:28:27.939561+0000","last_clean":"2024-03-24T12:28:27.939561+0000","last_became_active":"2024-03-24T12:26:05.358213+0000","last_became_peered":"2024-03-24T12:26:05.358213+0000","last_unstale":"2024-03-24T12:28:27.939561+0000","last_undegraded":"2024-03-24T12:28:27.939561+0000","last_fullsized":"2024-03-24T12:28:27.939561+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.242776+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.276034+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.936782+0000","last_change":"2024-03-24T12:27:27.661082+0000","last_active":"2024-03-24T12:28:27.936782+0000","last_peered":"2024-03-24T12:28:27.936782+0000","last_clean":"2024-03-24T12:28:27.936782+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:28:27.936782+0000","last_undegraded":"2024-03-24T12:28:27.936782+0000","last_fullsized":"2024-03-24T12:28:27.936782+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.660932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.997979+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940698+0000","last_change":"2024-03-24T12:27:39.473015+0000","last_active":"2024-03-24T12:28:27.940698+0000","last_peered":"2024-03-24T12:28:27.940698+0000","last_clean":"2024-03-24T12:28:27.940698+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:28:27.940698+0000","last_undegraded":"2024-03-24T12:28:27.940698+0000","last_fullsized":"2024-03-24T12:28:27.940698+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:39.472900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:39.472900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:04.877211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.937919+0000","last_change":"2024-03-24T12:27:06.422517+0000","last_active":"2024-03-24T12:28:27.937919+0000","last_peered":"2024-03-24T12:28:27.937919+0000","last_clean":"2024-03-24T12:28:27.937919+0000","last_became_active":"2024-03-24T12:26:03.166149+0000","last_became_peered":"2024-03-24T12:26:03.166149+0000","last_unstale":"2024-03-24T12:28:27.937919+0000","last_undegraded":"2024-03-24T12:28:27.937919+0000","last_fullsized":"2024-03-24T12:28:27.937919+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:06.422387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.422387+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.349386+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":102,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.948570+0000","last_change":"2024-03-24T12:27:30.384123+0000","last_active":"2024-03-24T12:28:27.948570+0000","last_peered":"2024-03-24T12:28:27.948570+0000","last_clean":"2024-03-24T12:28:27.948570+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:28:27.948570+0000","last_undegraded":"2024-03-24T12:28:27.948570+0000","last_fullsized":"2024-03-24T12:28:27.948570+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:30.384035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:30.384035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:48.792180+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938532+0000","last_change":"2024-03-24T12:27:42.333953+0000","last_active":"2024-03-24T12:28:27.938532+0000","last_peered":"2024-03-24T12:28:27.938532+0000","last_clean":"2024-03-24T12:28:27.938532+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:28:27.938532+0000","last_undegraded":"2024-03-24T12:28:27.938532+0000","last_fullsized":"2024-03-24T12:28:27.938532+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:42.333813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:42.333813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:01.854155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939921+0000","last_change":"2024-03-24T12:27:47.315049+0000","last_active":"2024-03-24T12:28:27.939921+0000","last_peered":"2024-03-24T12:28:27.939921+0000","last_clean":"2024-03-24T12:28:27.939921+0000","last_became_active":"2024-03-24T12:26:07.329117+0000","last_became_peered":"2024-03-24T12:26:07.329117+0000","last_unstale":"2024-03-24T12:28:27.939921+0000","last_undegraded":"2024-03-24T12:28:27.939921+0000","last_fullsized":"2024-03-24T12:28:27.939921+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:47.314935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:47.314935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:10.044132+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.935387+0000","last_change":"2024-03-24T12:27:06.584912+0000","last_active":"2024-03-24T12:28:27.935387+0000","last_peered":"2024-03-24T12:28:27.935387+0000","last_clean":"2024-03-24T12:28:27.935387+0000","last_became_active":"2024-03-24T12:26:03.161414+0000","last_became_peered":"2024-03-24T12:26:03.161414+0000","last_unstale":"2024-03-24T12:28:27.935387+0000","last_undegraded":"2024-03-24T12:28:27.935387+0000","last_fullsized":"2024-03-24T12:28:27.935387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:06.584809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.584809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:29.634126+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":95,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940938+0000","last_change":"2024-03-24T12:27:50.392580+0000","last_active":"2024-03-24T12:28:27.940938+0000","last_peered":"2024-03-24T12:28:27.940938+0000","last_clean":"2024-03-24T12:28:27.940938+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:28:27.940938+0000","last_undegraded":"2024-03-24T12:28:27.940938+0000","last_fullsized":"2024-03-24T12:28:27.940938+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T12:27:50.392486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:50.392486+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.667306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"102'3267","reported_seq":3291,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.365690+0000","last_change":"2024-03-24T12:28:05.793736+0000","last_active":"2024-03-24T12:28:42.365690+0000","last_peered":"2024-03-24T12:28:42.365690+0000","last_clean":"2024-03-24T12:28:42.365690+0000","last_became_active":"2024-03-24T12:28:05.793448+0000","last_became_peered":"2024-03-24T12:28:05.793448+0000","last_unstale":"2024-03-24T12:28:42.365690+0000","last_undegraded":"2024-03-24T12:28:42.365690+0000","last_fullsized":"2024-03-24T12:28:42.365690+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3267,"log_dups_size":0,"ondisk_log_size":3267,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:34.477118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214106112,"num_objects":3267,"num_object_clones":0,"num_object_copies":9801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3267,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3267,"num_write_kb":209088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939399+0000","last_change":"2024-03-24T12:27:52.239942+0000","last_active":"2024-03-24T12:28:27.939399+0000","last_peered":"2024-03-24T12:28:27.939399+0000","last_clean":"2024-03-24T12:28:27.939399+0000","last_became_active":"2024-03-24T12:26:09.643602+0000","last_became_peered":"2024-03-24T12:26:09.643602+0000","last_unstale":"2024-03-24T12:28:27.939399+0000","last_undegraded":"2024-03-24T12:28:27.939399+0000","last_fullsized":"2024-03-24T12:28:27.939399+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:52.239824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:52.239824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:59.157635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":102,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938764+0000","last_change":"2024-03-24T12:27:45.299862+0000","last_active":"2024-03-24T12:28:27.938764+0000","last_peered":"2024-03-24T12:28:27.938764+0000","last_clean":"2024-03-24T12:28:27.938764+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:28:27.938764+0000","last_undegraded":"2024-03-24T12:28:27.938764+0000","last_fullsized":"2024-03-24T12:28:27.938764+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:27:45.299747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:45.299747+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.791382+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":111,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939142+0000","last_change":"2024-03-24T12:27:17.288726+0000","last_active":"2024-03-24T12:28:27.939142+0000","last_peered":"2024-03-24T12:28:27.939142+0000","last_clean":"2024-03-24T12:28:27.939142+0000","last_became_active":"2024-03-24T12:26:05.358217+0000","last_became_peered":"2024-03-24T12:26:05.358217+0000","last_unstale":"2024-03-24T12:28:27.939142+0000","last_undegraded":"2024-03-24T12:28:27.939142+0000","last_fullsized":"2024-03-24T12:28:27.939142+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:27:17.288616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:17.288616+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:31.276817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":125,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:33.238053+0000","last_change":"2024-03-24T12:27:20.354464+0000","last_active":"2024-03-24T12:28:33.238053+0000","last_peered":"2024-03-24T12:28:33.238053+0000","last_clean":"2024-03-24T12:28:33.238053+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:28:33.238053+0000","last_undegraded":"2024-03-24T12:28:33.238053+0000","last_fullsized":"2024-03-24T12:28:33.238053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:20.354367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.354367+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.245481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"102'3166","reported_seq":3189,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.405003+0000","last_change":"2024-03-24T12:28:05.792920+0000","last_active":"2024-03-24T12:28:42.405003+0000","last_peered":"2024-03-24T12:28:42.405003+0000","last_clean":"2024-03-24T12:28:42.405003+0000","last_became_active":"2024-03-24T12:28:05.792457+0000","last_became_peered":"2024-03-24T12:28:05.792457+0000","last_unstale":"2024-03-24T12:28:42.405003+0000","last_undegraded":"2024-03-24T12:28:42.405003+0000","last_fullsized":"2024-03-24T12:28:42.405003+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3166,"log_dups_size":0,"ondisk_log_size":3166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:34.211693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207486976,"num_objects":3166,"num_object_clones":0,"num_object_copies":9498,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3166,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3166,"num_write_kb":202624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.941145+0000","last_change":"2024-03-24T12:27:52.426045+0000","last_active":"2024-03-24T12:28:27.941145+0000","last_peered":"2024-03-24T12:28:27.941145+0000","last_clean":"2024-03-24T12:28:27.941145+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:28:27.941145+0000","last_undegraded":"2024-03-24T12:28:27.941145+0000","last_fullsized":"2024-03-24T12:28:27.941145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:52.425959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:52.425959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:11.402231+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938965+0000","last_change":"2024-03-24T12:27:51.367179+0000","last_active":"2024-03-24T12:28:27.938965+0000","last_peered":"2024-03-24T12:28:27.938965+0000","last_clean":"2024-03-24T12:28:27.938965+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:28:27.938965+0000","last_undegraded":"2024-03-24T12:28:27.938965+0000","last_fullsized":"2024-03-24T12:28:27.938965+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:51.367099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:51.367099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:58.739439+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":109,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942406+0000","last_change":"2024-03-24T12:27:23.131685+0000","last_active":"2024-03-24T12:28:27.942406+0000","last_peered":"2024-03-24T12:28:27.942406+0000","last_clean":"2024-03-24T12:28:27.942406+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:28:27.942406+0000","last_undegraded":"2024-03-24T12:28:27.942406+0000","last_fullsized":"2024-03-24T12:28:27.942406+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:23.131573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.131573+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.851864+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.935751+0000","last_change":"2024-03-24T12:27:35.581463+0000","last_active":"2024-03-24T12:28:27.935751+0000","last_peered":"2024-03-24T12:28:27.935751+0000","last_clean":"2024-03-24T12:28:27.935751+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:28:27.935751+0000","last_undegraded":"2024-03-24T12:28:27.935751+0000","last_fullsized":"2024-03-24T12:28:27.935751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:35.581370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:35.581370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.191248+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"102'3215","reported_seq":3239,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.437991+0000","last_change":"2024-03-24T12:28:05.794720+0000","last_active":"2024-03-24T12:28:42.437991+0000","last_peered":"2024-03-24T12:28:42.437991+0000","last_clean":"2024-03-24T12:28:42.437991+0000","last_became_active":"2024-03-24T12:28:05.794429+0000","last_became_peered":"2024-03-24T12:28:05.794429+0000","last_unstale":"2024-03-24T12:28:42.437991+0000","last_undegraded":"2024-03-24T12:28:42.437991+0000","last_fullsized":"2024-03-24T12:28:42.437991+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3215,"log_dups_size":0,"ondisk_log_size":3215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:27.699549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":210698240,"num_objects":3215,"num_object_clones":0,"num_object_copies":9645,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3215,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3215,"num_write_kb":205760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.937719+0000","last_change":"2024-03-24T12:27:53.392878+0000","last_active":"2024-03-24T12:28:27.937719+0000","last_peered":"2024-03-24T12:28:27.937719+0000","last_clean":"2024-03-24T12:28:27.937719+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:28:27.937719+0000","last_undegraded":"2024-03-24T12:28:27.937719+0000","last_fullsized":"2024-03-24T12:28:27.937719+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:53.392787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:53.392787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:11.227342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.937002+0000","last_change":"2024-03-24T12:27:32.548089+0000","last_active":"2024-03-24T12:28:27.937002+0000","last_peered":"2024-03-24T12:28:27.937002+0000","last_clean":"2024-03-24T12:28:27.937002+0000","last_became_active":"2024-03-24T12:26:07.331416+0000","last_became_peered":"2024-03-24T12:26:07.331416+0000","last_unstale":"2024-03-24T12:28:27.937002+0000","last_undegraded":"2024-03-24T12:28:27.937002+0000","last_fullsized":"2024-03-24T12:28:27.937002+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.547951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.547951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.882159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938786+0000","last_change":"2024-03-24T12:27:37.366770+0000","last_active":"2024-03-24T12:28:27.938786+0000","last_peered":"2024-03-24T12:28:27.938786+0000","last_clean":"2024-03-24T12:28:27.938786+0000","last_became_active":"2024-03-24T12:26:03.165214+0000","last_became_peered":"2024-03-24T12:26:03.165214+0000","last_unstale":"2024-03-24T12:28:27.938786+0000","last_undegraded":"2024-03-24T12:28:27.938786+0000","last_fullsized":"2024-03-24T12:28:27.938786+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:37.366618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:37.366618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.039598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":117,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.936005+0000","last_change":"2024-03-24T12:27:15.668713+0000","last_active":"2024-03-24T12:28:27.936005+0000","last_peered":"2024-03-24T12:28:27.936005+0000","last_clean":"2024-03-24T12:28:27.936005+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:28:27.936005+0000","last_undegraded":"2024-03-24T12:28:27.936005+0000","last_fullsized":"2024-03-24T12:28:27.936005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:27:15.668556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:15.668556+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:43.755371+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"102'3250","reported_seq":3274,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.411963+0000","last_change":"2024-03-24T12:28:05.794676+0000","last_active":"2024-03-24T12:28:42.411963+0000","last_peered":"2024-03-24T12:28:42.411963+0000","last_clean":"2024-03-24T12:28:42.411963+0000","last_became_active":"2024-03-24T12:28:05.794375+0000","last_became_peered":"2024-03-24T12:28:05.794375+0000","last_unstale":"2024-03-24T12:28:42.411963+0000","last_undegraded":"2024-03-24T12:28:42.411963+0000","last_fullsized":"2024-03-24T12:28:42.411963+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3250,"log_dups_size":0,"ondisk_log_size":3250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:06.608478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":212992000,"num_objects":3250,"num_object_clones":0,"num_object_copies":9750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3250,"num_write_kb":208000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.947104+0000","last_change":"2024-03-24T12:26:09.350031+0000","last_active":"2024-03-24T12:28:27.947104+0000","last_peered":"2024-03-24T12:28:27.947104+0000","last_clean":"2024-03-24T12:28:27.947104+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:28:27.947104+0000","last_undegraded":"2024-03-24T12:28:27.947104+0000","last_fullsized":"2024-03-24T12:28:27.947104+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.937324+0000","last_change":"2024-03-24T12:27:38.596223+0000","last_active":"2024-03-24T12:28:27.937324+0000","last_peered":"2024-03-24T12:28:27.937324+0000","last_clean":"2024-03-24T12:28:27.937324+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:28:27.937324+0000","last_undegraded":"2024-03-24T12:28:27.937324+0000","last_fullsized":"2024-03-24T12:28:27.937324+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.596133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:52.862429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":97,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939223+0000","last_change":"2024-03-24T12:27:09.234408+0000","last_active":"2024-03-24T12:28:27.939223+0000","last_peered":"2024-03-24T12:28:27.939223+0000","last_clean":"2024-03-24T12:28:27.939223+0000","last_became_active":"2024-03-24T12:26:03.162562+0000","last_became_peered":"2024-03-24T12:26:03.162562+0000","last_unstale":"2024-03-24T12:28:27.939223+0000","last_undegraded":"2024-03-24T12:28:27.939223+0000","last_fullsized":"2024-03-24T12:28:27.939223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:27:09.234299+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":110,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.946895+0000","last_change":"2024-03-24T12:26:16.882489+0000","last_active":"2024-03-24T12:28:27.946895+0000","last_peered":"2024-03-24T12:28:27.946895+0000","last_clean":"2024-03-24T12:28:27.946895+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:28:27.946895+0000","last_undegraded":"2024-03-24T12:28:27.946895+0000","last_fullsized":"2024-03-24T12:28:27.946895+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"102'3160","reported_seq":3184,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.384606+0000","last_change":"2024-03-24T12:28:05.794739+0000","last_active":"2024-03-24T12:28:42.384606+0000","last_peered":"2024-03-24T12:28:42.384606+0000","last_clean":"2024-03-24T12:28:42.384606+0000","last_became_active":"2024-03-24T12:28:05.794459+0000","last_became_peered":"2024-03-24T12:28:05.794459+0000","last_unstale":"2024-03-24T12:28:42.384606+0000","last_undegraded":"2024-03-24T12:28:42.384606+0000","last_fullsized":"2024-03-24T12:28:42.384606+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3160,"log_dups_size":0,"ondisk_log_size":3160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:34.241798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207093760,"num_objects":3160,"num_object_clones":0,"num_object_copies":9480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3160,"num_write_kb":202240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940754+0000","last_change":"2024-03-24T12:27:24.439386+0000","last_active":"2024-03-24T12:28:27.940754+0000","last_peered":"2024-03-24T12:28:27.940754+0000","last_clean":"2024-03-24T12:28:27.940754+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:28:27.940754+0000","last_undegraded":"2024-03-24T12:28:27.940754+0000","last_fullsized":"2024-03-24T12:28:27.940754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.439296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.495934+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":99,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939209+0000","last_change":"2024-03-24T12:27:51.237927+0000","last_active":"2024-03-24T12:28:27.939209+0000","last_peered":"2024-03-24T12:28:27.939209+0000","last_clean":"2024-03-24T12:28:27.939209+0000","last_became_active":"2024-03-24T12:26:03.165771+0000","last_became_peered":"2024-03-24T12:26:03.165771+0000","last_unstale":"2024-03-24T12:28:27.939209+0000","last_undegraded":"2024-03-24T12:28:27.939209+0000","last_fullsized":"2024-03-24T12:28:27.939209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:27:51.237812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:51.237812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:14.735355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942789+0000","last_change":"2024-03-24T12:27:32.227062+0000","last_active":"2024-03-24T12:28:27.942789+0000","last_peered":"2024-03-24T12:28:27.942789+0000","last_clean":"2024-03-24T12:28:27.942789+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:28:27.942789+0000","last_undegraded":"2024-03-24T12:28:27.942789+0000","last_fullsized":"2024-03-24T12:28:27.942789+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.226982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.226982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.516101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"90'38","reported_seq":176,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:33.240145+0000","last_change":"2024-03-24T12:27:31.375798+0000","last_active":"2024-03-24T12:28:33.240145+0000","last_peered":"2024-03-24T12:28:33.240145+0000","last_clean":"2024-03-24T12:28:33.240145+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:28:33.240145+0000","last_undegraded":"2024-03-24T12:28:33.240145+0000","last_fullsized":"2024-03-24T12:28:33.240145+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'38","last_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:27:31.375629+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:55.069683+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.950223+0000","last_change":"2024-03-24T12:27:55.077766+0000","last_active":"2024-03-24T12:28:27.950223+0000","last_peered":"2024-03-24T12:28:27.950223+0000","last_clean":"2024-03-24T12:28:27.950223+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:28:27.950223+0000","last_undegraded":"2024-03-24T12:28:27.950223+0000","last_fullsized":"2024-03-24T12:28:27.950223+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:55.077683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:55.077683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:58.507582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"102'3277","reported_seq":3301,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.417299+0000","last_change":"2024-03-24T12:28:05.796673+0000","last_active":"2024-03-24T12:28:42.417299+0000","last_peered":"2024-03-24T12:28:42.417299+0000","last_clean":"2024-03-24T12:28:42.417299+0000","last_became_active":"2024-03-24T12:28:05.796293+0000","last_became_peered":"2024-03-24T12:28:05.796293+0000","last_unstale":"2024-03-24T12:28:42.417299+0000","last_undegraded":"2024-03-24T12:28:42.417299+0000","last_fullsized":"2024-03-24T12:28:42.417299+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:08.087553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214761472,"num_objects":3277,"num_object_clones":0,"num_object_copies":9831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3277,"num_write_kb":209728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938861+0000","last_change":"2024-03-24T12:27:18.336245+0000","last_active":"2024-03-24T12:28:27.938861+0000","last_peered":"2024-03-24T12:28:27.938861+0000","last_clean":"2024-03-24T12:28:27.938861+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:28:27.938861+0000","last_undegraded":"2024-03-24T12:28:27.938861+0000","last_fullsized":"2024-03-24T12:28:27.938861+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:18.336147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:18.336147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:37.167902+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.944933+0000","last_change":"2024-03-24T12:27:16.330188+0000","last_active":"2024-03-24T12:28:27.944933+0000","last_peered":"2024-03-24T12:28:27.944933+0000","last_clean":"2024-03-24T12:28:27.944933+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:28:27.944933+0000","last_undegraded":"2024-03-24T12:28:27.944933+0000","last_fullsized":"2024-03-24T12:28:27.944933+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:16.330050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.330050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:35.775200+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.936108+0000","last_change":"2024-03-24T12:27:24.614983+0000","last_active":"2024-03-24T12:28:27.936108+0000","last_peered":"2024-03-24T12:28:27.936108+0000","last_clean":"2024-03-24T12:28:27.936108+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:28:27.936108+0000","last_undegraded":"2024-03-24T12:28:27.936108+0000","last_fullsized":"2024-03-24T12:28:27.936108+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.614896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.614896+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.687839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":118,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.944250+0000","last_change":"2024-03-24T12:27:06.412196+0000","last_active":"2024-03-24T12:28:27.944250+0000","last_peered":"2024-03-24T12:28:27.944250+0000","last_clean":"2024-03-24T12:28:27.944250+0000","last_became_active":"2024-03-24T12:26:05.315639+0000","last_became_peered":"2024-03-24T12:26:05.315639+0000","last_unstale":"2024-03-24T12:28:27.944250+0000","last_undegraded":"2024-03-24T12:28:27.944250+0000","last_fullsized":"2024-03-24T12:28:27.944250+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:27:06.412074+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:27:06.412074+0000","last_clean_scrub_stamp":"2024-03-24T12:27:06.412074+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.735235+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.936356+0000","last_change":"2024-03-24T12:27:19.658489+0000","last_active":"2024-03-24T12:28:27.936356+0000","last_peered":"2024-03-24T12:28:27.936356+0000","last_clean":"2024-03-24T12:28:27.936356+0000","last_became_active":"2024-03-24T12:26:07.343010+0000","last_became_peered":"2024-03-24T12:26:07.343010+0000","last_unstale":"2024-03-24T12:28:27.936356+0000","last_undegraded":"2024-03-24T12:28:27.936356+0000","last_fullsized":"2024-03-24T12:28:27.936356+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:19.658367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:19.658367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.033169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"102'3273","reported_seq":3297,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.434542+0000","last_change":"2024-03-24T12:28:05.798207+0000","last_active":"2024-03-24T12:28:42.434542+0000","last_peered":"2024-03-24T12:28:42.434542+0000","last_clean":"2024-03-24T12:28:42.434542+0000","last_became_active":"2024-03-24T12:28:05.797838+0000","last_became_peered":"2024-03-24T12:28:05.797838+0000","last_unstale":"2024-03-24T12:28:42.434542+0000","last_undegraded":"2024-03-24T12:28:42.434542+0000","last_fullsized":"2024-03-24T12:28:42.434542+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3273,"log_dups_size":0,"ondisk_log_size":3273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:29.392832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214499328,"num_objects":3273,"num_object_clones":0,"num_object_copies":9819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3273,"num_write_kb":209472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939887+0000","last_change":"2024-03-24T12:27:12.274502+0000","last_active":"2024-03-24T12:28:27.939887+0000","last_peered":"2024-03-24T12:28:27.939887+0000","last_clean":"2024-03-24T12:28:27.939887+0000","last_became_active":"2024-03-24T12:26:09.347821+0000","last_became_peered":"2024-03-24T12:26:09.347821+0000","last_unstale":"2024-03-24T12:28:27.939887+0000","last_undegraded":"2024-03-24T12:28:27.939887+0000","last_fullsized":"2024-03-24T12:28:27.939887+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.274393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.274393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.457433+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":343,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938062+0000","last_change":"2024-03-24T12:27:08.342835+0000","last_active":"2024-03-24T12:28:27.938062+0000","last_peered":"2024-03-24T12:28:27.938062+0000","last_clean":"2024-03-24T12:28:27.938062+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:28:27.938062+0000","last_undegraded":"2024-03-24T12:28:27.938062+0000","last_fullsized":"2024-03-24T12:28:27.938062+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:27:08.342708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:27:08.342708+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.349456+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939273+0000","last_change":"2024-03-24T12:27:28.436490+0000","last_active":"2024-03-24T12:28:27.939273+0000","last_peered":"2024-03-24T12:28:27.939273+0000","last_clean":"2024-03-24T12:28:27.939273+0000","last_became_active":"2024-03-24T12:26:03.166467+0000","last_became_peered":"2024-03-24T12:26:03.166467+0000","last_unstale":"2024-03-24T12:28:27.939273+0000","last_undegraded":"2024-03-24T12:28:27.939273+0000","last_fullsized":"2024-03-24T12:28:27.939273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.436402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.436402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.490307+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":140,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:33.237677+0000","last_change":"2024-03-24T12:27:21.712724+0000","last_active":"2024-03-24T12:28:33.237677+0000","last_peered":"2024-03-24T12:28:33.237677+0000","last_clean":"2024-03-24T12:28:33.237677+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:28:33.237677+0000","last_undegraded":"2024-03-24T12:28:33.237677+0000","last_fullsized":"2024-03-24T12:28:33.237677+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:27:21.712602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.712602+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.700708+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938604+0000","last_change":"2024-03-24T12:27:46.341475+0000","last_active":"2024-03-24T12:28:27.938604+0000","last_peered":"2024-03-24T12:28:27.938604+0000","last_clean":"2024-03-24T12:28:27.938604+0000","last_became_active":"2024-03-24T12:26:07.337548+0000","last_became_peered":"2024-03-24T12:26:07.337548+0000","last_unstale":"2024-03-24T12:28:27.938604+0000","last_undegraded":"2024-03-24T12:28:27.938604+0000","last_fullsized":"2024-03-24T12:28:27.938604+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:46.341393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:46.341393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:14.879830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"102'3297","reported_seq":3321,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.370534+0000","last_change":"2024-03-24T12:28:05.794985+0000","last_active":"2024-03-24T12:28:42.370534+0000","last_peered":"2024-03-24T12:28:42.370534+0000","last_clean":"2024-03-24T12:28:42.370534+0000","last_became_active":"2024-03-24T12:28:05.793989+0000","last_became_peered":"2024-03-24T12:28:05.793989+0000","last_unstale":"2024-03-24T12:28:42.370534+0000","last_undegraded":"2024-03-24T12:28:42.370534+0000","last_fullsized":"2024-03-24T12:28:42.370534+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3297,"log_dups_size":0,"ondisk_log_size":3297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:23.119876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216072192,"num_objects":3297,"num_object_clones":0,"num_object_copies":9891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3297,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3297,"num_write_kb":211008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.944068+0000","last_change":"2024-03-24T12:27:25.409449+0000","last_active":"2024-03-24T12:28:27.944068+0000","last_peered":"2024-03-24T12:28:27.944068+0000","last_clean":"2024-03-24T12:28:27.944068+0000","last_became_active":"2024-03-24T12:26:09.349435+0000","last_became_peered":"2024-03-24T12:26:09.349435+0000","last_unstale":"2024-03-24T12:28:27.944068+0000","last_undegraded":"2024-03-24T12:28:27.944068+0000","last_fullsized":"2024-03-24T12:28:27.944068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:25.409346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:25.409346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.841693+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.947333+0000","last_change":"2024-03-24T12:27:27.333417+0000","last_active":"2024-03-24T12:28:27.947333+0000","last_peered":"2024-03-24T12:28:27.947333+0000","last_clean":"2024-03-24T12:28:27.947333+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:28:27.947333+0000","last_undegraded":"2024-03-24T12:28:27.947333+0000","last_fullsized":"2024-03-24T12:28:27.947333+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:27.333274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:27.333274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.779264+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":252,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:40.603180+0000","last_change":"2024-03-24T12:27:16.322773+0000","last_active":"2024-03-24T12:28:40.603180+0000","last_peered":"2024-03-24T12:28:40.603180+0000","last_clean":"2024-03-24T12:28:40.603180+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:28:40.603180+0000","last_undegraded":"2024-03-24T12:28:40.603180+0000","last_fullsized":"2024-03-24T12:28:40.603180+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:27:16.322655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:16.322655+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:44.213715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":152,"num_read_kb":147,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":104,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.947304+0000","last_change":"2024-03-24T12:27:38.328555+0000","last_active":"2024-03-24T12:28:27.947304+0000","last_peered":"2024-03-24T12:28:27.947304+0000","last_clean":"2024-03-24T12:28:27.947304+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:28:27.947304+0000","last_undegraded":"2024-03-24T12:28:27.947304+0000","last_fullsized":"2024-03-24T12:28:27.947304+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:38.328458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.328458+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.105741+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940244+0000","last_change":"2024-03-24T12:27:23.276910+0000","last_active":"2024-03-24T12:28:27.940244+0000","last_peered":"2024-03-24T12:28:27.940244+0000","last_clean":"2024-03-24T12:28:27.940244+0000","last_became_active":"2024-03-24T12:26:07.326457+0000","last_became_peered":"2024-03-24T12:26:07.326457+0000","last_unstale":"2024-03-24T12:28:27.940244+0000","last_undegraded":"2024-03-24T12:28:27.940244+0000","last_fullsized":"2024-03-24T12:28:27.940244+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.276789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.276789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:32.022163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"102'3231","reported_seq":3255,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.377340+0000","last_change":"2024-03-24T12:28:05.793024+0000","last_active":"2024-03-24T12:28:42.377340+0000","last_peered":"2024-03-24T12:28:42.377340+0000","last_clean":"2024-03-24T12:28:42.377340+0000","last_became_active":"2024-03-24T12:28:05.792706+0000","last_became_peered":"2024-03-24T12:28:05.792706+0000","last_unstale":"2024-03-24T12:28:42.377340+0000","last_undegraded":"2024-03-24T12:28:42.377340+0000","last_fullsized":"2024-03-24T12:28:42.377340+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3231,"log_dups_size":0,"ondisk_log_size":3231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:12.558322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211746816,"num_objects":3231,"num_object_clones":0,"num_object_copies":9693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3231,"num_write_kb":206784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.944312+0000","last_change":"2024-03-24T12:27:41.410577+0000","last_active":"2024-03-24T12:28:27.944312+0000","last_peered":"2024-03-24T12:28:27.944312+0000","last_clean":"2024-03-24T12:28:27.944312+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:28:27.944312+0000","last_undegraded":"2024-03-24T12:28:27.944312+0000","last_fullsized":"2024-03-24T12:28:27.944312+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:41.410495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:41.410495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:54.007651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.945451+0000","last_change":"2024-03-24T12:27:32.343982+0000","last_active":"2024-03-24T12:28:27.945451+0000","last_peered":"2024-03-24T12:28:27.945451+0000","last_clean":"2024-03-24T12:28:27.945451+0000","last_became_active":"2024-03-24T12:26:03.167244+0000","last_became_peered":"2024-03-24T12:26:03.167244+0000","last_unstale":"2024-03-24T12:28:27.945451+0000","last_undegraded":"2024-03-24T12:28:27.945451+0000","last_fullsized":"2024-03-24T12:28:27.945451+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:32.343880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.343880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:53.358101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":114,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.947006+0000","last_change":"2024-03-24T12:26:05.357149+0000","last_active":"2024-03-24T12:28:27.947006+0000","last_peered":"2024-03-24T12:28:27.947006+0000","last_clean":"2024-03-24T12:28:27.947006+0000","last_became_active":"2024-03-24T12:26:05.356745+0000","last_became_peered":"2024-03-24T12:26:05.356745+0000","last_unstale":"2024-03-24T12:28:27.947006+0000","last_undegraded":"2024-03-24T12:28:27.947006+0000","last_fullsized":"2024-03-24T12:28:27.947006+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:26:04.297367+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.936121+0000","last_change":"2024-03-24T12:27:23.763463+0000","last_active":"2024-03-24T12:28:27.936121+0000","last_peered":"2024-03-24T12:28:27.936121+0000","last_clean":"2024-03-24T12:28:27.936121+0000","last_became_active":"2024-03-24T12:26:07.337303+0000","last_became_peered":"2024-03-24T12:26:07.337303+0000","last_unstale":"2024-03-24T12:28:27.936121+0000","last_undegraded":"2024-03-24T12:28:27.936121+0000","last_fullsized":"2024-03-24T12:28:27.936121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.763326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:36.340669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"102'3266","reported_seq":3290,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.385775+0000","last_change":"2024-03-24T12:28:05.793835+0000","last_active":"2024-03-24T12:28:42.385775+0000","last_peered":"2024-03-24T12:28:42.385775+0000","last_clean":"2024-03-24T12:28:42.385775+0000","last_became_active":"2024-03-24T12:28:05.793524+0000","last_became_peered":"2024-03-24T12:28:05.793524+0000","last_unstale":"2024-03-24T12:28:42.385775+0000","last_undegraded":"2024-03-24T12:28:42.385775+0000","last_fullsized":"2024-03-24T12:28:42.385775+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3266,"log_dups_size":0,"ondisk_log_size":3266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:08.496568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214040576,"num_objects":3266,"num_object_clones":0,"num_object_copies":9798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3266,"num_write_kb":209024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.952347+0000","last_change":"2024-03-24T12:26:09.341274+0000","last_active":"2024-03-24T12:28:27.952347+0000","last_peered":"2024-03-24T12:28:27.952347+0000","last_clean":"2024-03-24T12:28:27.952347+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:28:27.952347+0000","last_undegraded":"2024-03-24T12:28:27.952347+0000","last_fullsized":"2024-03-24T12:28:27.952347+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939387+0000","last_change":"2024-03-24T12:27:26.215446+0000","last_active":"2024-03-24T12:28:27.939387+0000","last_peered":"2024-03-24T12:28:27.939387+0000","last_clean":"2024-03-24T12:28:27.939387+0000","last_became_active":"2024-03-24T12:26:03.168184+0000","last_became_peered":"2024-03-24T12:26:03.168184+0000","last_unstale":"2024-03-24T12:28:27.939387+0000","last_undegraded":"2024-03-24T12:28:27.939387+0000","last_fullsized":"2024-03-24T12:28:27.939387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:26.215348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:26.215348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:42.879847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"74'10","reported_seq":101,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.945316+0000","last_change":"2024-03-24T12:27:24.375037+0000","last_active":"2024-03-24T12:28:27.945316+0000","last_peered":"2024-03-24T12:28:27.945316+0000","last_clean":"2024-03-24T12:28:27.945316+0000","last_became_active":"2024-03-24T12:26:05.316739+0000","last_became_peered":"2024-03-24T12:26:05.316739+0000","last_unstale":"2024-03-24T12:28:27.945316+0000","last_undegraded":"2024-03-24T12:28:27.945316+0000","last_fullsized":"2024-03-24T12:28:27.945316+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:24.374935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.374935+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:47.083756+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":122,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:33.241735+0000","last_change":"2024-03-24T12:27:46.193858+0000","last_active":"2024-03-24T12:28:33.241735+0000","last_peered":"2024-03-24T12:28:33.241735+0000","last_clean":"2024-03-24T12:28:33.241735+0000","last_became_active":"2024-03-24T12:26:07.336080+0000","last_became_peered":"2024-03-24T12:26:07.336080+0000","last_unstale":"2024-03-24T12:28:33.241735+0000","last_undegraded":"2024-03-24T12:28:33.241735+0000","last_fullsized":"2024-03-24T12:28:33.241735+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:46.193732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:46.193732+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:14.150876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"102'3308","reported_seq":3332,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.400650+0000","last_change":"2024-03-24T12:28:05.793884+0000","last_active":"2024-03-24T12:28:42.400650+0000","last_peered":"2024-03-24T12:28:42.400650+0000","last_clean":"2024-03-24T12:28:42.400650+0000","last_became_active":"2024-03-24T12:28:05.793565+0000","last_became_peered":"2024-03-24T12:28:05.793565+0000","last_unstale":"2024-03-24T12:28:42.400650+0000","last_undegraded":"2024-03-24T12:28:42.400650+0000","last_fullsized":"2024-03-24T12:28:42.400650+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3308,"log_dups_size":0,"ondisk_log_size":3308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:10.079951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216793088,"num_objects":3308,"num_object_clones":0,"num_object_copies":9924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3308,"num_write_kb":211712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.938241+0000","last_change":"2024-03-24T12:27:33.429270+0000","last_active":"2024-03-24T12:28:27.938241+0000","last_peered":"2024-03-24T12:28:27.938241+0000","last_clean":"2024-03-24T12:28:27.938241+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:28:27.938241+0000","last_undegraded":"2024-03-24T12:28:27.938241+0000","last_fullsized":"2024-03-24T12:28:27.938241+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:33.429173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:33.429173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:33.664982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":91,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.947482+0000","last_change":"2024-03-24T12:27:12.294371+0000","last_active":"2024-03-24T12:28:27.947482+0000","last_peered":"2024-03-24T12:28:27.947482+0000","last_clean":"2024-03-24T12:28:27.947482+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:28:27.947482+0000","last_undegraded":"2024-03-24T12:28:27.947482+0000","last_fullsized":"2024-03-24T12:28:27.947482+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:27:12.294292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.294292+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.602110+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"93'21","reported_seq":114,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942300+0000","last_change":"2024-03-24T12:27:59.393778+0000","last_active":"2024-03-24T12:28:27.942300+0000","last_peered":"2024-03-24T12:28:27.942300+0000","last_clean":"2024-03-24T12:28:27.942300+0000","last_became_active":"2024-03-24T12:26:05.315454+0000","last_became_peered":"2024-03-24T12:26:05.315454+0000","last_unstale":"2024-03-24T12:28:27.942300+0000","last_undegraded":"2024-03-24T12:28:27.942300+0000","last_fullsized":"2024-03-24T12:28:27.942300+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'21","last_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:27:59.393687+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:07.125229+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":122,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:33.237977+0000","last_change":"2024-03-24T12:27:10.466243+0000","last_active":"2024-03-24T12:28:33.237977+0000","last_peered":"2024-03-24T12:28:33.237977+0000","last_clean":"2024-03-24T12:28:33.237977+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:28:33.237977+0000","last_undegraded":"2024-03-24T12:28:33.237977+0000","last_fullsized":"2024-03-24T12:28:33.237977+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:10.466147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.466147+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:41.829746+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"102'3277","reported_seq":3301,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.398552+0000","last_change":"2024-03-24T12:28:05.795095+0000","last_active":"2024-03-24T12:28:42.398552+0000","last_peered":"2024-03-24T12:28:42.398552+0000","last_clean":"2024-03-24T12:28:42.398552+0000","last_became_active":"2024-03-24T12:28:05.794668+0000","last_became_peered":"2024-03-24T12:28:05.794668+0000","last_unstale":"2024-03-24T12:28:42.398552+0000","last_undegraded":"2024-03-24T12:28:42.398552+0000","last_fullsized":"2024-03-24T12:28:42.398552+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3277,"log_dups_size":0,"ondisk_log_size":3277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:15.378862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214761472,"num_objects":3277,"num_object_clones":0,"num_object_copies":9831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3277,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3277,"num_write_kb":209728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":70,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942094+0000","last_change":"2024-03-24T12:26:09.352908+0000","last_active":"2024-03-24T12:28:27.942094+0000","last_peered":"2024-03-24T12:28:27.942094+0000","last_clean":"2024-03-24T12:28:27.942094+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:28:27.942094+0000","last_undegraded":"2024-03-24T12:28:27.942094+0000","last_fullsized":"2024-03-24T12:28:27.942094+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:26:08.325526+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:29.464790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940470+0000","last_change":"2024-03-24T12:27:36.482571+0000","last_active":"2024-03-24T12:28:27.940470+0000","last_peered":"2024-03-24T12:28:27.940470+0000","last_clean":"2024-03-24T12:28:27.940470+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:28:27.940470+0000","last_undegraded":"2024-03-24T12:28:27.940470+0000","last_fullsized":"2024-03-24T12:28:27.940470+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:36.482466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:36.482466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:03.061842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":118,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939395+0000","last_change":"2024-03-24T12:27:11.454130+0000","last_active":"2024-03-24T12:28:27.939395+0000","last_peered":"2024-03-24T12:28:27.939395+0000","last_clean":"2024-03-24T12:28:27.939395+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:28:27.939395+0000","last_undegraded":"2024-03-24T12:28:27.939395+0000","last_fullsized":"2024-03-24T12:28:27.939395+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T12:27:11.453973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:11.453973+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:28.332849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942169+0000","last_change":"2024-03-24T12:28:06.389115+0000","last_active":"2024-03-24T12:28:27.942169+0000","last_peered":"2024-03-24T12:28:27.942169+0000","last_clean":"2024-03-24T12:28:27.942169+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:28:27.942169+0000","last_undegraded":"2024-03-24T12:28:27.942169+0000","last_fullsized":"2024-03-24T12:28:27.942169+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:06.389004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:28:06.389004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:17.808218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"102'3165","reported_seq":3189,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.387334+0000","last_change":"2024-03-24T12:28:05.793244+0000","last_active":"2024-03-24T12:28:42.387334+0000","last_peered":"2024-03-24T12:28:42.387334+0000","last_clean":"2024-03-24T12:28:42.387334+0000","last_became_active":"2024-03-24T12:28:05.792903+0000","last_became_peered":"2024-03-24T12:28:05.792903+0000","last_unstale":"2024-03-24T12:28:42.387334+0000","last_undegraded":"2024-03-24T12:28:42.387334+0000","last_fullsized":"2024-03-24T12:28:42.387334+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3165,"log_dups_size":0,"ondisk_log_size":3165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:20.208831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207421440,"num_objects":3165,"num_object_clones":0,"num_object_copies":9495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3165,"num_write_kb":202560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939172+0000","last_change":"2024-03-24T12:27:53.270245+0000","last_active":"2024-03-24T12:28:27.939172+0000","last_peered":"2024-03-24T12:28:27.939172+0000","last_clean":"2024-03-24T12:28:27.939172+0000","last_became_active":"2024-03-24T12:26:09.351061+0000","last_became_peered":"2024-03-24T12:26:09.351061+0000","last_unstale":"2024-03-24T12:28:27.939172+0000","last_undegraded":"2024-03-24T12:28:27.939172+0000","last_fullsized":"2024-03-24T12:28:27.939172+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:53.270159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:53.270159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:55.900548+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":122,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:33.237689+0000","last_change":"2024-03-24T12:27:20.101666+0000","last_active":"2024-03-24T12:28:33.237689+0000","last_peered":"2024-03-24T12:28:33.237689+0000","last_clean":"2024-03-24T12:28:33.237689+0000","last_became_active":"2024-03-24T12:26:07.328595+0000","last_became_peered":"2024-03-24T12:26:07.328595+0000","last_unstale":"2024-03-24T12:28:33.237689+0000","last_undegraded":"2024-03-24T12:28:33.237689+0000","last_fullsized":"2024-03-24T12:28:33.237689+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:20.101554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:20.101554+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.424951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939305+0000","last_change":"2024-03-24T12:27:42.431823+0000","last_active":"2024-03-24T12:28:27.939305+0000","last_peered":"2024-03-24T12:28:27.939305+0000","last_clean":"2024-03-24T12:28:27.939305+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:28:27.939305+0000","last_undegraded":"2024-03-24T12:28:27.939305+0000","last_fullsized":"2024-03-24T12:28:27.939305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:42.431729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:42.431729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:00.318970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"92'22","reported_seq":93,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.935994+0000","last_change":"2024-03-24T12:27:42.646116+0000","last_active":"2024-03-24T12:28:27.935994+0000","last_peered":"2024-03-24T12:28:27.935994+0000","last_clean":"2024-03-24T12:28:27.935994+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:28:27.935994+0000","last_undegraded":"2024-03-24T12:28:27.935994+0000","last_fullsized":"2024-03-24T12:28:27.935994+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:27:42.645969+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.491698+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"102'3128","reported_seq":3152,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.366919+0000","last_change":"2024-03-24T12:28:05.794211+0000","last_active":"2024-03-24T12:28:42.366919+0000","last_peered":"2024-03-24T12:28:42.366919+0000","last_clean":"2024-03-24T12:28:42.366919+0000","last_became_active":"2024-03-24T12:28:05.793896+0000","last_became_peered":"2024-03-24T12:28:05.793896+0000","last_unstale":"2024-03-24T12:28:42.366919+0000","last_undegraded":"2024-03-24T12:28:42.366919+0000","last_fullsized":"2024-03-24T12:28:42.366919+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3128,"log_dups_size":0,"ondisk_log_size":3128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:05.089968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204996608,"num_objects":3128,"num_object_clones":0,"num_object_copies":9384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3128,"num_write_kb":200192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.939031+0000","last_change":"2024-03-24T12:27:40.439811+0000","last_active":"2024-03-24T12:28:27.939031+0000","last_peered":"2024-03-24T12:28:27.939031+0000","last_clean":"2024-03-24T12:28:27.939031+0000","last_became_active":"2024-03-24T12:26:09.643739+0000","last_became_peered":"2024-03-24T12:26:09.643739+0000","last_unstale":"2024-03-24T12:28:27.939031+0000","last_undegraded":"2024-03-24T12:28:27.939031+0000","last_fullsized":"2024-03-24T12:28:27.939031+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:40.439695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:40.439695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:04.958905+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940166+0000","last_change":"2024-03-24T12:27:08.227925+0000","last_active":"2024-03-24T12:28:27.940166+0000","last_peered":"2024-03-24T12:28:27.940166+0000","last_clean":"2024-03-24T12:28:27.940166+0000","last_became_active":"2024-03-24T12:26:07.332766+0000","last_became_peered":"2024-03-24T12:26:07.332766+0000","last_unstale":"2024-03-24T12:28:27.940166+0000","last_undegraded":"2024-03-24T12:28:27.940166+0000","last_fullsized":"2024-03-24T12:28:27.940166+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:08.227812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:08.227812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.942208+0000","last_change":"2024-03-24T12:27:31.177660+0000","last_active":"2024-03-24T12:28:27.942208+0000","last_peered":"2024-03-24T12:28:27.942208+0000","last_clean":"2024-03-24T12:28:27.942208+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:28:27.942208+0000","last_undegraded":"2024-03-24T12:28:27.942208+0000","last_fullsized":"2024-03-24T12:28:27.942208+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:31.177529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:31.177529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:49.777930+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":105,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.943082+0000","last_change":"2024-03-24T12:27:51.293056+0000","last_active":"2024-03-24T12:28:27.943082+0000","last_peered":"2024-03-24T12:28:27.943082+0000","last_clean":"2024-03-24T12:28:27.943082+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:28:27.943082+0000","last_undegraded":"2024-03-24T12:28:27.943082+0000","last_fullsized":"2024-03-24T12:28:27.943082+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:27:51.292951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:51.292951+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:01.554228+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"102'3256","reported_seq":3280,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.408269+0000","last_change":"2024-03-24T12:28:05.797388+0000","last_active":"2024-03-24T12:28:42.408269+0000","last_peered":"2024-03-24T12:28:42.408269+0000","last_clean":"2024-03-24T12:28:42.408269+0000","last_became_active":"2024-03-24T12:28:05.797061+0000","last_became_peered":"2024-03-24T12:28:05.797061+0000","last_unstale":"2024-03-24T12:28:42.408269+0000","last_undegraded":"2024-03-24T12:28:42.408269+0000","last_fullsized":"2024-03-24T12:28:42.408269+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3256,"log_dups_size":0,"ondisk_log_size":3256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:33.235641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":213385216,"num_objects":3256,"num_object_clones":0,"num_object_copies":9768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3256,"num_write_kb":208384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.936124+0000","last_change":"2024-03-24T12:27:46.592920+0000","last_active":"2024-03-24T12:28:27.936124+0000","last_peered":"2024-03-24T12:28:27.936124+0000","last_clean":"2024-03-24T12:28:27.936124+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:28:27.936124+0000","last_undegraded":"2024-03-24T12:28:27.936124+0000","last_fullsized":"2024-03-24T12:28:27.936124+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:46.592830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:46.592830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:46.815380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":82,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.946144+0000","last_change":"2024-03-24T12:27:23.411568+0000","last_active":"2024-03-24T12:28:27.946144+0000","last_peered":"2024-03-24T12:28:27.946144+0000","last_clean":"2024-03-24T12:28:27.946144+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:28:27.946144+0000","last_undegraded":"2024-03-24T12:28:27.946144+0000","last_fullsized":"2024-03-24T12:28:27.946144+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:23.411477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:23.411477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:38.665294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.935811+0000","last_change":"2024-03-24T12:27:12.636522+0000","last_active":"2024-03-24T12:28:27.935811+0000","last_peered":"2024-03-24T12:28:27.935811+0000","last_clean":"2024-03-24T12:28:27.935811+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:28:27.935811+0000","last_undegraded":"2024-03-24T12:28:27.935811+0000","last_fullsized":"2024-03-24T12:28:27.935811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:12.636423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:12.636423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:40.538773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"94'25","reported_seq":122,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.943058+0000","last_change":"2024-03-24T12:28:02.436287+0000","last_active":"2024-03-24T12:28:27.943058+0000","last_peered":"2024-03-24T12:28:27.943058+0000","last_clean":"2024-03-24T12:28:27.943058+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:28:27.943058+0000","last_undegraded":"2024-03-24T12:28:27.943058+0000","last_fullsized":"2024-03-24T12:28:27.943058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'25","last_scrub_stamp":"2024-03-24T12:28:02.436198+0000","last_deep_scrub":"94'25","last_deep_scrub_stamp":"2024-03-24T12:28:02.436198+0000","last_clean_scrub_stamp":"2024-03-24T12:28:02.436198+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:26.870834+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"102'3316","reported_seq":3340,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:42.428827+0000","last_change":"2024-03-24T12:28:05.798357+0000","last_active":"2024-03-24T12:28:42.428827+0000","last_peered":"2024-03-24T12:28:42.428827+0000","last_clean":"2024-03-24T12:28:42.428827+0000","last_became_active":"2024-03-24T12:28:05.798041+0000","last_became_peered":"2024-03-24T12:28:05.798041+0000","last_unstale":"2024-03-24T12:28:42.428827+0000","last_undegraded":"2024-03-24T12:28:42.428827+0000","last_fullsized":"2024-03-24T12:28:42.428827+0000","mapping_epoch":95,"log_start":"0'0","ondisk_log_start":"0'0","created":95,"last_epoch_clean":96,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:28:04.776854+0000","last_clean_scrub_stamp":"2024-03-24T12:28:04.776854+0000","objects_scrubbed":0,"log_size":3316,"log_dups_size":0,"ondisk_log_size":3316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:17.799646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217317376,"num_objects":3316,"num_object_clones":0,"num_object_copies":9948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3316,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3316,"num_write_kb":212224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":78,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.943235+0000","last_change":"2024-03-24T12:27:10.145724+0000","last_active":"2024-03-24T12:28:27.943235+0000","last_peered":"2024-03-24T12:28:27.943235+0000","last_clean":"2024-03-24T12:28:27.943235+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:28:27.943235+0000","last_undegraded":"2024-03-24T12:28:27.943235+0000","last_fullsized":"2024-03-24T12:28:27.943235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:10.145613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:10.145613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:29.464813+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":122,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:33.238274+0000","last_change":"2024-03-24T12:27:21.649282+0000","last_active":"2024-03-24T12:28:33.238274+0000","last_peered":"2024-03-24T12:28:33.238274+0000","last_clean":"2024-03-24T12:28:33.238274+0000","last_became_active":"2024-03-24T12:26:07.334798+0000","last_became_peered":"2024-03-24T12:26:07.334798+0000","last_unstale":"2024-03-24T12:28:33.238274+0000","last_undegraded":"2024-03-24T12:28:33.238274+0000","last_fullsized":"2024-03-24T12:28:33.238274+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:21.649181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:21.649181+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:51.389315+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":90,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.940431+0000","last_change":"2024-03-24T12:27:49.399753+0000","last_active":"2024-03-24T12:28:27.940431+0000","last_peered":"2024-03-24T12:28:27.940431+0000","last_clean":"2024-03-24T12:28:27.940431+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:28:27.940431+0000","last_undegraded":"2024-03-24T12:28:27.940431+0000","last_fullsized":"2024-03-24T12:28:27.940431+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:49.399671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:49.399671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:29:16.420862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":101,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-03-24T12:28:27.943058+0000","last_change":"2024-03-24T12:27:22.098052+0000","last_active":"2024-03-24T12:28:27.943058+0000","last_peered":"2024-03-24T12:28:27.943058+0000","last_clean":"2024-03-24T12:28:27.943058+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:28:27.943058+0000","last_undegraded":"2024-03-24T12:28:27.943058+0000","last_fullsized":"2024-03-24T12:28:27.943058+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:27:22.097943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:22.097943+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:28:30.541070+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":3398172672,"num_objects":51852,"num_object_clones":0,"num_object_copies":155556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51852,"num_write_kb":3318528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10195435520,"data_stored":10195435520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51852,"ondisk_log_size":51852,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":152,"num_read_kb":147,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":446,"ondisk_log_size":446,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037796,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1065364,"kb_used_data":1038348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92679788,"statfs":{"total":95995035648,"available":94904102912,"internally_reserved":0,"allocated":1063268352,"data_stored":1062880914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332138,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1069628,"kb_used_data":1007924,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":92675524,"statfs":{"total":95995035648,"available":94899736576,"internally_reserved":0,"allocated":1032114176,"data_stored":1031750726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":63175111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1275052,"kb_used_data":1229988,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":92470100,"statfs":{"total":95995035648,"available":94689382400,"internally_reserved":0,"allocated":1259507712,"data_stored":1259095544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1690388,"kb_used_data":1641164,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92054764,"statfs":{"total":95995035648,"available":94264078336,"internally_reserved":0,"allocated":1680551936,"data_stored":1680163842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1281720,"kb_used_data":1232496,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92463432,"statfs":{"total":95995035648,"available":94682554368,"internally_reserved":0,"allocated":1262075904,"data_stored":1261715466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117048,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1890784,"kb_used_data":1841560,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":91854368,"statfs":{"total":95995035648,"available":94058872832,"internally_reserved":0,"allocated":1885757440,"data_stored":1885357486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50395591},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920820,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1066960,"kb_used_data":1039944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":92678192,"statfs":{"total":95995035648,"available":94902468608,"internally_reserved":0,"allocated":1064902656,"data_stored":1064517912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626479,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":853636,"kb_used_data":822460,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":92891516,"statfs":{"total":95995035648,"available":95120912384,"internally_reserved":0,"allocated":842199040,"data_stored":841827688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1700397056,"data_stored":1700397056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1272774656,"data_stored":1272774656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1282801664,"data_stored":1282801664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1911291904,"data_stored":1911291904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1066401792,"data_stored":1066401792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":846462976,"data_stored":846462976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1047855104,"data_stored":1047855104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1067450368,"data_stored":1067450368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:28:45.964 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T12:28:45.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_0 pg_num 26 2024-03-24T12:28:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:45.999 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:45.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:46.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:46.128 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:46.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:46.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:46.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:28:46.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:28:46.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[29919]: pgmap v82: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-24T12:28:46.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:28:46.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[29919]: 6.4 scrub starts 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[29919]: 6.4 scrub ok 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[41500]: pgmap v82: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[41500]: 6.4 scrub starts 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[41500]: 6.4 scrub ok 2024-03-24T12:28:46.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:46.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:45 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:46.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:46.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:46.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:46.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:46.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:46.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:45 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:28:46.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:45 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:28:46.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:45 smithi190 ceph-mon[33748]: pgmap v82: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-03-24T12:28:46.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:45 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:28:46.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:45 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:28:46.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:45 smithi190 ceph-mon[33748]: 6.4 scrub starts 2024-03-24T12:28:46.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:45 smithi190 ceph-mon[33748]: 6.4 scrub ok 2024-03-24T12:28:46.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:45 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:46.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:45 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:46.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:46.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:46.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:46.533 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:28:46.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:46.538 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:46.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:46.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:46.677 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 33 16 57676 57660 109.183 80.625 0.00832388 0.00915216 2024-03-24T12:28:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:46.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:46.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:46.839 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:46.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:46.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:46.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:28:46.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:47.004 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:47.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:47.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:47.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:47.129 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:47.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:47.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:47.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:28:47.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:28:47.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: 5.6 scrub starts 2024-03-24T12:28:47.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: 5.6 scrub ok 2024-03-24T12:28:47.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: 3.2 scrub starts 2024-03-24T12:28:47.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: 3.2 scrub ok 2024-03-24T12:28:47.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T12:28:47.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: 5.1e scrub starts 2024-03-24T12:28:47.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: 5.1e scrub ok 2024-03-24T12:28:47.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: pgmap v84: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 919 op/s 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: 5.6 scrub starts 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: 5.6 scrub ok 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: 3.2 scrub starts 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: 3.2 scrub ok 2024-03-24T12:28:47.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T12:28:47.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: 5.1e scrub starts 2024-03-24T12:28:47.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: 5.1e scrub ok 2024-03-24T12:28:47.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: pgmap v84: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 919 op/s 2024-03-24T12:28:47.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:47.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:28:47.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:28:47.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:47.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:47.258 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:47.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:47.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:47.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:47.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: 5.6 scrub starts 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: 5.6 scrub ok 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: 3.2 scrub starts 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: 3.2 scrub ok 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: osdmap e104: 8 total, 8 up, 8 in 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: 5.1e scrub starts 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: 5.1e scrub ok 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: pgmap v84: 148 pgs: 148 active+clean; 3.2 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 57 MiB/s wr, 919 op/s 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:28:47.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:28:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:47.413 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:47.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:47.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:47.542 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:47.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:47.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:47.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:47.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:47.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:47.678 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 34 16 58763 58747 107.967 67.9375 0.0123302 0.00925698 2024-03-24T12:28:47.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:47.825 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:28:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:28:47.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:47.833 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:47.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:47.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:28:47.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:47.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:47.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:48.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:48.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:48.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:48.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:48.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: 3.c scrub starts 2024-03-24T12:28:48.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: 3.c scrub ok 2024-03-24T12:28:48.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:28:48.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:28:48.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T12:28:48.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: Reconfiguring daemon alertmanager.a on smithi029 2024-03-24T12:28:48.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:28:48.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:28:48.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:28:48.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:28:48.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: 3.c scrub starts 2024-03-24T12:28:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: 3.c scrub ok 2024-03-24T12:28:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:28:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:28:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T12:28:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: Reconfiguring daemon alertmanager.a on smithi029 2024-03-24T12:28:48.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:28:48.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:28:48.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:28:48.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:28:48.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:47 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:48.183 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:28:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:28:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:28:48.183 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:33724 7.f 7:f51a030d:::benchmark_data_smithi029_84282_object33723:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.132606+0000", 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.050035273999999, 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.25008881700000002, 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 33724 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.184 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132606+0000", 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132606+0000", 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132607+0000", 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.658e-06 2024-03-24T12:28:48.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132630+0000", 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2240999999999999e-05 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132642+0000", 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.208e-05 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.186 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132650+0000", 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5180000000000004e-06 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132756+0000", 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105579 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132887+0000", 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013075399999999999 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.187 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133078+0000", 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000191084 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138443+0000", 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0053652789999999997 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138657+0000", 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021411999999999999 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.188 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.382585+0000", 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.243927753 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.382641+0000", 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5952000000000002e-05 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.189 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.382695+0000", 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3798999999999998e-05 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:33725 7.4 7:227d6376:::benchmark_data_smithi029_84282_object33724:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.132672+0000", 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.049968773, 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.244135192, 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.190 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 33725 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132672+0000", 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132672+0000", 2024-03-24T12:28:48.191 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132673+0000", 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1699999999999998e-07 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132692+0000", 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9619e-05 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.192 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132699+0000", 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8839999999999997e-06 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132705+0000", 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4410000000000001e-06 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132777+0000", 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2444e-05 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.193 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.132889+0000", 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000111643 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,4", 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133058+0000", 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000168698 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139540+0000", 2024-03-24T12:28:48.194 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0064822550000000001 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139586+0000", 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6428999999999999e-05 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139597+0000", 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1241e-05 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.195 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139657+0000", 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9376999999999997e-05 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.376807+0000", 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23715064399999999 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:33729 7.4 7:26cba59c:::benchmark_data_smithi029_84282_object33728:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.133518+0000", 2024-03-24T12:28:48.196 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.049123559000002, 2024-03-24T12:28:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24495681499999999, 2024-03-24T12:28:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.197 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 33729 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133518+0000", 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133518+0000", 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.198 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133519+0000", 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0079999999999999e-06 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133577+0000", 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.834e-05 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.199 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133588+0000", 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1141000000000001e-05 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133596+0000", 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8820000000000001e-06 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.134324+0000", 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00072855800000000003 2024-03-24T12:28:48.200 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.134434+0000", 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010974 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,4", 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.134562+0000", 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012820799999999999 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.376824+0000", 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24226135700000001 2024-03-24T12:28:48.201 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.376877+0000", 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3331000000000002e-05 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.376979+0000", 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102264 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.202 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378425+0000", 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0014459010000000001 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378474+0000", 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9085000000000002e-05 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:33731 7.4 7:2f1e936a:::benchmark_data_smithi029_84282_object33730:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.203 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.133673+0000", 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.048967912999998, 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24503512899999999, 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 33731 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.204 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133673+0000", 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133673+0000", 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133674+0000", 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9100000000000004e-07 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.205 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133692+0000", 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8607999999999999e-05 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133699+0000", 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4529999999999999e-06 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.133704+0000", 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3279999999999996e-06 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.206 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.135955+0000", 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0022510469999999999 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.136059+0000", 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000103486 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,4", 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.136183+0000", 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012413100000000001 2024-03-24T12:28:48.207 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.376830+0000", 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24064699000000001 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.376941+0000", 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000111295 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378521+0000", 2024-03-24T12:28:48.208 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001580442 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378668+0000", 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000146969 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378708+0000", 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9888999999999997e-05 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.209 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:33732 7.4 7:24f5b525:::benchmark_data_smithi029_84282_object33731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.135843+0000", 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.046797722000001, 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24304420099999999, 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 33732 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.210 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.135843+0000", 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.135843+0000", 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.135845+0000", 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.328e-06 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.211 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.135875+0000", 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0378999999999998e-05 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.135888+0000", 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2842e-05 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.135897+0000", 2024-03-24T12:28:48.212 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8300000000000002e-06 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137621+0000", 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0017244599999999999 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138159+0000", 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00053816900000000002 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,4", 2024-03-24T12:28:48.213 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138472+0000", 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00031263599999999998 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.376895+0000", 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23842261200000001 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378754+0000", 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0018595339999999999 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.214 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378801+0000", 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7046000000000003e-05 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378849+0000", 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.829e-05 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378888+0000", 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8074999999999997e-05 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.215 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.14799.0:33734 7.d e102/95 7:b01058d9:::benchmark_data_smithi029_84282_object33733:head v 102'2049, mlcod=102'2047)", 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.137216+0000", 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.045424807, 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.242232322, 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137216+0000", 2024-03-24T12:28:48.216 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137216+0000", 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137217+0000", 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2200000000000003e-07 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137320+0000", 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010256599999999999 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137329+0000", 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7510000000000003e-06 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137337+0000", 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1420000000000004e-06 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.218 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137623+0000", 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00028670699999999998 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137679+0000", 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5923e-05 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.379282+0000", 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24160248000000001 2024-03-24T12:28:48.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.379449+0000", 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016693099999999999 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.14799.0:33725 7.4 e102/95 ondisk, result = 0)", 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.137526+0000", 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.045114978000001, 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23930504, 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.220 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137526+0000", 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137526+0000", 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137527+0000", 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8899999999999999e-07 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137531+0000", 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.76e-06 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137539+0000", 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0010000000000001e-06 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.137544+0000", 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1580000000000004e-06 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139575+0000", 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020312300000000002 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139581+0000", 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2339999999999999e-06 2024-03-24T12:28:48.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.376831+0000", 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23724996800000001 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:33736 7.4 7:23af1eb5:::benchmark_data_smithi029_84282_object33735:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.138155+0000", 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.044485656999999, 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24652769299999999, 2024-03-24T12:28:48.224 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 33736 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138155+0000", 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138155+0000", 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138157+0000", 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.48e-06 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138183+0000", 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6316999999999999e-05 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138194+0000", 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0633e-05 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138202+0000", 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.824e-06 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378928+0000", 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.240726305 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.379075+0000", 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014723399999999999 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,4", 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.379233+0000", 2024-03-24T12:28:48.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015809999999999999 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.382136+0000", 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0029030979999999998 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.382539+0000", 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00040278800000000001 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.230 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.384584+0000", 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002045172 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.384641+0000", 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6190999999999998e-05 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.384683+0000", 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2551e-05 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.231 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.14799.0:33729 7.4 e102/95 ondisk, result = 0)", 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.138402+0000", 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.044239480000002, 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24007980000000001, 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138402+0000", 2024-03-24T12:28:48.232 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138402+0000", 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138402+0000", 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1899999999999998e-07 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138405+0000", 2024-03-24T12:28:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4940000000000002e-06 2024-03-24T12:28:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138410+0000", 2024-03-24T12:28:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8839999999999999e-06 2024-03-24T12:28:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138415+0000", 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0799999999999996e-06 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.376968+0000", 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.238552926 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.376975+0000", 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2400000000000004e-06 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.235 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378481+0000", 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0015067570000000001 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.14799.0:33735 7.d e102/95 7:bc01c52a:::benchmark_data_smithi029_84282_object33734:head v 102'2050, mlcod=102'2047)", 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.138469+0000", 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.044171831, 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.241021183, 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.236 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138469+0000", 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138469+0000", 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138470+0000", 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5000000000000002e-07 2024-03-24T12:28:48.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138522+0000", 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2043999999999998e-05 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138530+0000", 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7579999999999992e-06 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138536+0000", 2024-03-24T12:28:48.238 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.426e-06 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138772+0000", 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023612399999999999 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.138790+0000", 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7520000000000002e-05 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.239 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.379452+0000", 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24066264000000001 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.379490+0000", 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8021000000000001e-05 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.14799.0:33731 7.4 e102/95 ondisk, result = 0)", 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:29.139371+0000", 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.043270438, 2024-03-24T12:28:48.240 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23934647100000001, 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139371+0000", 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139371+0000", 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.241 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139371+0000", 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1200000000000002e-07 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139375+0000", 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1359999999999999e-06 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139383+0000", 2024-03-24T12:28:48.242 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0509999999999999e-06 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.139388+0000", 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3369999999999999e-06 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378508+0000", 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23912036 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378516+0000", 2024-03-24T12:28:48.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4819999999999997e-06 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:29.378717+0000", 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020139299999999999 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:35316 7.f 7:f0d4994a:::benchmark_data_smithi029_84282_object35315:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:30.677185+0000", 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.505455645000001, 2024-03-24T12:28:48.244 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30957811899999998, 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 35316 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.677185+0000", 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.677185+0000", 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.677187+0000", 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1429999999999999e-06 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.677207+0000", 2024-03-24T12:28:48.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0587000000000001e-05 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.677218+0000", 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1022000000000001e-05 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.677226+0000", 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5399999999999998e-06 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.681167+0000", 2024-03-24T12:28:48.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003941071 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.681295+0000", 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000128107 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685100+0000", 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003805316 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.248 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.687776+0000", 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026755099999999999 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690623+0000", 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028476809999999999 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986677+0000", 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29605392899999999 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986726+0000", 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8773000000000002e-05 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986764+0000", 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7440000000000001e-05 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:35321 7.6 7:64fe24a6:::benchmark_data_smithi029_84282_object35320:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.250 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:30.680769+0000", 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.501871989000001, 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30592134700000001, 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 35321 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.680769+0000", 2024-03-24T12:28:48.251 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.680769+0000", 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.680770+0000", 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7999999999999993e-07 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.680793+0000", 2024-03-24T12:28:48.252 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3300000000000001e-05 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.680805+0000", 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1199000000000001e-05 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.680812+0000", 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4900000000000003e-06 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.253 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.681823+0000", 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010107689999999999 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.681951+0000", 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012853499999999999 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685063+0000", 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0031112190000000001 2024-03-24T12:28:48.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688223+0000", 2024-03-24T12:28:48.255 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003160206 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688387+0000", 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016459600000000001 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986583+0000", 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29819550500000003 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.256 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986643+0000", 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0439000000000002e-05 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986690+0000", 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7108999999999999e-05 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:35325 7.6 7:66c22023:::benchmark_data_smithi029_84282_object35324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:30.685107+0000", 2024-03-24T12:28:48.257 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.497533626999999, 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30170259399999999, 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 35325 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685107+0000", 2024-03-24T12:28:48.258 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685107+0000", 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685109+0000", 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4610000000000001e-06 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.259 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685131+0000", 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2439e-05 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685140+0000", 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.619e-06 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685146+0000", 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3629999999999999e-06 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.260 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.686718+0000", 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0015715379999999999 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.686843+0000", 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000124836 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.686996+0000", 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015369 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.261 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688410+0000", 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001413596 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688446+0000", 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5574999999999997e-05 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986728+0000", 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29828286999999998 2024-03-24T12:28:48.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986775+0000", 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6817999999999998e-05 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986810+0000", 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4789000000000002e-05 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:35328 7.f 7:f24a1c24:::benchmark_data_smithi029_84282_object35327:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:30.685158+0000", 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.497482779999999, 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.30172306500000001, 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 35328 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.264 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685158+0000", 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685158+0000", 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685159+0000", 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8999999999999997e-07 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.265 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685178+0000", 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9677000000000002e-05 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685185+0000", 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3890000000000002e-06 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.685191+0000", 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7599999999999999e-06 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.266 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.687807+0000", 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026164669999999999 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.687930+0000", 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000122428 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688047+0000", 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011768 2024-03-24T12:28:48.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690601+0000", 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0025540810000000001 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690692+0000", 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0562999999999993e-05 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986801+0000", 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29610877000000002 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986847+0000", 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6146999999999997e-05 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986881+0000", 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4613000000000002e-05 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.269 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:35336 7.f 7:f3fc4aea:::benchmark_data_smithi029_84282_object35335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:30.688412+0000", 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.494229534999999, 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29859238500000002, 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 35336 2024-03-24T12:28:48.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688412+0000", 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688412+0000", 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.271 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688413+0000", 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6500000000000008e-07 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688432+0000", 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9806e-05 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688443+0000", 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0448999999999999e-05 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.272 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.688451+0000", 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7880000000000007e-06 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.689336+0000", 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00088502899999999996 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.689443+0000", 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010762199999999999 2024-03-24T12:28:48.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.689569+0000", 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000125671 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690741+0000", 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00117256 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.692245+0000", 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0015037869999999999 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986923+0000", 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29467813900000001 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.986969+0000", 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5943999999999999e-05 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.275 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.987004+0000", 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4625000000000001e-05 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:35337 7.6 7:642887d6:::benchmark_data_smithi029_84282_object35336:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:30.690850+0000", 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.491791144, 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29820623299999999, 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.276 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 35337 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690850+0000", 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690850+0000", 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690851+0000", 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6899999999999996e-07 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690872+0000", 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1223e-05 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690882+0000", 2024-03-24T12:28:48.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0079e-05 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690890+0000", 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9370000000000004e-06 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.690939+0000", 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9137000000000001e-05 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.691063+0000", 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012351199999999999 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.691227+0000", 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016464 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.692592+0000", 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0013644009999999999 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.692673+0000", 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1343999999999999e-05 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.988978+0000", 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29630488300000002 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.989022+0000", 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3516999999999997e-05 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.281 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.282 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.989056+0000", 2024-03-24T12:28:48.282 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4591000000000001e-05 2024-03-24T12:28:48.282 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.282 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.282 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.282 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.283 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:35342 7.f 7:f6f4d2a1:::benchmark_data_smithi029_84282_object35341:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.283 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:30.692936+0000", 2024-03-24T12:28:48.283 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 17.489704818, 2024-03-24T12:28:48.283 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29418456599999998, 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 35342 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.692936+0000", 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.284 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.692936+0000", 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.692937+0000", 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9999999999999997e-07 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.692967+0000", 2024-03-24T12:28:48.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0190000000000001e-05 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.692976+0000", 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8170000000000005e-06 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.692983+0000", 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.272e-06 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.286 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.693055+0000", 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1660000000000002e-05 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.693178+0000", 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012286400000000001 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.693323+0000", 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000145601 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.694591+0000", 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001267467 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.694865+0000", 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027425699999999997 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.987042+0000", 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29217646600000002 2024-03-24T12:28:48.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.987088+0000", 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6786000000000002e-05 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:30.987121+0000", 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2486e-05 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:40400 7.4 7:2f13aa34:::benchmark_data_smithi029_84282_object40399:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:34.243571+0000", 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 13.939070084000001, 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24122691099999999, 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 40400 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.290 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.243571+0000", 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.243571+0000", 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.243572+0000", 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.362e-06 2024-03-24T12:28:48.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.243591+0000", 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8187000000000001e-05 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.243599+0000", 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0010000000000001e-06 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.243605+0000", 2024-03-24T12:28:48.292 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2659999999999998e-06 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.243732+0000", 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012695 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.243872+0000", 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014032199999999999 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,4", 2024-03-24T12:28:48.293 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.244023+0000", 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015106000000000001 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.247671+0000", 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003647937 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.247707+0000", 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5738000000000002e-05 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.294 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.484693+0000", 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.236986373 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.484752+0000", 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9296000000000003e-05 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.484798+0000", 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5419e-05 2024-03-24T12:28:48.295 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:40413 7.f 7:f43bcd1d:::benchmark_data_smithi029_84282_object40412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e102)", 2024-03-24T12:28:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:28:34.249284+0000", 2024-03-24T12:28:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 13.933357223, 2024-03-24T12:28:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23923593600000001, 2024-03-24T12:28:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:28:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 40413 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.249284+0000", 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.298 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.249284+0000", 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.249285+0000", 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1999999999999998e-07 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.249311+0000", 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6346000000000002e-05 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.249321+0000", 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7440000000000002e-06 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.249329+0000", 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6650000000000003e-06 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.249383+0000", 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4602999999999999e-05 2024-03-24T12:28:48.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.249501+0000", 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011807 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,7", 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.249699+0000", 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019764699999999999 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.483457+0000", 2024-03-24T12:28:48.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23375802200000001 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.483827+0000", 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00037057399999999998 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.488302+0000", 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0044741980000000004 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:28:48.302 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.488471+0000", 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016976400000000001 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:28:34.488520+0000", 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8383000000000001e-05 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:28:48.303 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:28:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:48.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: 3.c scrub starts 2024-03-24T12:28:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: 3.c scrub ok 2024-03-24T12:28:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:28:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:28:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-24T12:28:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: Reconfiguring daemon alertmanager.a on smithi029 2024-03-24T12:28:48.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:28:48.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:28:48.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:28:48.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:28:48.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:47 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:48.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:48.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:48.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:48.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:48.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:48.579 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:48.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:48.678 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 35 16 60116 60100 107.298 84.5625 0.0175212 0.00931444 2024-03-24T12:28:48.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:28:48.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:48.735 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:48.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:48.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:48.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:48.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:48.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:48.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:49.009 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:49.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:49.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:49.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:49.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[41500]: 4.15 scrub starts 2024-03-24T12:28:49.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[41500]: 4.15 scrub ok 2024-03-24T12:28:49.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:28:49.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:28:49.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[41500]: 3.11 scrub starts 2024-03-24T12:28:49.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[41500]: 3.11 scrub ok 2024-03-24T12:28:49.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[41500]: pgmap v85: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s 2024-03-24T12:28:49.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[29919]: 4.15 scrub starts 2024-03-24T12:28:49.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[29919]: 4.15 scrub ok 2024-03-24T12:28:49.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:28:49.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:28:49.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[29919]: 3.11 scrub starts 2024-03-24T12:28:49.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[29919]: 3.11 scrub ok 2024-03-24T12:28:49.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:48 smithi029 ceph-mon[29919]: pgmap v85: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s 2024-03-24T12:28:49.161 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:49.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:49.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:49.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:49.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:49.311 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:49.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:48 smithi190 ceph-mon[33748]: 4.15 scrub starts 2024-03-24T12:28:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:48 smithi190 ceph-mon[33748]: 4.15 scrub ok 2024-03-24T12:28:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:48 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:28:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:48 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:28:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:48 smithi190 ceph-mon[33748]: 3.11 scrub starts 2024-03-24T12:28:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:48 smithi190 ceph-mon[33748]: 3.11 scrub ok 2024-03-24T12:28:49.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:48 smithi190 ceph-mon[33748]: pgmap v85: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s 2024-03-24T12:28:49.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:49.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:49.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:49.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:49.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:49.577 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:49.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:49.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:49.678 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 36 16 61512 61496 106.741 87.25 0.00759707 0.00936305 2024-03-24T12:28:49.679 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:49.716 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:49.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:49.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:49.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:49.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:49.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:49.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:49.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:49.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:50.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:50.115 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:50 smithi029 ceph-mon[41500]: 3.1c scrub starts 2024-03-24T12:28:50.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:50 smithi029 ceph-mon[41500]: 3.1c scrub ok 2024-03-24T12:28:50.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:50 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:28:50.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:50 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:28:50.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:50 smithi029 ceph-mon[41500]: 5.2 scrub starts 2024-03-24T12:28:50.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:50 smithi029 ceph-mon[41500]: 5.2 scrub ok 2024-03-24T12:28:50.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:49 smithi029 ceph-mon[29919]: 3.1c scrub starts 2024-03-24T12:28:50.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:49 smithi029 ceph-mon[29919]: 3.1c scrub ok 2024-03-24T12:28:50.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:49 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:28:50.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:49 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:28:50.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:49 smithi029 ceph-mon[29919]: 5.2 scrub starts 2024-03-24T12:28:50.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:49 smithi029 ceph-mon[29919]: 5.2 scrub ok 2024-03-24T12:28:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:50.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:50.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:50.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T12:28:50.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:50.250 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:50.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:50.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:50.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:50.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:49 smithi190 ceph-mon[33748]: 3.1c scrub starts 2024-03-24T12:28:50.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:49 smithi190 ceph-mon[33748]: 3.1c scrub ok 2024-03-24T12:28:50.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:49 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:28:50.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:49 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:28:50.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:49 smithi190 ceph-mon[33748]: 5.2 scrub starts 2024-03-24T12:28:50.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:49 smithi190 ceph-mon[33748]: 5.2 scrub ok 2024-03-24T12:28:50.397 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:50.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:50.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:50.547 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:50.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:50.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:50.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 37 16 62847 62831 106.108 83.4375 0.0122238 0.00941835 2024-03-24T12:28:50.701 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:50.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:50.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:50.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:50.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:50.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:50.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:50.968 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:50.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:51.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:51.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:51.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:51.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:51.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[41500]: 6.7 scrub starts 2024-03-24T12:28:51.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[41500]: 6.7 scrub ok 2024-03-24T12:28:51.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:28:51.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:28:51.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:28:51.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:28:51.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:28:51.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:28:51.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[41500]: pgmap v86: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-24T12:28:51.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[29919]: 6.7 scrub starts 2024-03-24T12:28:51.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[29919]: 6.7 scrub ok 2024-03-24T12:28:51.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:28:51.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:28:51.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:28:51.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:28:51.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:28:51.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:28:51.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:51 smithi029 ceph-mon[29919]: pgmap v86: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-24T12:28:51.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:51.227 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:28:51.248 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:51.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:51.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:51.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:51 smithi190 ceph-mon[33748]: 6.7 scrub starts 2024-03-24T12:28:51.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:51 smithi190 ceph-mon[33748]: 6.7 scrub ok 2024-03-24T12:28:51.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:51 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:28:51.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:51 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:28:51.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:51 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:28:51.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:51 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:28:51.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:51 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:28:51.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:51 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:28:51.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:51 smithi190 ceph-mon[33748]: pgmap v86: 148 pgs: 148 active+clean; 3.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-24T12:28:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:51.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:51.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:51.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:51.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:51.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:51.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:51.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:51.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:51.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 38 16 64212 64196 105.56 85.3125 0.0173276 0.00946747 2024-03-24T12:28:51.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:51.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:51.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:51.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:51.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:51.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:51.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:52.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:52.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:52.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:52.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:52.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:52.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:52.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:52 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:28:52.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:52 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:28:52.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:52 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:28:52.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:52 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:28:52.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:52 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:28:52.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:52 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:28:52.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:52 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:28:52.284 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:52 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:28:52.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:52.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:52.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:52.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:52 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:28:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:52 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:28:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:52 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:28:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:52 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:28:52.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:52.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:52.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:52.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:52.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:52.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 39 16 65536 65520 104.975 82.75 0.0131021 0.00952136 2024-03-24T12:28:52.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:52.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:52.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:52.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:52.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:52.887 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:52.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:52.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:52.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:53.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:53.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:53.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:53.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:53.154 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:53.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:53.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:53.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:53.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:53.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: 4.8 scrub starts 2024-03-24T12:28:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: 4.8 scrub ok 2024-03-24T12:28:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:28:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:28:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:28:53.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:28:53.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:28:53.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:28:53.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:28:53.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:28:53.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:53 smithi190 ceph-mon[33748]: pgmap v87: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T12:28:53.418 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:53.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: 4.8 scrub starts 2024-03-24T12:28:53.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: 4.8 scrub ok 2024-03-24T12:28:53.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:28:53.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:28:53.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:28:53.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:28:53.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:28:53.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:28:53.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:28:53.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:28:53.423 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[41500]: pgmap v87: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T12:28:53.424 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: 4.8 scrub starts 2024-03-24T12:28:53.424 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: 4.8 scrub ok 2024-03-24T12:28:53.424 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:28:53.425 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:28:53.425 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:28:53.425 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:28:53.425 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:28:53.425 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:28:53.425 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:28:53.425 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:28:53.426 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:53 smithi029 ceph-mon[29919]: pgmap v87: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T12:28:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:53.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:53.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:53.569 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:53.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:53.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:53.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:53.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:28:53.684738+0000 min lat: 0.00114673 max lat: 0.335061 avg lat: 0.00959273 2024-03-24T12:28:53.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:28:53.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 40 16 66714 66698 104.189 73.625 0.0125321 0.00959273 2024-03-24T12:28:53.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:53.723 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:53.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:53.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:53.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:53.848 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:53.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:53.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:53.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:53.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:54.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:54.109 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:54.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:54.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:54.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:54.244 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:54.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:54.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:54 smithi190 ceph-mon[33748]: 5.0 scrub starts 2024-03-24T12:28:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:54 smithi190 ceph-mon[33748]: 5.0 scrub ok 2024-03-24T12:28:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:54 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:28:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:54 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:28:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:54.388 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:54.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:54.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:54 smithi029 ceph-mon[29919]: 5.0 scrub starts 2024-03-24T12:28:54.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:54 smithi029 ceph-mon[29919]: 5.0 scrub ok 2024-03-24T12:28:54.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:54 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:28:54.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:54 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:28:54.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:54 smithi029 ceph-mon[41500]: 5.0 scrub starts 2024-03-24T12:28:54.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:54 smithi029 ceph-mon[41500]: 5.0 scrub ok 2024-03-24T12:28:54.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:54 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:28:54.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:54 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:28:54.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:54.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:54.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:54.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:54.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:54.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:54.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:54.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 41 16 67772 67756 103.26 66.125 0.012673 0.00967885 2024-03-24T12:28:54.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:54.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:54.804 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:54.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:54.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:54.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:54.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:55.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:55.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:55.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:55.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:55.166 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:55.206 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:55.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:55.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:55.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:55.333 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:55.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:55 smithi190 ceph-mon[33748]: 5.7 scrub starts 2024-03-24T12:28:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:55 smithi190 ceph-mon[33748]: 5.7 scrub ok 2024-03-24T12:28:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:55 smithi190 ceph-mon[33748]: 4.0 scrub starts 2024-03-24T12:28:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:55 smithi190 ceph-mon[33748]: 4.0 scrub ok 2024-03-24T12:28:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:55 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:28:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:55 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:28:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:55 smithi190 ceph-mon[33748]: pgmap v88: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T12:28:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[29919]: 5.7 scrub starts 2024-03-24T12:28:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[29919]: 5.7 scrub ok 2024-03-24T12:28:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[29919]: 4.0 scrub starts 2024-03-24T12:28:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[29919]: 4.0 scrub ok 2024-03-24T12:28:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:28:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:28:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[29919]: pgmap v88: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T12:28:55.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[41500]: 5.7 scrub starts 2024-03-24T12:28:55.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[41500]: 5.7 scrub ok 2024-03-24T12:28:55.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[41500]: 4.0 scrub starts 2024-03-24T12:28:55.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[41500]: 4.0 scrub ok 2024-03-24T12:28:55.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:28:55.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:28:55.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:55 smithi029 ceph-mon[41500]: pgmap v88: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T12:28:55.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:55.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:55.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:55.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:55.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:55.608 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:55.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:55.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:55.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 42 16 68767 68751 102.283 62.1875 0.00584938 0.00977164 2024-03-24T12:28:55.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:55.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:55.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:55.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:55.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:55.860 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:55.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:55.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:56.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:56.003 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:56.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:56.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:56.130 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:56.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:56.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:56.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:56.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:56.276 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:56.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:56.377 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:56.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:56 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:28:56.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:56 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:28:56.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:56 smithi190 ceph-mon[33748]: 4.10 scrub starts 2024-03-24T12:28:56.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:56 smithi190 ceph-mon[33748]: 4.10 scrub ok 2024-03-24T12:28:56.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:56 smithi190 ceph-mon[33748]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T12:28:56.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:56 smithi190 ceph-mon[33748]: 6.1d scrub starts 2024-03-24T12:28:56.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:56 smithi190 ceph-mon[33748]: 6.1d scrub ok 2024-03-24T12:28:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:56.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:56.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:56.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:28:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:28:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[41500]: 4.10 scrub starts 2024-03-24T12:28:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[41500]: 4.10 scrub ok 2024-03-24T12:28:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[41500]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T12:28:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[41500]: 6.1d scrub starts 2024-03-24T12:28:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[41500]: 6.1d scrub ok 2024-03-24T12:28:56.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:28:56.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:28:56.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[29919]: 4.10 scrub starts 2024-03-24T12:28:56.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[29919]: 4.10 scrub ok 2024-03-24T12:28:56.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[29919]: osdmap e105: 8 total, 8 up, 8 in 2024-03-24T12:28:56.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[29919]: 6.1d scrub starts 2024-03-24T12:28:56.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:56 smithi029 ceph-mon[29919]: 6.1d scrub ok 2024-03-24T12:28:56.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:56.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:56.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:56.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:56.681 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:56.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 43 15 69668 69653 101.215 56.375 0.00836567 0.00987438 2024-03-24T12:28:56.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:56.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:56.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:56.825 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:56.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:56.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:56.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:56.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:57.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:57.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:57.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:57.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:57.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:57.227 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:57.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:57.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:57.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[41500]: 6.19 scrub starts 2024-03-24T12:28:57.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[41500]: 6.19 scrub ok 2024-03-24T12:28:57.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:28:57.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:28:57.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[41500]: 5.13 scrub starts 2024-03-24T12:28:57.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[41500]: 5.13 scrub ok 2024-03-24T12:28:57.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[41500]: 4.2 scrub starts 2024-03-24T12:28:57.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[41500]: 4.2 scrub ok 2024-03-24T12:28:57.304 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[41500]: pgmap v90: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T12:28:57.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[29919]: 6.19 scrub starts 2024-03-24T12:28:57.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[29919]: 6.19 scrub ok 2024-03-24T12:28:57.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:28:57.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:28:57.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[29919]: 5.13 scrub starts 2024-03-24T12:28:57.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[29919]: 5.13 scrub ok 2024-03-24T12:28:57.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[29919]: 4.2 scrub starts 2024-03-24T12:28:57.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[29919]: 4.2 scrub ok 2024-03-24T12:28:57.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-mon[29919]: pgmap v90: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T12:28:57.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:57.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:57.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:57.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:57 smithi190 ceph-mon[33748]: 6.19 scrub starts 2024-03-24T12:28:57.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:57 smithi190 ceph-mon[33748]: 6.19 scrub ok 2024-03-24T12:28:57.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:57 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:28:57.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:57 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:28:57.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:57 smithi190 ceph-mon[33748]: 5.13 scrub starts 2024-03-24T12:28:57.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:57 smithi190 ceph-mon[33748]: 5.13 scrub ok 2024-03-24T12:28:57.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:57 smithi190 ceph-mon[33748]: 4.2 scrub starts 2024-03-24T12:28:57.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:57 smithi190 ceph-mon[33748]: 4.2 scrub ok 2024-03-24T12:28:57.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:57 smithi190 ceph-mon[33748]: pgmap v90: 148 pgs: 148 active+clean; 4.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T12:28:57.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:57.483 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:57.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:57.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:57.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:57.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 44 16 70434 70418 100.001 47.8125 0.00883504 0.00999477 2024-03-24T12:28:57.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:57.742 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:57.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:57.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:28:57.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:28:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:28:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:57.885 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:57.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:28:57.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:28:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:28:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:28:57.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:58.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:58.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:58.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:58.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:58.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:58.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:58.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:58.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:58.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:58.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:58.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:58.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:58 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:28:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:58 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:28:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:58 smithi190 ceph-mon[33748]: 3.18 scrub starts 2024-03-24T12:28:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:58 smithi190 ceph-mon[33748]: 3.18 scrub ok 2024-03-24T12:28:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:58 smithi190 ceph-mon[33748]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T12:28:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:58 smithi190 ceph-mon[33748]: 3.5 scrub starts 2024-03-24T12:28:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:58 smithi190 ceph-mon[33748]: 3.5 scrub ok 2024-03-24T12:28:58.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:58.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:58.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:58.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:28:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:28:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[41500]: 3.18 scrub starts 2024-03-24T12:28:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[41500]: 3.18 scrub ok 2024-03-24T12:28:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[41500]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T12:28:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[41500]: 3.5 scrub starts 2024-03-24T12:28:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[41500]: 3.5 scrub ok 2024-03-24T12:28:58.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:28:58.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:28:58.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[29919]: 3.18 scrub starts 2024-03-24T12:28:58.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[29919]: 3.18 scrub ok 2024-03-24T12:28:58.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[29919]: osdmap e106: 8 total, 8 up, 8 in 2024-03-24T12:28:58.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:28:58.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[29919]: 3.5 scrub starts 2024-03-24T12:28:58.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:58 smithi029 ceph-mon[29919]: 3.5 scrub ok 2024-03-24T12:28:58.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:28:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:58.542 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:58.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:28:58.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:28:58.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:58.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:58.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:28:58.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 45 16 71392 71376 99.1098 59.875 0.00585422 0.0100683 2024-03-24T12:28:58.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:58.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:58.793 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:58.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:58.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:28:58.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:58.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:58.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:28:59.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:59.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:59.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:59.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:59.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:59.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:59.187 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:59.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:59.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:28:59.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:59.313 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:59.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:28:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:59 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:28:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:59 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:28:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:59 smithi190 ceph-mon[33748]: 4.b scrub starts 2024-03-24T12:28:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:59 smithi190 ceph-mon[33748]: 4.b scrub ok 2024-03-24T12:28:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:28:59 smithi190 ceph-mon[33748]: pgmap v92: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-24T12:28:59.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:59 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:28:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:59 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:28:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:59 smithi029 ceph-mon[29919]: 4.b scrub starts 2024-03-24T12:28:59.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:59 smithi029 ceph-mon[29919]: 4.b scrub ok 2024-03-24T12:28:59.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:28:59 smithi029 ceph-mon[29919]: pgmap v92: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-24T12:28:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:59 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:28:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:59 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:28:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:59 smithi029 ceph-mon[41500]: 4.b scrub starts 2024-03-24T12:28:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:59 smithi029 ceph-mon[41500]: 4.b scrub ok 2024-03-24T12:28:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:28:59 smithi029 ceph-mon[41500]: pgmap v92: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-03-24T12:28:59.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:59.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:59.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:28:59.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:59.591 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:59.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:28:59.682 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 46 15 72248 72233 98.1196 53.5625 0.00891778 0.0101868 2024-03-24T12:28:59.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:28:59.721 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:28:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:59.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:28:59.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:28:59.859 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:28:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:28:59.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:28:59.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:00.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:00.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:00.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:00.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:00.133 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:00.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:00.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:00.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:00.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:00.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:00.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:00 smithi190 ceph-mon[33748]: 6.9 scrub starts 2024-03-24T12:29:00.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:00 smithi190 ceph-mon[33748]: 6.9 scrub ok 2024-03-24T12:29:00.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:00 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:29:00.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:00 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:29:00.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:00 smithi190 ceph-mon[33748]: 5.12 scrub starts 2024-03-24T12:29:00.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:00 smithi190 ceph-mon[33748]: 5.12 scrub ok 2024-03-24T12:29:00.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:00.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:00.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[29919]: 6.9 scrub starts 2024-03-24T12:29:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[29919]: 6.9 scrub ok 2024-03-24T12:29:00.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:29:00.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:29:00.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[29919]: 5.12 scrub starts 2024-03-24T12:29:00.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[29919]: 5.12 scrub ok 2024-03-24T12:29:00.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[41500]: 6.9 scrub starts 2024-03-24T12:29:00.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[41500]: 6.9 scrub ok 2024-03-24T12:29:00.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:29:00.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:29:00.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[41500]: 5.12 scrub starts 2024-03-24T12:29:00.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:00 smithi029 ceph-mon[41500]: 5.12 scrub ok 2024-03-24T12:29:00.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:00.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:00.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:00.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:00.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:00.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:00.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:00.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 47 16 73137 73121 97.2129 55.5 0.0147973 0.0102821 2024-03-24T12:29:00.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:00.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:00.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:00.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:00.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:00.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:00.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:01.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:01.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:01.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:01.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:01.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:01.177 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:01.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:01.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:01.302 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:01.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:01 smithi190 ceph-mon[33748]: pgmap v93: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:29:01.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:01.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:01 smithi029 ceph-mon[29919]: pgmap v93: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:29:01.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:01 smithi029 ceph-mon[41500]: pgmap v93: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:29:01.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:01.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:01.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:01.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:01.581 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:01.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:01.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 48 16 74030 74014 96.3504 55.8125 0.00554582 0.0103737 2024-03-24T12:29:01.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:01.724 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:01.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:01.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:01.877 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:01.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:01.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:02.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:02.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:02.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:02.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:02.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[41500]: 3.1f scrub starts 2024-03-24T12:29:02.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[41500]: 3.1f scrub ok 2024-03-24T12:29:02.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[41500]: 4.19 scrub starts 2024-03-24T12:29:02.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[41500]: 4.19 scrub ok 2024-03-24T12:29:02.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:02.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1895377896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:29:02.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[29919]: 3.1f scrub starts 2024-03-24T12:29:02.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[29919]: 3.1f scrub ok 2024-03-24T12:29:02.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[29919]: 4.19 scrub starts 2024-03-24T12:29:02.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[29919]: 4.19 scrub ok 2024-03-24T12:29:02.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:02.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:02 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1895377896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:29:02.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:02.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:02.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:02.286 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:02.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:02 smithi190 ceph-mon[33748]: 3.1f scrub starts 2024-03-24T12:29:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:02 smithi190 ceph-mon[33748]: 3.1f scrub ok 2024-03-24T12:29:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:02 smithi190 ceph-mon[33748]: 4.19 scrub starts 2024-03-24T12:29:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:02 smithi190 ceph-mon[33748]: 4.19 scrub ok 2024-03-24T12:29:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:02 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:02 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1895377896' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:29:02.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:02.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:02.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:02.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:02.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:02.548 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:02.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:02.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:02.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:02.682 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 49 16 75188 75172 95.861 72.375 0.00942129 0.0104265 2024-03-24T12:29:02.702 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:02.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:02.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:02.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:02.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:02.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:02.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:02.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:02.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:02.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:02.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:03.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:03.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:03.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:03.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:03.249 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:03.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:03.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:03 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:29:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:03 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:29:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:03 smithi190 ceph-mon[33748]: 5.d scrub starts 2024-03-24T12:29:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:03 smithi190 ceph-mon[33748]: 5.d scrub ok 2024-03-24T12:29:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:03 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:29:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:03 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:29:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:03 smithi190 ceph-mon[33748]: pgmap v94: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 884 op/s 2024-03-24T12:29:03.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:03.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:03.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:03.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:29:03.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:29:03.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[41500]: 5.d scrub starts 2024-03-24T12:29:03.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[41500]: 5.d scrub ok 2024-03-24T12:29:03.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:29:03.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:29:03.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[41500]: pgmap v94: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 884 op/s 2024-03-24T12:29:03.403 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:29:03.403 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:29:03.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[29919]: 5.d scrub starts 2024-03-24T12:29:03.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[29919]: 5.d scrub ok 2024-03-24T12:29:03.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:29:03.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:29:03.404 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:03 smithi029 ceph-mon[29919]: pgmap v94: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 884 op/s 2024-03-24T12:29:03.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:03.542 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:03.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:03.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:03.555 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:29:03.556 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:29:03.556 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:29:03.556 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:29:03.556 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:29:03.556 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:29:03.556 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:29:03.556 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:29:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:29:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:29:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:29:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:29:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:29:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:29:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:29:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:29:03.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:03.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:03.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:03.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:03.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 50 16 76441 76425 95.51 78.3125 0.0108346 0.0104649 2024-03-24T12:29:03.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:03.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:03.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:03.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:03.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:03.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:03.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:04.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:04.068 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:04.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:04.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:04 smithi029 ceph-mon[29919]: 5.1a scrub starts 2024-03-24T12:29:04.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:04 smithi029 ceph-mon[29919]: 5.1a scrub ok 2024-03-24T12:29:04.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:04 smithi029 ceph-mon[41500]: 5.1a scrub starts 2024-03-24T12:29:04.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:04 smithi029 ceph-mon[41500]: 5.1a scrub ok 2024-03-24T12:29:04.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:04.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:04.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:04.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:04.332 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:04.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:04.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:04.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:04 smithi190 ceph-mon[33748]: 5.1a scrub starts 2024-03-24T12:29:04.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:04 smithi190 ceph-mon[33748]: 5.1a scrub ok 2024-03-24T12:29:04.434 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:04.476 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:04.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:04.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:04.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:04.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:04.682 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 51 15 77691 77676 95.1702 78.1875 0.019935 0.010503 2024-03-24T12:29:04.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:04.738 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:04.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:04.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:04.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:04.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:04.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:04.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:05.035 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:05.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:05.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:05.062 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:29:05.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:05.167 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:05.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:05.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:05.265 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:05 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:29:05.265 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:05 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:29:05.265 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:05 smithi029 ceph-mon[41500]: 4.3 scrub starts 2024-03-24T12:29:05.265 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:05 smithi029 ceph-mon[41500]: 4.3 scrub ok 2024-03-24T12:29:05.265 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:05 smithi029 ceph-mon[41500]: pgmap v95: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T12:29:05.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:05 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:29:05.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:05 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:29:05.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:05 smithi029 ceph-mon[29919]: 4.3 scrub starts 2024-03-24T12:29:05.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:05 smithi029 ceph-mon[29919]: 4.3 scrub ok 2024-03-24T12:29:05.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:05 smithi029 ceph-mon[29919]: pgmap v95: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T12:29:05.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:05.299 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:05.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:05.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:05.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:05 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:29:05.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:05 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:29:05.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:05 smithi190 ceph-mon[33748]: 4.3 scrub starts 2024-03-24T12:29:05.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:05 smithi190 ceph-mon[33748]: 4.3 scrub ok 2024-03-24T12:29:05.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:05 smithi190 ceph-mon[33748]: pgmap v95: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T12:29:05.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:05.438 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:05.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:05.439 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:05.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:05.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:05.582 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:05.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:05.682 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 52 16 78842 78826 94.7222 71.875 0.00755948 0.010552 2024-03-24T12:29:05.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:05.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:05.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:05.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:05.859 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:05.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:05.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:05.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:05.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:06.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:06.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:06.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:06.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:06.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:06.260 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:06.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:06.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:06.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:06.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:06 smithi190 ceph-mon[33748]: 6.3 scrub starts 2024-03-24T12:29:06.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:06 smithi190 ceph-mon[33748]: 6.3 scrub ok 2024-03-24T12:29:06.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:06 smithi190 ceph-mon[33748]: 3.4 scrub starts 2024-03-24T12:29:06.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:06 smithi190 ceph-mon[33748]: 3.4 scrub ok 2024-03-24T12:29:06.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:06 smithi190 ceph-mon[33748]: 6.11 scrub starts 2024-03-24T12:29:06.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:06 smithi190 ceph-mon[33748]: 6.11 scrub ok 2024-03-24T12:29:06.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:06 smithi190 ceph-mon[33748]: 3.12 scrub starts 2024-03-24T12:29:06.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:06 smithi190 ceph-mon[33748]: 3.12 scrub ok 2024-03-24T12:29:06.405 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:06.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:06.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:06.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[41500]: 6.3 scrub starts 2024-03-24T12:29:06.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[41500]: 6.3 scrub ok 2024-03-24T12:29:06.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[41500]: 3.4 scrub starts 2024-03-24T12:29:06.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[41500]: 3.4 scrub ok 2024-03-24T12:29:06.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[41500]: 6.11 scrub starts 2024-03-24T12:29:06.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[41500]: 6.11 scrub ok 2024-03-24T12:29:06.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[41500]: 3.12 scrub starts 2024-03-24T12:29:06.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[41500]: 3.12 scrub ok 2024-03-24T12:29:06.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[29919]: 6.3 scrub starts 2024-03-24T12:29:06.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[29919]: 6.3 scrub ok 2024-03-24T12:29:06.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[29919]: 3.4 scrub starts 2024-03-24T12:29:06.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[29919]: 3.4 scrub ok 2024-03-24T12:29:06.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[29919]: 6.11 scrub starts 2024-03-24T12:29:06.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[29919]: 6.11 scrub ok 2024-03-24T12:29:06.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[29919]: 3.12 scrub starts 2024-03-24T12:29:06.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:06 smithi029 ceph-mon[29919]: 3.12 scrub ok 2024-03-24T12:29:06.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:06.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:06.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:06.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:29:06.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:06.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:06.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:06.682 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 53 16 79555 79539 93.7758 44.5625 0.0404471 0.0106583 2024-03-24T12:29:06.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:06.806 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:06.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:06.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:06.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:06.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:06.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:07.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:07.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:07.064 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:07.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:07.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:07.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:07.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:07.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:07.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:07.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:07.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:07 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:29:07.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:07 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:29:07.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:07 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:29:07.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:07 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:29:07.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:07 smithi190 ceph-mon[33748]: pgmap v96: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T12:29:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:29:07.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:29:07.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:29:07.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:29:07.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-mon[41500]: pgmap v96: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T12:29:07.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:29:07.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:29:07.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:29:07.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:29:07.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-mon[29919]: pgmap v96: 148 pgs: 148 active+clean; 4.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 976 op/s 2024-03-24T12:29:07.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:07.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:07.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:07.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:07.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:07.601 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:07.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:07.682 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 54 16 80418 80402 93.0381 53.9375 0.00586153 0.0107429 2024-03-24T12:29:07.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:07.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:07.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:07.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:07.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:07.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:07.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:29:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:29:07.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:29:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:29:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:29:07.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:08.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:08.011 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:08.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:08.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:08.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:08.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:08.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:08.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:08.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:08.268 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:08.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:08.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:08.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:08 smithi190 ceph-mon[33748]: 5.a scrub starts 2024-03-24T12:29:08.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:08 smithi190 ceph-mon[33748]: 5.a scrub ok 2024-03-24T12:29:08.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:08 smithi190 ceph-mon[33748]: 5.e scrub starts 2024-03-24T12:29:08.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:08 smithi190 ceph-mon[33748]: 5.e scrub ok 2024-03-24T12:29:08.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:08.396 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:08.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:08.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:08 smithi029 ceph-mon[29919]: 5.a scrub starts 2024-03-24T12:29:08.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:08 smithi029 ceph-mon[29919]: 5.a scrub ok 2024-03-24T12:29:08.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:08 smithi029 ceph-mon[29919]: 5.e scrub starts 2024-03-24T12:29:08.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:08 smithi029 ceph-mon[29919]: 5.e scrub ok 2024-03-24T12:29:08.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:08.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:08 smithi029 ceph-mon[41500]: 5.a scrub starts 2024-03-24T12:29:08.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:08 smithi029 ceph-mon[41500]: 5.a scrub ok 2024-03-24T12:29:08.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:08 smithi029 ceph-mon[41500]: 5.e scrub starts 2024-03-24T12:29:08.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:08 smithi029 ceph-mon[41500]: 5.e scrub ok 2024-03-24T12:29:08.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:08.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:08.523 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:08.523 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:08.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:08.661 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:08.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:08.682 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 55 16 81251 81235 92.2931 52.0625 0.0122483 0.0108298 2024-03-24T12:29:08.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:08.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:08.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:08.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:08.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:08.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:08.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:09.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:09.050 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:09.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:09.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:09.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:09.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:09.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:09.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:09.309 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:09.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:09.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:09.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[29919]: 5.b scrub starts 2024-03-24T12:29:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[29919]: 5.b scrub ok 2024-03-24T12:29:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:29:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:29:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:29:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:29:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[29919]: pgmap v97: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T12:29:09.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[41500]: 5.b scrub starts 2024-03-24T12:29:09.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[41500]: 5.b scrub ok 2024-03-24T12:29:09.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:29:09.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:29:09.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:29:09.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:29:09.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:09 smithi029 ceph-mon[41500]: pgmap v97: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T12:29:09.453 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:09.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:09.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:09.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:09.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:09.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:09.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:09 smithi190 ceph-mon[33748]: 5.b scrub starts 2024-03-24T12:29:09.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:09 smithi190 ceph-mon[33748]: 5.b scrub ok 2024-03-24T12:29:09.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:09 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:29:09.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:09 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:29:09.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:09 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:29:09.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:09 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:29:09.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:09 smithi190 ceph-mon[33748]: pgmap v97: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.3 KiB/s rd, 83 MiB/s wr, 1.33k op/s 2024-03-24T12:29:09.682 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 56 16 82141 82125 91.6383 55.625 0.0108952 0.0109072 2024-03-24T12:29:09.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:09.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:09.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:09.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:09.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:09.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:09.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:09.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:09.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:09.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:09.992 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:09.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:10.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:10.139 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:10.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:10.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:10.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:10.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:10.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:10.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:10.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:10.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:10.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:10.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:29:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:29:10.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[29919]: 3.14 scrub starts 2024-03-24T12:29:10.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[29919]: 3.14 scrub ok 2024-03-24T12:29:10.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[29919]: 3.1d scrub starts 2024-03-24T12:29:10.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[29919]: 3.1d scrub ok 2024-03-24T12:29:10.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:29:10.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:29:10.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[41500]: 3.14 scrub starts 2024-03-24T12:29:10.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[41500]: 3.14 scrub ok 2024-03-24T12:29:10.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[41500]: 3.1d scrub starts 2024-03-24T12:29:10.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:10 smithi029 ceph-mon[41500]: 3.1d scrub ok 2024-03-24T12:29:10.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:10.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:10 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:29:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:10 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:29:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:10 smithi190 ceph-mon[33748]: 3.14 scrub starts 2024-03-24T12:29:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:10 smithi190 ceph-mon[33748]: 3.14 scrub ok 2024-03-24T12:29:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:10 smithi190 ceph-mon[33748]: 3.1d scrub starts 2024-03-24T12:29:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:10 smithi190 ceph-mon[33748]: 3.1d scrub ok 2024-03-24T12:29:10.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:10.683 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:10.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:10.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:10.686 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 57 16 82970 82954 90.9343 51.8125 0.0253811 0.0109907 2024-03-24T12:29:10.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:10.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:10.851 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:10.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:10.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:10.998 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:10.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:11.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:11.140 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:11.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:11.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:11.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:11.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:11.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:11.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:11.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[29919]: 7.8 scrub starts 2024-03-24T12:29:11.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:29:11.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:29:11.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[29919]: 3.1 scrub starts 2024-03-24T12:29:11.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[29919]: pgmap v98: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 826 op/s 2024-03-24T12:29:11.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[29919]: 3.1 scrub ok 2024-03-24T12:29:11.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[41500]: 7.8 scrub starts 2024-03-24T12:29:11.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:29:11.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:29:11.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[41500]: 3.1 scrub starts 2024-03-24T12:29:11.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[41500]: pgmap v98: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 826 op/s 2024-03-24T12:29:11.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:11 smithi029 ceph-mon[41500]: 3.1 scrub ok 2024-03-24T12:29:11.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:11.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:11.559 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:11.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:11.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:11 smithi190 ceph-mon[33748]: 7.8 scrub starts 2024-03-24T12:29:11.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:11 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:29:11.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:11 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:29:11.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:11 smithi190 ceph-mon[33748]: 3.1 scrub starts 2024-03-24T12:29:11.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:11 smithi190 ceph-mon[33748]: pgmap v98: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 826 op/s 2024-03-24T12:29:11.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:11 smithi190 ceph-mon[33748]: 3.1 scrub ok 2024-03-24T12:29:11.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:11.686 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 58 16 83989 83973 90.4646 63.6875 0.0306 0.0110483 2024-03-24T12:29:11.689 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:11.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:11.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:11.820 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:11.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:11.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:11.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:11.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:11.952 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:11.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:12.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:12.082 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:12.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:12.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:12.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:12.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:12.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:12.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:12.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:12.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:12.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:12.415 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:29:12.415 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:29:12.415 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[41500]: 6.1e scrub starts 2024-03-24T12:29:12.415 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[41500]: 6.1e scrub ok 2024-03-24T12:29:12.415 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[41500]: 7.8 scrub ok 2024-03-24T12:29:12.415 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[41500]: 3.16 scrub starts 2024-03-24T12:29:12.415 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[41500]: 3.16 scrub ok 2024-03-24T12:29:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:29:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:29:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[29919]: 6.1e scrub starts 2024-03-24T12:29:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[29919]: 6.1e scrub ok 2024-03-24T12:29:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[29919]: 7.8 scrub ok 2024-03-24T12:29:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[29919]: 3.16 scrub starts 2024-03-24T12:29:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:12 smithi029 ceph-mon[29919]: 3.16 scrub ok 2024-03-24T12:29:12.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:12.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:12.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:12.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:12.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:12.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:12.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:12.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:12.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:12 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:29:12.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:12 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:29:12.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:12 smithi190 ceph-mon[33748]: 6.1e scrub starts 2024-03-24T12:29:12.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:12 smithi190 ceph-mon[33748]: 6.1e scrub ok 2024-03-24T12:29:12.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:12 smithi190 ceph-mon[33748]: 7.8 scrub ok 2024-03-24T12:29:12.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:12 smithi190 ceph-mon[33748]: 3.16 scrub starts 2024-03-24T12:29:12.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:12 smithi190 ceph-mon[33748]: 3.16 scrub ok 2024-03-24T12:29:12.686 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 59 16 85089 85073 90.0952 68.75 0.0113951 0.011094 2024-03-24T12:29:12.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:12.763 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:12.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:12.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:12.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:12.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:12.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:12.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:12.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:13.029 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:13.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:13.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:13.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:13.193 INFO:teuthology.orchestra.run.smithi029.stderr:set pool 7 pg_num to 26 2024-03-24T12:29:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:13.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:13.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:13.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:13.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:29:13.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:29:13.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:29:13.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:29:13.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: 3.d scrub starts 2024-03-24T12:29:13.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: 3.d scrub ok 2024-03-24T12:29:13.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2902436411' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:29:13.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: pgmap v99: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 971 op/s 2024-03-24T12:29:13.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: 6.d scrub starts 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: 6.d scrub ok 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[29919]: 7.d scrub starts 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: 3.d scrub starts 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: 3.d scrub ok 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2902436411' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: pgmap v99: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 971 op/s 2024-03-24T12:29:13.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: 6.d scrub starts 2024-03-24T12:29:13.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: 6.d scrub ok 2024-03-24T12:29:13.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:13 smithi029 ceph-mon[41500]: 7.d scrub starts 2024-03-24T12:29:13.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:13.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:13.495 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:13.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:13.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:13.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:13.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:29:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:29:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:29:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:29:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: 3.d scrub starts 2024-03-24T12:29:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: 3.d scrub ok 2024-03-24T12:29:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2902436411' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:29:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: pgmap v99: 148 pgs: 148 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 971 op/s 2024-03-24T12:29:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: 6.d scrub starts 2024-03-24T12:29:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: 6.d scrub ok 2024-03-24T12:29:13.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:13 smithi190 ceph-mon[33748]: 7.d scrub starts 2024-03-24T12:29:13.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:29:13.690621+0000 min lat: 0.00114673 max lat: 0.356207 avg lat: 0.0111568 2024-03-24T12:29:13.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:29:13.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 60 16 86047 86031 89.5915 59.875 0.0150275 0.0111568 2024-03-24T12:29:13.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:13.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:13.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:13.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:13.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:13.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:13.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:13.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:14.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:14.050 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:14.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:14.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:14.175 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:14.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:14.181 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:29:14.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:14.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:14.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:14.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:14.315 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:29:14.315 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:29:14.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:14.443 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:14.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:14.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:14.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:14.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:14.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:14.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:14 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:29:14.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:14 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:29:14.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:14 smithi190 ceph-mon[33748]: 6.1f scrub starts 2024-03-24T12:29:14.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:14 smithi190 ceph-mon[33748]: 6.1f scrub ok 2024-03-24T12:29:14.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:14 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2902436411' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:29:14.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:14 smithi190 ceph-mon[33748]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T12:29:14.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:14 smithi190 ceph-mon[33748]: 6.17 scrub starts 2024-03-24T12:29:14.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:14 smithi190 ceph-mon[33748]: 6.17 scrub ok 2024-03-24T12:29:14.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:14 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3010091487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:29:14.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:29:14.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[41500]: 6.1f scrub starts 2024-03-24T12:29:14.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[41500]: 6.1f scrub ok 2024-03-24T12:29:14.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2902436411' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:29:14.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[41500]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T12:29:14.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[41500]: 6.17 scrub starts 2024-03-24T12:29:14.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[41500]: 6.17 scrub ok 2024-03-24T12:29:14.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3010091487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:29:14.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:29:14.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[29919]: 6.1f scrub starts 2024-03-24T12:29:14.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[29919]: 6.1f scrub ok 2024-03-24T12:29:14.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2902436411' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:29:14.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[29919]: osdmap e107: 8 total, 8 up, 8 in 2024-03-24T12:29:14.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[29919]: 6.17 scrub starts 2024-03-24T12:29:14.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[29919]: 6.17 scrub ok 2024-03-24T12:29:14.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:14 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3010091487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:29:14.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:14.686 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 61 16 87010 86994 89.1095 60.1875 0.0111416 0.0112171 2024-03-24T12:29:14.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:14.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:14.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:14.729 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:29:14.729 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:29:14.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:14.847 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:14.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:14.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:14.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:14.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:14.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:14.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:15.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:15.113 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:15.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:15.114 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:15.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:15.238 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:15.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:15.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:15.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:15.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:15.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:15.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:15.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[41500]: 7.d scrub ok 2024-03-24T12:29:15.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:29:15.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[41500]: Cluster is now healthy 2024-03-24T12:29:15.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3010091487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:29:15.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[41500]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T12:29:15.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[41500]: pgmap v102: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-24T12:29:15.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:29:15.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[41500]: 7.7 scrub starts 2024-03-24T12:29:15.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[41500]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[29919]: 7.d scrub ok 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[29919]: Cluster is now healthy 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3010091487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[29919]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[29919]: pgmap v102: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[29919]: 7.7 scrub starts 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:29:15.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:15 smithi029 ceph-mon[29919]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T12:29:15.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:15.488 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:15.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:15.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:15.614 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:15.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:15.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:15.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:15 smithi190 ceph-mon[33748]: 7.d scrub ok 2024-03-24T12:29:15.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:15 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:29:15.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:15 smithi190 ceph-mon[33748]: Cluster is now healthy 2024-03-24T12:29:15.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:15 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3010091487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:29:15.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:15 smithi190 ceph-mon[33748]: osdmap e108: 8 total, 8 up, 8 in 2024-03-24T12:29:15.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:15 smithi190 ceph-mon[33748]: pgmap v102: 148 pgs: 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-24T12:29:15.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:15 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:29:15.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:15 smithi190 ceph-mon[33748]: 7.7 scrub starts 2024-03-24T12:29:15.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:15 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:29:15.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:15 smithi190 ceph-mon[33748]: osdmap e109: 8 total, 8 up, 8 in 2024-03-24T12:29:15.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 62 16 87217 87201 87.8811 12.9375 0.0027152 0.0112394 2024-03-24T12:29:15.716 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:15.741 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:15.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:15.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:15.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:15.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:15.993 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:15.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:16.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:16.127 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:16.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:16.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:16.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:16.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:16.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:16.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:16.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:16.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:16.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:16.513 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:16.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:16.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:16.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:16.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:16 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:29:16.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:16 smithi190 ceph-mon[33748]: 4.4 scrub ok 2024-03-24T12:29:16.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:16 smithi190 ceph-mon[33748]: 4.14 scrub starts 2024-03-24T12:29:16.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:16 smithi190 ceph-mon[33748]: 4.14 scrub ok 2024-03-24T12:29:16.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:16 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:29:16.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:16 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:29:16.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:16 smithi190 ceph-mon[33748]: 6.a scrub starts 2024-03-24T12:29:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:16 smithi190 ceph-mon[33748]: 6.a scrub ok 2024-03-24T12:29:16.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:16 smithi190 ceph-mon[33748]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T12:29:16.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:16.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:16.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:29:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[41500]: 4.4 scrub ok 2024-03-24T12:29:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[41500]: 4.14 scrub starts 2024-03-24T12:29:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[41500]: 4.14 scrub ok 2024-03-24T12:29:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:29:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:29:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[41500]: 6.a scrub starts 2024-03-24T12:29:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[41500]: 6.a scrub ok 2024-03-24T12:29:16.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[41500]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T12:29:16.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:29:16.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[29919]: 4.4 scrub ok 2024-03-24T12:29:16.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[29919]: 4.14 scrub starts 2024-03-24T12:29:16.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[29919]: 4.14 scrub ok 2024-03-24T12:29:16.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:29:16.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:29:16.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[29919]: 6.a scrub starts 2024-03-24T12:29:16.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[29919]: 6.a scrub ok 2024-03-24T12:29:16.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:16 smithi029 ceph-mon[29919]: osdmap e110: 8 total, 8 up, 8 in 2024-03-24T12:29:16.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 63 16 87709 87693 86.9743 30.75 0.0173603 0.0114916 2024-03-24T12:29:16.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:16.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:16.794 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:16.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:16.895 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:16.936 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:16.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:17.038 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:17.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:17.079 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:17.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:17.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:17.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:17.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:17.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:17.343 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:17.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:17.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:17.470 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:17.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:17.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: 7.e scrub starts 2024-03-24T12:29:17.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: 7.5 scrub starts 2024-03-24T12:29:17.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: 7.1 scrub starts 2024-03-24T12:29:17.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: 7.2 scrub starts 2024-03-24T12:29:17.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: pgmap v105: 158 pgs: 10 unknown, 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 45 MiB/s wr, 712 op/s 2024-03-24T12:29:17.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:29:17.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: 7.7 scrub starts 2024-03-24T12:29:17.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: 7.0 scrub starts 2024-03-24T12:29:17.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:29:17.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:17.531 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-mon[33748]: 7.1 scrub ok 2024-03-24T12:29:17.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:17.595 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:17.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:17.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:17.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: 7.e scrub starts 2024-03-24T12:29:17.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: 7.5 scrub starts 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: 7.1 scrub starts 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: 7.2 scrub starts 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: pgmap v105: 158 pgs: 10 unknown, 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 45 MiB/s wr, 712 op/s 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: 7.7 scrub starts 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: 7.0 scrub starts 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[29919]: 7.1 scrub ok 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: 7.e scrub starts 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: 7.5 scrub starts 2024-03-24T12:29:17.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: 7.1 scrub starts 2024-03-24T12:29:17.608 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: 7.2 scrub starts 2024-03-24T12:29:17.608 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: pgmap v105: 158 pgs: 10 unknown, 148 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 45 MiB/s wr, 712 op/s 2024-03-24T12:29:17.608 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:29:17.608 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: 7.7 scrub starts 2024-03-24T12:29:17.608 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: 7.0 scrub starts 2024-03-24T12:29:17.608 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:29:17.608 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:17.608 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-mon[41500]: 7.1 scrub ok 2024-03-24T12:29:17.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 64 16 88150 88134 86.0462 27.5625 0.0189815 0.0115377 2024-03-24T12:29:17.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:17.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:17.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:17.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:17.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:17.853 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:17.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:17.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:29:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:29:17.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:29:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:29:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:29:17.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:17.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:17.982 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:18.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:18.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:18.109 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:18.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:18.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:18.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:18.244 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:18.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:18.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:18.370 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:18.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:18.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:18.496 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:18.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:18.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:18.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:18.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:18.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.9 scrub starts 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.8 deep-scrub starts 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.5 scrub ok 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.15 scrub starts 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.3 scrub starts 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 5.19 scrub starts 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.7 scrub ok 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 5.19 scrub ok 2024-03-24T12:29:18.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.0 scrub ok 2024-03-24T12:29:18.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.9 scrub ok 2024-03-24T12:29:18.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.2 scrub ok 2024-03-24T12:29:18.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.4 scrub starts 2024-03-24T12:29:18.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:18.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:18 smithi190 ceph-mon[33748]: 7.e scrub ok 2024-03-24T12:29:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.9 scrub starts 2024-03-24T12:29:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.8 deep-scrub starts 2024-03-24T12:29:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.5 scrub ok 2024-03-24T12:29:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.15 scrub starts 2024-03-24T12:29:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:29:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T12:29:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.3 scrub starts 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 5.19 scrub starts 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.7 scrub ok 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 5.19 scrub ok 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.0 scrub ok 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.9 scrub ok 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.2 scrub ok 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.4 scrub starts 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[29919]: 7.e scrub ok 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.9 scrub starts 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.8 deep-scrub starts 2024-03-24T12:29:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.5 scrub ok 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.15 scrub starts 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: osdmap e111: 8 total, 8 up, 8 in 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.3 scrub starts 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 5.19 scrub starts 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.7 scrub ok 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 5.19 scrub ok 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.0 scrub ok 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.9 scrub ok 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.2 scrub ok 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.4 scrub starts 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:18 smithi029 ceph-mon[41500]: 7.e scrub ok 2024-03-24T12:29:18.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 65 16 88150 88134 84.7226 0 - 0.0115377 2024-03-24T12:29:18.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:18.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:18.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:18.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:18.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:18.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:18.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:18.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:19.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:19.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:19.017 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:29:19.017 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:29:19.017 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-24T12:29:19.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:29:19.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:19.138 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:19.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:19.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:19.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:19.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:19.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:19.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:19.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:19.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:19.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:19.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[29919]: 7.3 scrub ok 2024-03-24T12:29:19.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[29919]: 7.4 scrub ok 2024-03-24T12:29:19.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[29919]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T12:29:19.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[29919]: pgmap v108: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+clean+scrubbing, 144 active+clean, 10 unknown; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:29:19.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[29919]: 7.6 deep-scrub starts 2024-03-24T12:29:19.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[29919]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T12:29:19.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[41500]: 7.3 scrub ok 2024-03-24T12:29:19.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[41500]: 7.4 scrub ok 2024-03-24T12:29:19.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[41500]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T12:29:19.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[41500]: pgmap v108: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+clean+scrubbing, 144 active+clean, 10 unknown; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:29:19.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[41500]: 7.6 deep-scrub starts 2024-03-24T12:29:19.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:19 smithi029 ceph-mon[41500]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T12:29:19.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:19.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:19.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:19.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:19.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:19 smithi190 ceph-mon[33748]: 7.3 scrub ok 2024-03-24T12:29:19.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:19 smithi190 ceph-mon[33748]: 7.4 scrub ok 2024-03-24T12:29:19.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:19 smithi190 ceph-mon[33748]: osdmap e112: 8 total, 8 up, 8 in 2024-03-24T12:29:19.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:19 smithi190 ceph-mon[33748]: pgmap v108: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+clean+scrubbing, 144 active+clean, 10 unknown; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:29:19.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:19 smithi190 ceph-mon[33748]: 7.6 deep-scrub starts 2024-03-24T12:29:19.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:19 smithi190 ceph-mon[33748]: osdmap e113: 8 total, 8 up, 8 in 2024-03-24T12:29:19.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:19.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:19.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:19.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 66 16 88232 88216 83.5168 2.5625 0.0244213 0.0119648 2024-03-24T12:29:19.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:19.823 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:19.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:19.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:19.835 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:19 smithi029 systemd[1]: Stopping Ceph alertmanager.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:29:19.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T12:29:19.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:19.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:20.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:20.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:20.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:20.138 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:29:20.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:20.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:20.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:20.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:20.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:20.442 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:20.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:20.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:20.596 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:20.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:20.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:20.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:20 smithi190 ceph-mon[33748]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 110'5551 2024-03-24T12:29:20.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:20 smithi190 ceph-mon[33748]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 110'5559 2024-03-24T12:29:20.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:20 smithi190 ceph-mon[33748]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 110'5537 2024-03-24T12:29:20.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:20 smithi190 ceph-mon[33748]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 110'5559 2024-03-24T12:29:20.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:20 smithi190 ceph-mon[33748]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 110'5522 2024-03-24T12:29:20.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:20 smithi190 ceph-mon[33748]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 110'5537 2024-03-24T12:29:20.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:20 smithi190 ceph-mon[33748]: 5.18 scrub starts 2024-03-24T12:29:20.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:20 smithi190 ceph-mon[33748]: 5.18 scrub ok 2024-03-24T12:29:20.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[41500]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 110'5551 2024-03-24T12:29:20.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[41500]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 110'5559 2024-03-24T12:29:20.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[41500]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 110'5537 2024-03-24T12:29:20.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[41500]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 110'5559 2024-03-24T12:29:20.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[41500]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 110'5522 2024-03-24T12:29:20.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[41500]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 110'5537 2024-03-24T12:29:20.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[41500]: 5.18 scrub starts 2024-03-24T12:29:20.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[41500]: 5.18 scrub ok 2024-03-24T12:29:20.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[29919]: 7.15 starting backfill to osd.1 from (0'0,0'0] MAX to 110'5551 2024-03-24T12:29:20.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[29919]: 7.13 starting backfill to osd.0 from (0'0,0'0] MAX to 110'5559 2024-03-24T12:29:20.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[29919]: 7.12 starting backfill to osd.0 from (0'0,0'0] MAX to 110'5537 2024-03-24T12:29:20.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[29919]: 7.13 starting backfill to osd.7 from (0'0,0'0] MAX to 110'5559 2024-03-24T12:29:20.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[29919]: 7.11 starting backfill to osd.7 from (0'0,0'0] MAX to 110'5522 2024-03-24T12:29:20.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[29919]: 7.12 starting backfill to osd.6 from (0'0,0'0] MAX to 110'5537 2024-03-24T12:29:20.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[29919]: 5.18 scrub starts 2024-03-24T12:29:20.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:20 smithi029 ceph-mon[29919]: 5.18 scrub ok 2024-03-24T12:29:20.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 67 16 88785 88769 82.7862 34.5625 0.0277218 0.0120706 2024-03-24T12:29:20.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:29:20.765 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:20.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:20.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:20.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:20.961 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:29:21.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:21.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:21.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:21.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:21.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:21.208 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:21.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:21.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:21.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:21.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:21.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:21.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:21.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:21 smithi029 ceph-mon[29919]: pgmap v110: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+clean+scrubbing, 144 active+clean, 10 unknown; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:29:21.361 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:21 smithi029 ceph-mon[41500]: pgmap v110: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+clean+scrubbing, 144 active+clean, 10 unknown; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:29:21.361 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:21 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[79813]: ts=2024-03-24T12:29:21.234Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T12:29:21.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:21.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:21.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:21.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:21.620 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:21.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:21.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:21.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:21 smithi190 ceph-mon[33748]: pgmap v110: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+clean+scrubbing, 144 active+clean, 10 unknown; 4.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:29:21.668 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:21 smithi029 podman[90429]: 2024-03-24 12:29:21.375184173 +0000 UTC m=+0.589252335 container died 28b4d8a961099913ae6effbb628288a788d351cbdaa1c7ad2a330f79001f6c30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:29:21.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 68 16 89365 89349 82.1019 36.25 0.00551216 0.0121681 2024-03-24T12:29:21.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:21.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:21.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:21.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:21.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:22.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:22.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:22.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:22.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:22.219 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:22.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:22.261 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:22 smithi029 podman[90429]: 2024-03-24 12:29:22.086084357 +0000 UTC m=+1.300152518 container cleanup 28b4d8a961099913ae6effbb628288a788d351cbdaa1c7ad2a330f79001f6c30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:29:22.261 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:22 smithi029 bash[90429]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T12:29:22.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:22.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:22.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:22.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:22.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:22.483 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:22.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:22.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:22.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:22.608 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:22.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:22.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:22.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:22 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:29:22.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:22 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:29:22.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:22 smithi190 ceph-mon[33748]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T12:29:22.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:22 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:29:22.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:22 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:29:22.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:22 smithi029 ceph-mon[29919]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T12:29:22.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:22 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:29:22.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:22 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:29:22.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:22 smithi029 ceph-mon[41500]: osdmap e114: 8 total, 8 up, 8 in 2024-03-24T12:29:22.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 69 16 89903 89887 81.3994 33.625 0.168804 0.0122649 2024-03-24T12:29:22.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:22.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:22.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:22.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:29:22.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:22.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:22.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:22.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:23.035 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:23.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:23.136 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:23.190 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:23.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:23.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:23.323 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:23.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:23.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:23.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:23.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:23.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:23.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:23.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:23.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:23.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:23.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:23 smithi190 ceph-mon[33748]: pgmap v112: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+clean+scrubbing, 144 active+clean, 1 active+remapped+backfilling, 9 unknown; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 2311/200376 objects misplaced (1.153%); 6.3 MiB/s, 101 objects/s recovering 2024-03-24T12:29:23.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:23 smithi029 ceph-mon[41500]: pgmap v112: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+clean+scrubbing, 144 active+clean, 1 active+remapped+backfilling, 9 unknown; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 2311/200376 objects misplaced (1.153%); 6.3 MiB/s, 101 objects/s recovering 2024-03-24T12:29:23.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:23 smithi029 ceph-mon[29919]: pgmap v112: 158 pgs: 1 active+clean+scrubbing+deep, 3 active+clean+scrubbing, 144 active+clean, 1 active+remapped+backfilling, 9 unknown; 4.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.9 KiB/s rd, 1 op/s; 2311/200376 objects misplaced (1.153%); 6.3 MiB/s, 101 objects/s recovering 2024-03-24T12:29:23.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 70 16 90524 90508 80.791 38.8125 0.0147021 0.0123696 2024-03-24T12:29:23.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:23.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:23.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:23.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:23.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:23.860 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:23.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:23.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:23.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:23.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:24.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:24.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:24.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:24.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:24.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:24.245 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:24.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:24.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:24.391 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:24.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:24.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:24.536 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:24.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:24.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:24.625 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:24 smithi029 ceph-mon[41500]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:29:24.626 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:24 smithi029 ceph-mon[41500]: Health check failed: Degraded data redundancy: 5498/270672 objects degraded (2.031%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:24.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:24 smithi029 ceph-mon[29919]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:29:24.626 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:24 smithi029 ceph-mon[29919]: Health check failed: Degraded data redundancy: 5498/270672 objects degraded (2.031%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:24.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:24.669 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:24.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:24.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 71 16 91040 91024 80.1074 32.25 0.00518956 0.0124749 2024-03-24T12:29:24.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:24.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:24.802 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:24.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:24.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:24 smithi190 ceph-mon[33748]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:29:24.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:24 smithi190 ceph-mon[33748]: Health check failed: Degraded data redundancy: 5498/270672 objects degraded (2.031%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:24.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:24.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:24.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:25.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:25.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:25.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:25.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:25.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:25.208 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:25.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:25.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:25.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:25.382 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:25.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:25.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:25.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:25.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:25.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:25.642 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:25.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:25.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:25.688 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 72 16 91610 91594 79.4897 35.625 0.0126153 0.0125624 2024-03-24T12:29:25.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:25.785 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:25.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:25.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:25.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:25.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:25.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:26.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:26.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:26.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:26.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:25 smithi029 ceph-mon[41500]: pgmap v113: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 824 B/s rd, 184 MiB/s wr, 2.94k op/s; 5498/270672 objects degraded (2.031%); 45327/270672 objects misplaced (16.746%); 11 MiB/s, 177 objects/s recovering 2024-03-24T12:29:26.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:25 smithi029 ceph-mon[41500]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T12:29:26.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:25 smithi029 ceph-mon[29919]: pgmap v113: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 824 B/s rd, 184 MiB/s wr, 2.94k op/s; 5498/270672 objects degraded (2.031%); 45327/270672 objects misplaced (16.746%); 11 MiB/s, 177 objects/s recovering 2024-03-24T12:29:26.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:25 smithi029 ceph-mon[29919]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T12:29:26.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:26.198 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:26.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:26.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:26.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:26.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:26.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:26.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:25 smithi190 ceph-mon[33748]: pgmap v113: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 824 B/s rd, 184 MiB/s wr, 2.94k op/s; 5498/270672 objects degraded (2.031%); 45327/270672 objects misplaced (16.746%); 11 MiB/s, 177 objects/s recovering 2024-03-24T12:29:26.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:25 smithi190 ceph-mon[33748]: osdmap e115: 8 total, 8 up, 8 in 2024-03-24T12:29:26.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:26.491 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:26.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:26.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:26.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:26.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:26.688 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 73 16 92100 92084 78.8203 30.625 0.0215676 0.0126785 2024-03-24T12:29:26.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:26.751 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:26.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:26.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:26.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:26.881 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:26.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:26.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:26.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:27.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:27.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:27.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:27.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:27.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:27.352 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:27.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:27.418 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:27 smithi029 podman[90525]: 2024-03-24 12:29:27.322885162 +0000 UTC m=+6.034682612 container remove 28b4d8a961099913ae6effbb628288a788d351cbdaa1c7ad2a330f79001f6c30 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:29:27.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:27.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:27.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:27.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:27.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:27.641 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:27.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:27.688 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 74 16 92723 92707 78.2814 38.9375 0.0243443 0.0127687 2024-03-24T12:29:27.738 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:29:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:29:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:29:27.738 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:27 smithi029 podman[90525]: 2024-03-24 12:29:27.424988256 +0000 UTC m=+6.136785699 volume remove 583749c3a572e709713b654ec197f91e4522282999499df624e34bb043c56c65 2024-03-24T12:29:27.738 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:27 smithi029 bash[91050]: Error: no container with name or ID "ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a" found: no such container 2024-03-24T12:29:27.739 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:27 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T12:29:27.739 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:27 smithi029 systemd[1]: Stopped Ceph alertmanager.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:29:27.739 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:27 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.228s CPU time. 2024-03-24T12:29:27.739 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:27 smithi029 systemd[1]: Starting Ceph alertmanager.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:29:27.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:27.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:27.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:27.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:27.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:29:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:29:27.936 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:27.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:27.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:28.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:28.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:28.068 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:28.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:28.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:27 smithi029 ceph-mon[41500]: pgmap v115: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 710 B/s rd, 159 MiB/s wr, 2.54k op/s; 5498/270672 objects degraded (2.031%); 45327/270672 objects misplaced (16.746%); 9.5 MiB/s, 152 objects/s recovering 2024-03-24T12:29:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:27 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:29:28.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:27 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:29:28.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:27 smithi029 ceph-mon[29919]: pgmap v115: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 710 B/s rd, 159 MiB/s wr, 2.54k op/s; 5498/270672 objects degraded (2.031%); 45327/270672 objects misplaced (16.746%); 9.5 MiB/s, 152 objects/s recovering 2024-03-24T12:29:28.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:27 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:29:28.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:27 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:29:28.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:28.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:28.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:28.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:28.352 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:28.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:28.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:28.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:27 smithi190 ceph-mon[33748]: pgmap v115: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 710 B/s rd, 159 MiB/s wr, 2.54k op/s; 5498/270672 objects degraded (2.031%); 45327/270672 objects misplaced (16.746%); 9.5 MiB/s, 152 objects/s recovering 2024-03-24T12:29:28.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:27 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:29:28.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:27 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:29:28.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:28.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:28.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:28.549 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:29:28.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:28.609 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:29:28.612 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:28.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:28.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:28.652 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:29:28.689 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 75 16 93307 93291 77.723 36.5 0.0197191 0.0128587 2024-03-24T12:29:28.705 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:29:28.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:28.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:28.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:28.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:28.747 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:29:28.790 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:29:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:29:28.833 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:29:28.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:28.887 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:29:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:28.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:29:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:29:28.900 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:29:28.901 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:29:28.901 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:29:28.901 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:29:28.901 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:29:28.901 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:29:29.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:29.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:29.058 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[41500]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T12:29:29.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[41500]: 7.8 deep-scrub ok 2024-03-24T12:29:29.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:29:29.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:29:29.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1740094538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:29:29.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:29.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[29919]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T12:29:29.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[29919]: 7.8 deep-scrub ok 2024-03-24T12:29:29.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:29:29.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:29:29.059 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:28 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1740094538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:29:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:29.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:29.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:29.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:29.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:29.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:29.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:29.293 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:29:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:29:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-03-24T12:29:29.293 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-03-24T12:29:29.293 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:29:29.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:29.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:29.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:28 smithi190 ceph-mon[33748]: osdmap e116: 8 total, 8 up, 8 in 2024-03-24T12:29:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:28 smithi190 ceph-mon[33748]: 7.8 deep-scrub ok 2024-03-24T12:29:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:28 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:29:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:28 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:29:29.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:28 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1740094538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:29:29.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:29.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:29.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:29.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:29.618 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:29.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:29.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:29.689 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 76 16 93848 93832 77.1452 33.8125 0.0812036 0.0129474 2024-03-24T12:29:29.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:29.761 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:29.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:29.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:29.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:29.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:29.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:29.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:29.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:30.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:30.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:30.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:30.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:30.151 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:30.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:30.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:30.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:30.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:30.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:30.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:29 smithi190 ceph-mon[33748]: pgmap v117: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 184 MiB/s wr, 2.94k op/s; 5498/278979 objects degraded (1.971%); 43158/278979 objects misplaced (15.470%); 24 MiB/s, 389 objects/s recovering 2024-03-24T12:29:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:29 smithi190 ceph-mon[33748]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T12:29:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:29 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 5498/278979 objects degraded (1.971%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:30.410 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:30.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:30.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:30.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:29 smithi029 ceph-mon[29919]: pgmap v117: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 184 MiB/s wr, 2.94k op/s; 5498/278979 objects degraded (1.971%); 43158/278979 objects misplaced (15.470%); 24 MiB/s, 389 objects/s recovering 2024-03-24T12:29:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:29 smithi029 ceph-mon[29919]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T12:29:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:29 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 5498/278979 objects degraded (1.971%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:30.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:29 smithi029 ceph-mon[41500]: pgmap v117: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.4 KiB/s rd, 184 MiB/s wr, 2.94k op/s; 5498/278979 objects degraded (1.971%); 43158/278979 objects misplaced (15.470%); 24 MiB/s, 389 objects/s recovering 2024-03-24T12:29:30.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:29 smithi029 ceph-mon[41500]: osdmap e117: 8 total, 8 up, 8 in 2024-03-24T12:29:30.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:29 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 5498/278979 objects degraded (1.971%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:30.513 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:30.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:30.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:30.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:30.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:30.666 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:30.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:30.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 77 16 94407 94391 76.5971 34.9375 0.00264981 0.0130206 2024-03-24T12:29:30.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:30.810 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:30.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:30.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:30.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:30.961 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:30.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:31.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:31.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:31.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:31.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:31.266 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:31.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:31.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:31.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:31.391 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:31.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:31.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:31.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:31.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:31.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:31.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:31.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:31.689 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 78 16 94870 94854 75.9861 28.9375 0.0114081 0.0131515 2024-03-24T12:29:31.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:31.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:31.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:31.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:32.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:32.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:32.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:32.163 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:32.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:32.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:32.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:32.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:32.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:32.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:31 smithi190 ceph-mon[33748]: pgmap v119: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 29 MiB/s wr, 462 op/s; 5498/278979 objects degraded (1.971%); 43158/278979 objects misplaced (15.470%); 18 MiB/s, 281 objects/s recovering 2024-03-24T12:29:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:31 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:29:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:31 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:29:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:31 smithi190 ceph-mon[33748]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T12:29:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:29:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:32.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[29919]: pgmap v119: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 29 MiB/s wr, 462 op/s; 5498/278979 objects degraded (1.971%); 43158/278979 objects misplaced (15.470%); 18 MiB/s, 281 objects/s recovering 2024-03-24T12:29:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:29:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:29:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[29919]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T12:29:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:29:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:32.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[41500]: pgmap v119: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 29 MiB/s wr, 462 op/s; 5498/278979 objects degraded (1.971%); 43158/278979 objects misplaced (15.470%); 18 MiB/s, 281 objects/s recovering 2024-03-24T12:29:32.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:29:32.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:29:32.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[41500]: osdmap e118: 8 total, 8 up, 8 in 2024-03-24T12:29:32.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:29:32.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:32.445 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:32.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:32.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:32.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:32.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:32.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:32.576 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:32.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:32.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 79 16 95393 95377 75.4381 32.6875 0.00438019 0.0132324 2024-03-24T12:29:32.702 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:32.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:32.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:32.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:32.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:32.832 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:32.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:32.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:32.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:32.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:33.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:33.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:33.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:33.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:33.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:33.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:33.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:33.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:33.360 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:33.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:33.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:33.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:33.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:33.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:33.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:33.634 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:33.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:33.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:33.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:29:33.693845+0000 min lat: 0.00114673 max lat: 2.39623 avg lat: 0.0133496 2024-03-24T12:29:33.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:29:33.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 80 16 95861 95845 74.8608 29.25 0.00896099 0.0133496 2024-03-24T12:29:33.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:33.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:33.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:33.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:33.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:33.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:34.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:34.029 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:34.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:34.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:34.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:34.168 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:34.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:34.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:34.204 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:33 smithi029 ceph-mon[29919]: pgmap v121: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 40 MiB/s wr, 643 op/s; 5498/282225 objects degraded (1.948%); 41696/282225 objects misplaced (14.774%); 30 MiB/s, 479 objects/s recovering 2024-03-24T12:29:34.204 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:33 smithi029 ceph-mon[41500]: pgmap v121: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 40 MiB/s wr, 643 op/s; 5498/282225 objects degraded (1.948%); 41696/282225 objects misplaced (14.774%); 30 MiB/s, 479 objects/s recovering 2024-03-24T12:29:34.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:34.298 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:34.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:34.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:33 smithi190 ceph-mon[33748]: pgmap v121: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 2 active+clean+scrubbing+deep, 146 active+clean, 2 active+remapped+backfilling; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 40 MiB/s wr, 643 op/s; 5498/282225 objects degraded (1.948%); 41696/282225 objects misplaced (14.774%); 30 MiB/s, 479 objects/s recovering 2024-03-24T12:29:34.400 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:34.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:34.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:34.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:34.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:34.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:34.551 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:34.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:34.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:34.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 81 16 96220 96204 74.2137 22.4375 0.0180139 0.013467 2024-03-24T12:29:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:34.700 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:34.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:34.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:34.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:34.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:34.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:34.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:34.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:34.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:35.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:35.086 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:35.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:35.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:35.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:35.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:35.217 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:35.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:35.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:35.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:35.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:34 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1855518768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:29:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:34 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 5498/282225 objects degraded (1.948%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:34 smithi190 ceph-mon[33748]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T12:29:35.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:34 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1855518768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:29:35.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:34 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 5498/282225 objects degraded (1.948%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:35.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:34 smithi029 ceph-mon[41500]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T12:29:35.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:34 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1855518768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:29:35.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:34 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 5498/282225 objects degraded (1.948%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:35.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:34 smithi029 ceph-mon[29919]: osdmap e119: 8 total, 8 up, 8 in 2024-03-24T12:29:35.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:35.493 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:35.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:35.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:35.595 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:35.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:35.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:35.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 82 16 96700 96684 73.6745 30 0.00539164 0.0135624 2024-03-24T12:29:35.726 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:35.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:35.770 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:35.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:35.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:35.915 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:35.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:36.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:36.040 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:36.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:36.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:36.165 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:36.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:36.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:36.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:36.292 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:36.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:35 smithi190 ceph-mon[33748]: pgmap v122: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean, 2 active+remapped+backfilling; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 784 B/s rd, 0 op/s; 5498/287253 objects degraded (1.914%); 41025/287253 objects misplaced (14.282%); 16 MiB/s, 253 objects/s recovering 2024-03-24T12:29:36.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:36.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:35 smithi029 ceph-mon[41500]: pgmap v122: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean, 2 active+remapped+backfilling; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 784 B/s rd, 0 op/s; 5498/287253 objects degraded (1.914%); 41025/287253 objects misplaced (14.282%); 16 MiB/s, 253 objects/s recovering 2024-03-24T12:29:36.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:35 smithi029 ceph-mon[29919]: pgmap v122: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean, 2 active+remapped+backfilling; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 784 B/s rd, 0 op/s; 5498/287253 objects degraded (1.914%); 41025/287253 objects misplaced (14.282%); 16 MiB/s, 253 objects/s recovering 2024-03-24T12:29:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:36.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:36.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:36.574 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:36.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:36.575 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:36.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:36.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 83 16 97071 97055 73.0663 23.1875 0.0100901 0.0136672 2024-03-24T12:29:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:36.700 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:36.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:36.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:36.826 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:36.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:36.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:36.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:37.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:37.102 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:37.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:37.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:37.245 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:37.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:37.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:37.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:37.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:36 smithi190 ceph-mon[33748]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T12:29:37.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:36 smithi190 ceph-mon[33748]: pgmap v125: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean, 2 active+remapped+backfilling; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 5498/287253 objects degraded (1.914%); 41025/287253 objects misplaced (14.282%); 17 MiB/s, 274 objects/s recovering 2024-03-24T12:29:37.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:36 smithi029 ceph-mon[41500]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T12:29:37.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:36 smithi029 ceph-mon[41500]: pgmap v125: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean, 2 active+remapped+backfilling; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 5498/287253 objects degraded (1.914%); 41025/287253 objects misplaced (14.282%); 17 MiB/s, 274 objects/s recovering 2024-03-24T12:29:37.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:36 smithi029 ceph-mon[29919]: osdmap e120: 8 total, 8 up, 8 in 2024-03-24T12:29:37.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:36 smithi029 ceph-mon[29919]: pgmap v125: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 1 active+clean+scrubbing+deep, 147 active+clean, 2 active+remapped+backfilling; 5.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 5498/287253 objects degraded (1.914%); 41025/287253 objects misplaced (14.282%); 17 MiB/s, 274 objects/s recovering 2024-03-24T12:29:37.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:37.516 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:37.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:37.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:37.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:37.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:37.668 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:37.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:37.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 84 16 97407 97391 72.4466 21 0.0496467 0.0137967 2024-03-24T12:29:37.729 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 118, 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout: "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout: "created": "2024-03-24T12:20:18.712998+0000", 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout: "modified": "2024-03-24T12:29:30.941018+0000", 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout: "last_up_change": "2024-03-24T12:25:48.676565+0000", 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout: "last_in_change": "2024-03-24T12:25:30.369664+0000", 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_num": 5799936, 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_set": [ 2024-03-24T12:29:37.730 INFO:teuthology.orchestra.run.smithi029.stdout: "pglog_hardlimit", 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "purged_snapdirs", 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_deletes", 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "sortbitwise" 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_version": 18, 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_snapshot": "", 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_max": 7, 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd": 8, 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "min_compat_client": "luminous", 2024-03-24T12:29:37.731 INFO:teuthology.orchestra.run.smithi029.stdout: "require_osd_release": "squid", 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_crimson": false, 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "pools": [ 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 1, 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".mgr", 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:23:41.491597+0000", 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:29:37.732 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 1, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 1, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 1, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 1, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 1, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:29:37.733 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "22", 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:29:37.734 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:29:37.735 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:29:37.736 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:29:37.736 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:29:37.736 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:29:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:29:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:29:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:29:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:29:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:29:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:29:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:29:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:29:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_max": 32, 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_min": 1 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": {} 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:29:37.738 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 7.8899998664855957, 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 7.8899998664855957, 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.37999999523162842, 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 3, 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 3, 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.739 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 2, 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "datapool", 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:01.465080+0000", 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 8193, 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:29:37.740 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:29:37.741 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:29:37.741 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 3, 2024-03-24T12:29:37.741 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 3, 2024-03-24T12:29:37.741 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 3, 2024-03-24T12:29:37.741 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 3, 2024-03-24T12:29:37.741 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 3, 2024-03-24T12:29:37.741 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:29:37.742 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:29:37.743 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:29:37.744 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:29:37.744 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:29:37.744 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:29:37.744 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:29:37.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.744 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "71", 2024-03-24T12:29:37.745 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:29:37.745 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:29:37.745 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:29:37.745 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:29:37.745 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:29:37.745 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 2, 2024-03-24T12:29:37.746 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 71, 2024-03-24T12:29:37.746 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:29:37.746 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:29:37.746 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:29:37.746 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:29:37.746 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:29:37.747 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:29:37.747 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:29:37.747 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:29:37.747 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:29:37.747 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:29:37.747 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:29:37.748 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:29:37.748 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:29:37.748 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:29:37.748 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:29:37.748 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:29:37.748 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:29:37.749 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:29:37.749 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:29:37.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.749 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:29:37.749 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:29:37.749 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:29:37.749 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:29:37.750 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:29:37.750 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:29:37.750 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:29:37.751 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:29:37.751 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:29:37.752 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:29:37.752 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:29:37.752 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:29:37.752 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:29:37.752 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd": {} 2024-03-24T12:29:37.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.753 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:29:37.753 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:29:37.753 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.6500000953674316, 2024-03-24T12:29:37.753 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.6500000953674316, 2024-03-24T12:29:37.753 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:29:37.753 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-24T12:29:37.754 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-24T12:29:37.754 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:29:37.754 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:29:37.754 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:29:37.754 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.755 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 3, 2024-03-24T12:29:37.755 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".rgw.root", 2024-03-24T12:29:37.755 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:02.100021+0000", 2024-03-24T12:29:37.755 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:29:37.755 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:29:37.756 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:29:37.756 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:29:37.756 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:29:37.756 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:29:37.756 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:29:37.756 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:29:37.757 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:29:37.757 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:29:37.757 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:29:37.757 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:29:37.757 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:29:37.757 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:29:37.758 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:29:37.758 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:29:37.758 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:29:37.758 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:29:37.758 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:29:37.759 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:29:37.759 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:29:37.759 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:29:37.759 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:29:37.759 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:29:37.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.760 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "68", 2024-03-24T12:29:37.760 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:29:37.760 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:29:37.760 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:29:37.760 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:29:37.761 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:29:37.761 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:29:37.761 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:29:37.761 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:29:37.761 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:29:37.761 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:29:37.762 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:29:37.762 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:29:37.762 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:29:37.762 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:29:37.762 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:29:37.762 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:29:37.763 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:29:37.763 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:29:37.763 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:29:37.763 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:29:37.763 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:29:37.763 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:29:37.764 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:29:37.764 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:29:37.764 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:29:37.764 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:29:37.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.764 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:29:37.765 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:29:37.765 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:29:37.765 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:29:37.765 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:29:37.765 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:29:37.765 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:29:37.766 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:29:37.766 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:29:37.766 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:29:37.766 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:29:37.766 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:29:37.767 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:29:37.767 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:29:37.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.767 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:29:37.767 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:29:37.767 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.5, 2024-03-24T12:29:37.768 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.5, 2024-03-24T12:29:37.768 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:29:37.768 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.5, 2024-03-24T12:29:37.768 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.5, 2024-03-24T12:29:37.768 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:29:37.768 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:29:37.769 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:29:37.769 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.769 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 4, 2024-03-24T12:29:37.769 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:29:37.770 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:03.473862+0000", 2024-03-24T12:29:37.770 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:29:37.770 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:29:37.770 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:29:37.770 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:29:37.771 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:29:37.771 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:29:37.771 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:29:37.771 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:29:37.771 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:29:37.772 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:29:37.772 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:29:37.772 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:29:37.772 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:29:37.772 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:29:37.773 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:29:37.773 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:29:37.773 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:29:37.773 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:29:37.773 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:29:37.773 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:29:37.774 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:29:37.774 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:29:37.774 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:29:37.774 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:29:37.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.775 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "70", 2024-03-24T12:29:37.775 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:29:37.775 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:29:37.776 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:29:37.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:29:37.778 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.25, 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.25, 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.25, 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2.25, 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 5, 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:05.573828+0000", 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:29:37.780 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:29:37.781 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "72", 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:29:37.782 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:29:37.783 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:29:37.784 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.25, 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.25, 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.25, 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.25, 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:29:37.785 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 6, 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:07.645083+0000", 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:29:37.786 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:29:37.787 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "74", 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:29:37.788 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:29:37.789 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:29:37.790 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_bias": 4 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.75, 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.75, 2024-03-24T12:29:37.791 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.75, 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.75, 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 7, 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "unique_pool_0", 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:28:04.290397+0000", 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:29:37.792 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 26, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 26, 2024-03-24T12:29:37.793 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 26, 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 26, 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 26, 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "111", 2024-03-24T12:29:37.794 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "109", 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:29:37.795 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:29:37.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:29:37.797 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "rados": {} 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.4600000381469727, 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.8500000238418579, 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.4600000381469727, 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.8500000238418579, 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:29:37.798 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b", 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:29:37.799 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 8, 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 112, 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6801", 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:29:37.800 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6802", 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6804", 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.801 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6803", 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6801/2514818464", 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6802/2514818464", 2024-03-24T12:29:37.802 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6804/2514818464", 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6803/2514818464", 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc", 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:29:37.803 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 13, 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 112, 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6805", 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:29:37.804 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6806", 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6808", 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6807", 2024-03-24T12:29:37.806 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6805/2923396572", 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6806/2923396572", 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6808/2923396572", 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6807/2923396572", 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "c3f83505-25ee-4217-b60a-081c36092e92", 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 18, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 112, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:29:37.808 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6809", 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6810", 2024-03-24T12:29:37.809 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6812", 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6811", 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6809/126032915", 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6810/126032915", 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6812/126032915", 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6811/126032915", 2024-03-24T12:29:37.811 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96", 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:29:37.812 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 27, 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 112, 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6813", 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.813 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6814", 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6816", 2024-03-24T12:29:37.814 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6815", 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.815 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6813/1080171824", 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6814/1080171824", 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6816/1080171824", 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6815/1080171824", 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "5e420304-2798-4d11-9688-d0d89482008c", 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:29:37.816 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 33, 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 111, 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6800", 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:29:37.817 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6801", 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:29:37.818 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6803", 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6802", 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.819 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6800/3690258869", 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6801/3690258869", 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6803/3690258869", 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6802/3690258869", 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "1213431a-652c-4cbf-b32a-36335430c666", 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:29:37.820 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 42, 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 109, 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6804", 2024-03-24T12:29:37.821 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6805", 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.822 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6807", 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6806", 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:29:37.823 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6804/1327220488", 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6805/1327220488", 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6807/1327220488", 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6806/1327220488", 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:29:37.824 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c", 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 51, 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 111, 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6808", 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6809", 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:29:37.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6811", 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6810", 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6808/1295064976", 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6809/1295064976", 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6811/1295064976", 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6810/1295064976", 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:29:37.828 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda", 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:29:37.829 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 60, 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 112, 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6812", 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.830 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6813", 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.831 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6815", 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6814", 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.832 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6812/110963949", 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6813/110963949", 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6815/110963949", 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6814/110963949", 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_xinfo": [ 2024-03-24T12:29:37.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:28:57.006151+0000", 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:29:37.834 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:29:28.213452+0000", 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:29:37.835 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:28:42.762815+0000", 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:28:45.010123+0000", 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:29:30.861346+0000", 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:29:37.837 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:28:54.639476+0000", 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:29:21.054689+0000", 2024-03-24T12:29:37.838 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:29:27.828822+0000", 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.839 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap": [], 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_items": [ 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.b", 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.f", 2024-03-24T12:29:37.840 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.12", 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 5 2024-03-24T12:29:37.841 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_primaries": [], 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_temp": [ 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "7.10", 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: 3, 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: 1 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "7.11", 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: 3, 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: 6, 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: 4 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "7.12", 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: 1, 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: 2, 2024-03-24T12:29:37.843 INFO:teuthology.orchestra.run.smithi029.stdout: 3 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "7.13", 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: 2, 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: 3, 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: 4 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "7.14", 2024-03-24T12:29:37.844 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: 0, 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: 2 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "7.15", 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: 7, 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: 2, 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: 0 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.845 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "7.16", 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: 0, 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: 7 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "7.17", 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: 0, 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: 2 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "7.18", 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: 2, 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: 7 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "7.19", 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: 0, 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: 2 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.847 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_temp": [], 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklist": { 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2633022206": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2612078930": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2365028933": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4113609697": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4130171883": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3697529912": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2929430696": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/327626505": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3715448582": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/861146182": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2109454792": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1631806715": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:29:37.848 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/852212806": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1823252077": "2024-03-25T12:21:15.151901+0000" 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "range_blocklist": {}, 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profiles": { 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "default": { 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-failure-domain": "osd", 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "k": "2", 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "m": "1", 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "plugin": "jerasure", 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "technique": "reed_sol_van" 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps_queue": [], 2024-03-24T12:29:37.849 INFO:teuthology.orchestra.run.smithi029.stdout: "new_removed_snaps": [], 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: "new_purged_snaps": [], 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_node_flags": {}, 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: "device_class_flags": {}, 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode": { 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_enabled": false, 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_bucket_count": 0, 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_bucket": 0 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:29:37.850 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:29:37.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:37.878 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:37.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:37.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:37.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:29:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:29:37.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:29:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:29:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:29:37.980 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:38.024 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:38.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:38.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:38.150 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:38.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:38.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:38.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:38.276 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:38.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:38.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:37 smithi029 ceph-mon[41500]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T12:29:38.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:37 smithi029 ceph-mon[29919]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T12:29:38.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:38 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:38.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:37 smithi190 ceph-mon[33748]: osdmap e121: 8 total, 8 up, 8 in 2024-03-24T12:29:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:37 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:38.424 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:38.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:38.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:38.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:38.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:38.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:38.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:38.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:38.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:38 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:38.685 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:38.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:38.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:38.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 85 16 97913 97897 71.9663 31.625 0.026841 0.0138879 2024-03-24T12:29:38.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:38.811 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:38.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:38.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:38.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:38.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:38.942 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:38.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:39.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:39.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:39.096 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:39.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:39.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:39 smithi029 ceph-mon[41500]: pgmap v127: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 148 active+clean, 1 active+remapped+backfilling; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 9.1 MiB/s wr, 147 op/s; 5498/293355 objects degraded (1.874%); 38934/293355 objects misplaced (13.272%); 21 MiB/s, 330 objects/s recovering 2024-03-24T12:29:39.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:39 smithi029 ceph-mon[41500]: 7.6 deep-scrub ok 2024-03-24T12:29:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:39 smithi029 ceph-mon[29919]: pgmap v127: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 148 active+clean, 1 active+remapped+backfilling; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 9.1 MiB/s wr, 147 op/s; 5498/293355 objects degraded (1.874%); 38934/293355 objects misplaced (13.272%); 21 MiB/s, 330 objects/s recovering 2024-03-24T12:29:39.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:39 smithi029 ceph-mon[29919]: 7.6 deep-scrub ok 2024-03-24T12:29:39.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:39.237 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:39.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:39.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:39.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:39 smithi190 ceph-mon[33748]: pgmap v127: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 148 active+clean, 1 active+remapped+backfilling; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 9.1 MiB/s wr, 147 op/s; 5498/293355 objects degraded (1.874%); 38934/293355 objects misplaced (13.272%); 21 MiB/s, 330 objects/s recovering 2024-03-24T12:29:39.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:39 smithi190 ceph-mon[33748]: 7.6 deep-scrub ok 2024-03-24T12:29:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:39.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:39.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:39.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:39.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:39.517 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:39.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:39.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:39.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:39.642 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:39.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:39.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 86 16 98265 98249 71.3854 22 0.0309111 0.0139867 2024-03-24T12:29:39.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:39.787 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:39.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:39.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:39.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:39.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:40.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:40.062 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:40.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:40.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:40.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:40.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:40.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:40.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:40.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:40.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:40.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:40 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 5498/293355 objects degraded (1.874%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:40.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:40 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 5498/293355 objects degraded (1.874%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:40.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:40 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 5498/293355 objects degraded (1.874%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:40.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:40.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:40.484 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:40.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:40.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:40.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:40.635 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:40.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:40.690 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 87 16 99027 99011 71.1123 47.625 0.0239583 0.0140564 2024-03-24T12:29:40.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:40.767 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:40.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:40.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:40.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:40.914 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:40.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:40.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:41.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:41.041 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:41.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:41.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:41.167 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:41.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:41.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:41.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:41.298 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:41.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:41 smithi190 ceph-mon[33748]: pgmap v128: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 148 active+clean, 1 active+remapped+backfilling; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 21 MiB/s wr, 339 op/s; 5498/293355 objects degraded (1.874%); 38934/293355 objects misplaced (13.272%); 16 MiB/s, 253 objects/s recovering 2024-03-24T12:29:41.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:41.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:41 smithi029 ceph-mon[29919]: pgmap v128: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 148 active+clean, 1 active+remapped+backfilling; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 21 MiB/s wr, 339 op/s; 5498/293355 objects degraded (1.874%); 38934/293355 objects misplaced (13.272%); 16 MiB/s, 253 objects/s recovering 2024-03-24T12:29:41.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:41 smithi029 ceph-mon[41500]: pgmap v128: 158 pgs: 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 148 active+clean, 1 active+remapped+backfilling; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 21 MiB/s wr, 339 op/s; 5498/293355 objects degraded (1.874%); 38934/293355 objects misplaced (13.272%); 16 MiB/s, 253 objects/s recovering 2024-03-24T12:29:41.430 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:41.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:41.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:41.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:41.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:41.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:41.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:41.691 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 88 16 99600 99584 70.7109 35.8125 0.0224705 0.0141364 2024-03-24T12:29:41.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:41.705 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:41.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:41.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:41.830 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:41.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:41.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:41.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:41.958 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:41.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:41.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:42.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:42.105 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:42.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:42.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:42.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:42.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:42.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:42.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:42.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:42.511 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:42.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:42.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:42.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:42.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:42.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:42.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:42.691 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 89 16 100144 100128 70.2984 34 0.015077 0.0142067 2024-03-24T12:29:42.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:42.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:42.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:42.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:42.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:42.894 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:42.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:42.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:42.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:43.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:43.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:43.128 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:43.169 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:43.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:43.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:43.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:43.299 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:43.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:43.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:43.444 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:43.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:43.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:43.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:43.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:43.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:43.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:43.691 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 90 3 100760 100757 69.9541 39.3125 0.0179837 0.0142889 2024-03-24T12:29:43.704 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:43.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:43.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:43.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:43.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:43.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:43 smithi190 ceph-mon[33748]: pgmap v129: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 148 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 25 MiB/s wr, 407 op/s; 5498/296442 objects degraded (1.855%); 38244/296442 objects misplaced (12.901%); 16 MiB/s, 252 objects/s recovering 2024-03-24T12:29:43.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:43 smithi029 ceph-mon[29919]: pgmap v129: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 148 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 25 MiB/s wr, 407 op/s; 5498/296442 objects degraded (1.855%); 38244/296442 objects misplaced (12.901%); 16 MiB/s, 252 objects/s recovering 2024-03-24T12:29:43.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:43 smithi029 ceph-mon[41500]: pgmap v129: 158 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing+deep, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 148 active+clean; 6.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 25 MiB/s wr, 407 op/s; 5498/296442 objects degraded (1.855%); 38244/296442 objects misplaced (12.901%); 16 MiB/s, 252 objects/s recovering 2024-03-24T12:29:43.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:43.968 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:43.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:44.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:44.104 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:44.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:44.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:44.241 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:44.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:44.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:44.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:44.368 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:44.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:44.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:44.494 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:44.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:44.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:44.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:44.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:44.621 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:44.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:44.691 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Total time run: 90.1356 2024-03-24T12:29:44.691 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Total writes made: 100760 2024-03-24T12:29:44.691 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Write size: 65536 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Object size: 65536 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Bandwidth (MB/sec): 69.8669 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev Bandwidth: 41.1955 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max bandwidth (MB/sec): 202.875 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min bandwidth (MB/sec): 0 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Average IOPS: 1117 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev IOPS: 659.128 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max IOPS: 3246 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min IOPS: 0 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Average Latency(s): 0.0142926 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev Latency(s): 0.039638 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max latency(s): 2.39623 2024-03-24T12:29:44.692 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min latency(s): 0.00114673 2024-03-24T12:29:44.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:44.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:44.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:44.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:44.848 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:44.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:44.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:44.890 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T12:29:44.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:45.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:45.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:45.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:45.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:45.133 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:45.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:45.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:45.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:45.281 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:45.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:45.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:45.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:45.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:45.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:45.540 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:45.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:45.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:45.671 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:45.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:45.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:45.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:45.802 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:45.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:45.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:45 smithi190 ceph-mon[33748]: pgmap v130: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 233 op/s; 5498/301905 objects degraded (1.821%); 37555/301905 objects misplaced (12.439%); 18 MiB/s, 289 objects/s recovering 2024-03-24T12:29:45.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:45 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 5498/301905 objects degraded (1.821%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:45.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:45.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:45 smithi029 ceph-mon[41500]: pgmap v130: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 233 op/s; 5498/301905 objects degraded (1.821%); 37555/301905 objects misplaced (12.439%); 18 MiB/s, 289 objects/s recovering 2024-03-24T12:29:45.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:45 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 5498/301905 objects degraded (1.821%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:45.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:45 smithi029 ceph-mon[29919]: pgmap v130: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 233 op/s; 5498/301905 objects degraded (1.821%); 37555/301905 objects misplaced (12.439%); 18 MiB/s, 289 objects/s recovering 2024-03-24T12:29:45.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:45 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 5498/301905 objects degraded (1.821%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:45.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:46.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:46.100 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:46.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:46.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:46.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:46.225 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:46.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:46.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:46.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:46.351 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:46.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:46.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:46.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:46.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:46.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:46.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:46.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:46.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:46.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:46.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:46.744 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:46.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:46.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:46.894 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:46.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:46.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:46.996 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:47.044 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:47.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:47.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:47.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:47.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:47.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:47.304 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:47.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:47.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:47.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:47.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:47.452 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:47.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:47.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:47.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:47.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:47.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:47.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:47.710 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:47.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:47.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:47.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:47.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:47.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:47.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:47.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:47 smithi190 ceph-mon[33748]: pgmap v131: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 12 MiB/s wr, 197 op/s; 5498/301905 objects degraded (1.821%); 37555/301905 objects misplaced (12.439%); 15 MiB/s, 244 objects/s recovering 2024-03-24T12:29:47.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:47.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:29:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:29:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:47 smithi029 ceph-mon[29919]: pgmap v131: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 12 MiB/s wr, 197 op/s; 5498/301905 objects degraded (1.821%); 37555/301905 objects misplaced (12.439%); 15 MiB/s, 244 objects/s recovering 2024-03-24T12:29:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:47.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:29:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:29:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:29:47.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:47 smithi029 ceph-mon[41500]: pgmap v131: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 12 MiB/s wr, 197 op/s; 5498/301905 objects degraded (1.821%); 37555/301905 objects misplaced (12.439%); 15 MiB/s, 244 objects/s recovering 2024-03-24T12:29:47.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:29:47.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:47.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:47.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:47.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:48.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:48.117 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:48.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:48.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:48.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:48.242 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:48.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:48.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:48.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:48.389 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:48.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:48.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:48.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:48.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:48.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:48.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:48.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:48.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:48.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:48.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:48.799 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:48.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:48.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:48 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:48.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:48.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:48 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:48.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:48 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:48.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:48.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:49.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:49.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:49.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:49.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:49.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:49.203 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:49.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:49.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:49.343 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:49.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:49.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:49.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:49.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:49.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:49.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:49.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:49.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:49.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:49.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:49.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:49.726 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:49.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:49.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:49.856 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:49.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:49.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:49 smithi190 ceph-mon[33748]: pgmap v132: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 13 MiB/s wr, 319 op/s; 5498/299718 objects degraded (1.834%); 35625/299718 objects misplaced (11.886%); 19 MiB/s, 301 objects/s recovering 2024-03-24T12:29:49.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:49 smithi029 ceph-mon[41500]: pgmap v132: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 13 MiB/s wr, 319 op/s; 5498/299718 objects degraded (1.834%); 35625/299718 objects misplaced (11.886%); 19 MiB/s, 301 objects/s recovering 2024-03-24T12:29:49.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:49 smithi029 ceph-mon[29919]: pgmap v132: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.4 KiB/s rd, 13 MiB/s wr, 319 op/s; 5498/299718 objects degraded (1.834%); 35625/299718 objects misplaced (11.886%); 19 MiB/s, 301 objects/s recovering 2024-03-24T12:29:49.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:49.996 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:49.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:50.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:50.142 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:50.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:50.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:50.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:50.273 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:50.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:50.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:50.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:50.398 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:50.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:50.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:50.524 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:50.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:50.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:50.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:50.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:50 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 5498/299718 objects degraded (1.834%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:50.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:50 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 5498/299718 objects degraded (1.834%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:50.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:50.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:50.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:50.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:50.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:50 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 5498/299718 objects degraded (1.834%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:50.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:50.932 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:50.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:50.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:51.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:51.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:51.083 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:51.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:51.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:51.208 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:51.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:51.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:51.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:51.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:51.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:51.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:51.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:51.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:51.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:51.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:51.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:51.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:51.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:51.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:51.745 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:51.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:51.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:51.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:51.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:51.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:51.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:52.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:52.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:52.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:52.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:52.133 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:52.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:52.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:51 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:29:52.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:51 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:29:52.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:51 smithi029 ceph-mon[29919]: pgmap v133: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 938 B/s rd, 2.4 MiB/s wr, 192 op/s; 5344/298017 objects degraded (1.793%); 34742/298017 objects misplaced (11.658%); 13 MiB/s, 201 objects/s recovering 2024-03-24T12:29:52.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:51 smithi029 ceph-mon[29919]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T12:29:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:51 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:29:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:51 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:29:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:51 smithi029 ceph-mon[41500]: pgmap v133: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 938 B/s rd, 2.4 MiB/s wr, 192 op/s; 5344/298017 objects degraded (1.793%); 34742/298017 objects misplaced (11.658%); 13 MiB/s, 201 objects/s recovering 2024-03-24T12:29:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:51 smithi029 ceph-mon[41500]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T12:29:52.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:52.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:52.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:52.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:52.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:51 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:29:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:51 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:29:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:51 smithi190 ceph-mon[33748]: pgmap v133: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 938 B/s rd, 2.4 MiB/s wr, 192 op/s; 5344/298017 objects degraded (1.793%); 34742/298017 objects misplaced (11.658%); 13 MiB/s, 201 objects/s recovering 2024-03-24T12:29:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:51 smithi190 ceph-mon[33748]: osdmap e122: 8 total, 8 up, 8 in 2024-03-24T12:29:52.390 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:52.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:52.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:52.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:52.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:52.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:52.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:52.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:52.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:52.645 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:52.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:52.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:52.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:52.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:52.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:52.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:52.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:52.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:53.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:53.076 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:53.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:53.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:53.168 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:29:52 smithi029 podman[92953]: 2024-03-24 12:29:52.790705529 +0000 UTC m=+0.042889229 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-24T12:29:53.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:53.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:53.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:53.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:53.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:53.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:53.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:53.434 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:53.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:53.458 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:53.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:53.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:53.583 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:53.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:53.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:53.721 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:53.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:53.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:53.846 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:53.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:53.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:53.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:53.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:53.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:53.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:54.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:54.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:54.097 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:54.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:54.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:54.228 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:54.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:54.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:54.330 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:54.380 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:54.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:54.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:53 smithi190 ceph-mon[33748]: pgmap v135: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 233 op/s; 5344/294864 objects degraded (1.812%); 34038/294864 objects misplaced (11.544%); 15 MiB/s, 238 objects/s recovering 2024-03-24T12:29:54.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:53 smithi190 ceph-mon[33748]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T12:29:54.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:53 smithi029 ceph-mon[41500]: pgmap v135: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 233 op/s; 5344/294864 objects degraded (1.812%); 34038/294864 objects misplaced (11.544%); 15 MiB/s, 238 objects/s recovering 2024-03-24T12:29:54.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:54 smithi029 ceph-mon[41500]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T12:29:54.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:53 smithi029 ceph-mon[29919]: pgmap v135: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 6.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 233 op/s; 5344/294864 objects degraded (1.812%); 34038/294864 objects misplaced (11.544%); 15 MiB/s, 238 objects/s recovering 2024-03-24T12:29:54.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:54 smithi029 ceph-mon[29919]: osdmap e123: 8 total, 8 up, 8 in 2024-03-24T12:29:54.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:54.524 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:54.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:54.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:54.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:54.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:54.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:54.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:54.794 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:54.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:54.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:54.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:54.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:54.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:55.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:55.053 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:55.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:55.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:55.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:55.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:55.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:55.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:55.338 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:55.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:55.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:55 smithi029 ceph-mon[41500]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T12:29:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:55 smithi029 ceph-mon[41500]: pgmap v138: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 629 KiB/s wr, 556 op/s; 5200/290067 objects degraded (1.793%); 32850/290067 objects misplaced (11.325%); 15 MiB/s, 234 objects/s recovering 2024-03-24T12:29:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:55 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 5344/294864 objects degraded (1.812%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:55.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:55 smithi029 ceph-mon[41500]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T12:29:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:55 smithi029 ceph-mon[29919]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T12:29:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:55 smithi029 ceph-mon[29919]: pgmap v138: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 629 KiB/s wr, 556 op/s; 5200/290067 objects degraded (1.793%); 32850/290067 objects misplaced (11.325%); 15 MiB/s, 234 objects/s recovering 2024-03-24T12:29:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:55 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 5344/294864 objects degraded (1.812%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:55 smithi029 ceph-mon[29919]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T12:29:55.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:55.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:55.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:55.584 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:55.614 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:55.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:55.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:55.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:55 smithi190 ceph-mon[33748]: osdmap e124: 8 total, 8 up, 8 in 2024-03-24T12:29:55.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:55 smithi190 ceph-mon[33748]: pgmap v138: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 629 KiB/s wr, 556 op/s; 5200/290067 objects degraded (1.793%); 32850/290067 objects misplaced (11.325%); 15 MiB/s, 234 objects/s recovering 2024-03-24T12:29:55.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:55 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 5344/294864 objects degraded (1.812%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:29:55.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:55 smithi190 ceph-mon[33748]: osdmap e125: 8 total, 8 up, 8 in 2024-03-24T12:29:55.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:55.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:55.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:55.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:55.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:55.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:55.871 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:55.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:55.973 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:55.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:55.997 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:55.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:56.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:56.147 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:56.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:56.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:56.305 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:56.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:56.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:56.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:56.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:56.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:56.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:56.587 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:56.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:56.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:56.712 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:56.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:56.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:56.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:56.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:56.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:56.939 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:56.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:56.964 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:56.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:57.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:57.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:57.104 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:57.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:57.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:57.246 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:57.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:57.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:57.389 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:57.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:57.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:57.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:57.515 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:57.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:57.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:57.641 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:57.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:57.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:57.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:57.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:57.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:57.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:57.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:57 smithi190 ceph-mon[33748]: pgmap v140: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 926 B/s rd, 0 B/s wr, 480 op/s; 5200/290067 objects degraded (1.793%); 32850/290067 objects misplaced (11.325%); 10 MiB/s, 166 objects/s recovering 2024-03-24T12:29:57.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:29:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:29:57.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:57.897 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:57.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:57.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:57 smithi029 ceph-mon[29919]: pgmap v140: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 926 B/s rd, 0 B/s wr, 480 op/s; 5200/290067 objects degraded (1.793%); 32850/290067 objects misplaced (11.325%); 10 MiB/s, 166 objects/s recovering 2024-03-24T12:29:57.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:29:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:29:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:29:57.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:57 smithi029 ceph-mon[41500]: pgmap v140: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 149 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 926 B/s rd, 0 B/s wr, 480 op/s; 5200/290067 objects degraded (1.793%); 32850/290067 objects misplaced (11.325%); 10 MiB/s, 166 objects/s recovering 2024-03-24T12:29:57.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:58.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:58.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:58.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:58.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:29:58.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:58.158 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:58.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:29:58.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:58.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:58.307 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:58.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:58.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:58.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:58.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:58.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:58.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:58.560 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:58.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:58.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:58.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:29:58.689 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:58.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:58.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:29:58.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:58.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:58.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:58.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:58 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:58.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:58 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:29:58.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:29:58.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:58.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:58.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:29:59.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:29:59.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:59.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:59.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:29:59.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:59.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:59.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:59.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:59.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:29:59.410 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:59.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:29:59.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:59.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:59.541 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:29:59.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:29:59.671 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:59.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:59.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:29:59.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:29:59.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:59.817 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:29:59.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:29:59.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:29:59 smithi190 ceph-mon[33748]: pgmap v141: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 585 op/s; 5200/284343 objects degraded (1.829%); 32207/284343 objects misplaced (11.327%); 8.8 MiB/s, 140 objects/s recovering 2024-03-24T12:29:59.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:29:59.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:29:59 smithi029 ceph-mon[41500]: pgmap v141: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 585 op/s; 5200/284343 objects degraded (1.829%); 32207/284343 objects misplaced (11.327%); 8.8 MiB/s, 140 objects/s recovering 2024-03-24T12:29:59.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:29:59 smithi029 ceph-mon[29919]: pgmap v141: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 585 op/s; 5200/284343 objects degraded (1.829%); 32207/284343 objects misplaced (11.327%); 8.8 MiB/s, 140 objects/s recovering 2024-03-24T12:29:59.950 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:29:59.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:29:59.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:00.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:00.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:00.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:00.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:00.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:00.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:00.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:00.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:00.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:00.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:00.353 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:00.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:00.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:00.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:00.499 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:00.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:00.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:00.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:00.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:00.726 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:00.749 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:00.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:00.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:00.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:00.875 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:00.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:00.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:00 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 5200/284343 objects degraded (1.829%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:00.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:00 smithi190 ceph-mon[33748]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull; Degraded data redundancy: 5200/284343 objects degraded (1.829%), 1 pg degraded 2024-03-24T12:30:00.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:00 smithi190 ceph-mon[33748]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-03-24T12:30:00.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:00 smithi190 ceph-mon[33748]: pg 7.11 is active+remapped+backfill_toofull, acting [3,6,4] 2024-03-24T12:30:00.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:00 smithi190 ceph-mon[33748]: pg 7.13 is active+remapped+backfill_toofull, acting [2,3,4] 2024-03-24T12:30:00.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:00 smithi190 ceph-mon[33748]: [WRN] PG_DEGRADED: Degraded data redundancy: 5200/284343 objects degraded (1.829%), 1 pg degraded 2024-03-24T12:30:00.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:00 smithi190 ceph-mon[33748]: pg 7.19 is active+recovery_wait+undersized+degraded+remapped, acting [0,2] 2024-03-24T12:30:00.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 5200/284343 objects degraded (1.829%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[41500]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull; Degraded data redundancy: 5200/284343 objects degraded (1.829%), 1 pg degraded 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[41500]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[41500]: pg 7.11 is active+remapped+backfill_toofull, acting [3,6,4] 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[41500]: pg 7.13 is active+remapped+backfill_toofull, acting [2,3,4] 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[41500]: [WRN] PG_DEGRADED: Degraded data redundancy: 5200/284343 objects degraded (1.829%), 1 pg degraded 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[41500]: pg 7.19 is active+recovery_wait+undersized+degraded+remapped, acting [0,2] 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 5200/284343 objects degraded (1.829%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[29919]: Health detail: HEALTH_WARN Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull; Degraded data redundancy: 5200/284343 objects degraded (1.829%), 1 pg degraded 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[29919]: [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[29919]: pg 7.11 is active+remapped+backfill_toofull, acting [3,6,4] 2024-03-24T12:30:00.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[29919]: pg 7.13 is active+remapped+backfill_toofull, acting [2,3,4] 2024-03-24T12:30:00.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[29919]: [WRN] PG_DEGRADED: Degraded data redundancy: 5200/284343 objects degraded (1.829%), 1 pg degraded 2024-03-24T12:30:00.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:00 smithi029 ceph-mon[29919]: pg 7.19 is active+recovery_wait+undersized+degraded+remapped, acting [0,2] 2024-03-24T12:30:00.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:01.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:01.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:01.151 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:01.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:01.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:01.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:01.277 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:01.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:01.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:01.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:01.403 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:01.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:01.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:01.532 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:01.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:01.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:01.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:01.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:01.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:01.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:01.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:01.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:01.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:01.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:01 smithi190 ceph-mon[33748]: pgmap v142: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 681 B/s rd, 0 B/s wr, 513 op/s; 5018/282075 objects degraded (1.779%); 31465/282075 objects misplaced (11.155%); 9.8 MiB/s, 156 objects/s recovering 2024-03-24T12:30:01.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:01.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:01 smithi029 ceph-mon[29919]: pgmap v142: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 681 B/s rd, 0 B/s wr, 513 op/s; 5018/282075 objects degraded (1.779%); 31465/282075 objects misplaced (11.155%); 9.8 MiB/s, 156 objects/s recovering 2024-03-24T12:30:01.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:01 smithi029 ceph-mon[41500]: pgmap v142: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 681 B/s rd, 0 B/s wr, 513 op/s; 5018/282075 objects degraded (1.779%); 31465/282075 objects misplaced (11.155%); 9.8 MiB/s, 156 objects/s recovering 2024-03-24T12:30:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:01.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:01.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:02.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:02.071 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:02.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:02.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:02.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:02.218 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:02.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:02.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:02.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:02.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:02.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:02.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:02.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:02.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:02.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:02.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:02.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:02.671 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:02.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:02.773 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:02.804 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:02.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:02.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:02 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:02.905 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:02.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:02 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:02.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:02 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:02.945 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:02.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:02.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:03.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:03.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:03.071 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:03.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:03.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:03.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:03.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:03.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:03.325 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:03.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:03.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:03.474 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:03.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:03.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:03.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:03.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:03.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:03.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:03.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:03.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:03.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:03.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:03.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:03.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:03 smithi190 ceph-mon[33748]: pgmap v143: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 428 op/s; 5018/279411 objects degraded (1.796%); 31465/279411 objects misplaced (11.261%); 5.2 MiB/s, 83 objects/s recovering 2024-03-24T12:30:03.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:03 smithi029 ceph-mon[29919]: pgmap v143: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 428 op/s; 5018/279411 objects degraded (1.796%); 31465/279411 objects misplaced (11.261%); 5.2 MiB/s, 83 objects/s recovering 2024-03-24T12:30:03.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:03 smithi029 ceph-mon[41500]: pgmap v143: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 428 op/s; 5018/279411 objects degraded (1.796%); 31465/279411 objects misplaced (11.261%); 5.2 MiB/s, 83 objects/s recovering 2024-03-24T12:30:03.977 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:04.010 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:04.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:04.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:04.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:04.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:04.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:04.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:04.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:04.283 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:04.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:04.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:04.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:04.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:04.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:04.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:04.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:04.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:04.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:04.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:04.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:04.666 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:04.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:04.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:04.810 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:04.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:04.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:04.941 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:04.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:05.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:05.087 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:05.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:05.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:05.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:05.217 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:05.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:05.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:05.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:05.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:05.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:05.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:05.470 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:05.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:05.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:05.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:05.620 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:05.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:05.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:05.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:05.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:05.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:05.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:05.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:05 smithi190 ceph-mon[33748]: pgmap v144: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 564 op/s; 4850/273147 objects degraded (1.776%); 29909/273147 objects misplaced (10.950%); 9.3 MiB/s, 148 objects/s recovering 2024-03-24T12:30:05.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:05 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 4850/273147 objects degraded (1.776%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:05.892 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:05.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:05.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:05 smithi029 ceph-mon[29919]: pgmap v144: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 564 op/s; 4850/273147 objects degraded (1.776%); 29909/273147 objects misplaced (10.950%); 9.3 MiB/s, 148 objects/s recovering 2024-03-24T12:30:05.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:05 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 4850/273147 objects degraded (1.776%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:05.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:05 smithi029 ceph-mon[41500]: pgmap v144: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 564 op/s; 4850/273147 objects degraded (1.776%); 29909/273147 objects misplaced (10.950%); 9.3 MiB/s, 148 objects/s recovering 2024-03-24T12:30:05.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:05 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 4850/273147 objects degraded (1.776%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:05.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:06.022 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:06.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:06.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:06.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:06.151 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:06.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:06.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:06.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:06.282 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:06.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:06.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:06.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:06.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:06.430 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:06.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:06.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:06.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:06.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:06.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:06.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:06.689 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:06.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:06.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:06.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:06.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:06.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:06.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:06.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:07.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:07.085 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:07.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:07.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:07.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:07.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:07.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:07.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:07.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:07.363 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:07.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:07.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:07.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:07.510 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:07.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:07.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:07.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:07.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:07.636 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:07.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:07.761 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:07.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:07.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:07 smithi190 ceph-mon[33748]: pgmap v145: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 887 B/s rd, 0 B/s wr, 489 op/s; 4850/273147 objects degraded (1.776%); 29909/273147 objects misplaced (10.950%); 8.1 MiB/s, 129 objects/s recovering 2024-03-24T12:30:07.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:30:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:30:07.907 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:07.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:07.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:07 smithi029 ceph-mon[29919]: pgmap v145: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 887 B/s rd, 0 B/s wr, 489 op/s; 4850/273147 objects degraded (1.776%); 29909/273147 objects misplaced (10.950%); 8.1 MiB/s, 129 objects/s recovering 2024-03-24T12:30:07.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:07 smithi029 ceph-mon[41500]: pgmap v145: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 887 B/s rd, 0 B/s wr, 489 op/s; 4850/273147 objects degraded (1.776%); 29909/273147 objects misplaced (10.950%); 8.1 MiB/s, 129 objects/s recovering 2024-03-24T12:30:07.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:30:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:30:08.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:08.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:08.050 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:08.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:08.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:08.175 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:08.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:08.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:08.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:08.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:08.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:08.431 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:08.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:08.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:08.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:08.583 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:08.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:08.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:08.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:08.732 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:08.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:08.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:08.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:08.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:08.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:09.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:09.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:09.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:09.153 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:09.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:09.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:09.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:09.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:09.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:09.409 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:09.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:09.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:09.511 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:09.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:09.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:09.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:09.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:09.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:09.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:09.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:09.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:09 smithi190 ceph-mon[33748]: pgmap v146: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 637 op/s; 4850/267171 objects degraded (1.815%); 29360/267171 objects misplaced (10.989%); 8.9 MiB/s, 142 objects/s recovering 2024-03-24T12:30:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:09 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:30:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:09 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:30:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:09 smithi029 ceph-mon[41500]: pgmap v146: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 637 op/s; 4850/267171 objects degraded (1.815%); 29360/267171 objects misplaced (10.989%); 8.9 MiB/s, 142 objects/s recovering 2024-03-24T12:30:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:09 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:30:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:09 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:30:09.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:09 smithi029 ceph-mon[29919]: pgmap v146: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 637 op/s; 4850/267171 objects degraded (1.815%); 29360/267171 objects misplaced (10.989%); 8.9 MiB/s, 142 objects/s recovering 2024-03-24T12:30:09.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:09 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:30:09.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:09 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:30:09.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:09.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:09.944 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:09.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:10.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:10.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:10.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:10.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:10.201 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:10.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:10.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:10.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:10.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:10.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:10.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:10.458 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:10.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:10.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:10.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:10.606 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:10.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:10.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:10 smithi029 ceph-mon[41500]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T12:30:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:10 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 4850/267171 objects degraded (1.815%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:10 smithi029 ceph-mon[41500]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T12:30:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:10 smithi029 ceph-mon[29919]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T12:30:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:10 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 4850/267171 objects degraded (1.815%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:10 smithi029 ceph-mon[29919]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T12:30:10.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:10.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:10.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:10.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:10.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:10.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:10.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:10.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:10 smithi190 ceph-mon[33748]: osdmap e126: 8 total, 8 up, 8 in 2024-03-24T12:30:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:10 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 4850/267171 objects degraded (1.815%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:10 smithi190 ceph-mon[33748]: osdmap e127: 8 total, 8 up, 8 in 2024-03-24T12:30:10.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:11.009 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:11.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:11.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:11.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:11.135 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:11.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:11.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:11.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:11.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:11.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:11.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:11.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:11.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:11.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:11.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:11.524 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:11.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:11.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:11.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:11.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:11.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:11.781 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:11.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:11.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:11.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:11.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:12.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:12.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:12.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:12.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:12.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:11 smithi029 ceph-mon[29919]: pgmap v149: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 609 op/s; 4656/264795 objects degraded (1.758%); 28630/264795 objects misplaced (10.812%); 11 MiB/s, 181 objects/s recovering 2024-03-24T12:30:12.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:11 smithi029 ceph-mon[41500]: pgmap v149: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 609 op/s; 4656/264795 objects degraded (1.758%); 28630/264795 objects misplaced (10.812%); 11 MiB/s, 181 objects/s recovering 2024-03-24T12:30:12.186 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:12.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:12.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:12.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:12.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:12.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:12.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:11 smithi190 ceph-mon[33748]: pgmap v149: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 609 op/s; 4656/264795 objects degraded (1.758%); 28630/264795 objects misplaced (10.812%); 11 MiB/s, 181 objects/s recovering 2024-03-24T12:30:12.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:12.463 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:12.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:12.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:12.588 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:12.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:12.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:12.715 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:12.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:12.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:12.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:12.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:12.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:12.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:12.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:12.992 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:12.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:13.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:13.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:13.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:13.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:13.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:13.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:13.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:13.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:13.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:13.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:13.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:13.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:13.503 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:13.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:13.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:13.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:13.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:13.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:13.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:13.791 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:13.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:13.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:13.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:13.945 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:13.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:14.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:14.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:14.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:14.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:13 smithi029 ceph-mon[29919]: pgmap v150: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 469 op/s; 4656/261909 objects degraded (1.778%); 28630/261909 objects misplaced (10.931%); 5.1 MiB/s, 81 objects/s recovering 2024-03-24T12:30:14.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:13 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:30:14.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:13 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:30:14.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:13 smithi029 ceph-mon[41500]: pgmap v150: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 469 op/s; 4656/261909 objects degraded (1.778%); 28630/261909 objects misplaced (10.931%); 5.1 MiB/s, 81 objects/s recovering 2024-03-24T12:30:14.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:13 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:30:14.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:13 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:30:14.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:14.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:14.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:14.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:14.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:14.321 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:14.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:14.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:13 smithi190 ceph-mon[33748]: pgmap v150: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.3 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 469 op/s; 4656/261909 objects degraded (1.778%); 28630/261909 objects misplaced (10.931%); 5.1 MiB/s, 81 objects/s recovering 2024-03-24T12:30:14.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:13 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:30:14.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:13 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:30:14.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:14.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:14.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:14.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:14.591 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:14.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:14.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:14.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:14.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:14.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:14.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:14.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:14.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:14.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:14.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:14.976 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:14.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:14.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:15.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:15.124 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:15.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:15.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:15.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:15.255 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:15.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:15.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:15.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:15.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:15.381 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:15.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:14 smithi190 ceph-mon[33748]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T12:30:15.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:14 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 4656/261909 objects degraded (1.778%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:15.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:14 smithi190 ceph-mon[33748]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T12:30:15.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:14 smithi029 ceph-mon[41500]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T12:30:15.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:15 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 4656/261909 objects degraded (1.778%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:15.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:15 smithi029 ceph-mon[41500]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T12:30:15.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:14 smithi029 ceph-mon[29919]: osdmap e128: 8 total, 8 up, 8 in 2024-03-24T12:30:15.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:15 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 4656/261909 objects degraded (1.778%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:15.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:15 smithi029 ceph-mon[29919]: osdmap e129: 8 total, 8 up, 8 in 2024-03-24T12:30:15.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:15.533 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:15.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:15.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:15.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:15.677 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:15.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:15.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:15.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:15.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:15.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:15.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:15.947 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:15.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:16.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:16.081 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:16.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:16.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:16.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:16.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:16.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:16.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:16.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:16.333 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:16.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:15 smithi190 ceph-mon[33748]: pgmap v152: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 507 op/s; 4656/258051 objects degraded (1.804%); 27991/258051 objects misplaced (10.847%); 7.3 MiB/s, 116 objects/s recovering 2024-03-24T12:30:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:15 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:30:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:15 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:30:16.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:15 smithi029 ceph-mon[41500]: pgmap v152: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 507 op/s; 4656/258051 objects degraded (1.804%); 27991/258051 objects misplaced (10.847%); 7.3 MiB/s, 116 objects/s recovering 2024-03-24T12:30:16.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:15 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:30:16.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:15 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:30:16.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:15 smithi029 ceph-mon[29919]: pgmap v152: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 507 op/s; 4656/258051 objects degraded (1.804%); 27991/258051 objects misplaced (10.847%); 7.3 MiB/s, 116 objects/s recovering 2024-03-24T12:30:16.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:15 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:30:16.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:15 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:30:16.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:16.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:16.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:16.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:16.607 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:16.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:16.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:16.781 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:16.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:16.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:16.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:17.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:17.050 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:17.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:17.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:17.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:17.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:17.175 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:17.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:17.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:17.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:17.423 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:17.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:17.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:17.590 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:17.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:17.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:17.715 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:17.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:17.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:17.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:17.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:17.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:17.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:30:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:30:17.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:30:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:30:17.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:17.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:17.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:17.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:18.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:18.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:18.099 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:18.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:18.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:18.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:18.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:18.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:18.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:17 smithi190 ceph-mon[33748]: pgmap v154: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 782 B/s rd, 0 B/s wr, 481 op/s; 4460/255366 objects degraded (1.747%); 27063/255366 objects misplaced (10.598%); 7.9 MiB/s, 126 objects/s recovering 2024-03-24T12:30:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:17 smithi029 ceph-mon[41500]: pgmap v154: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 782 B/s rd, 0 B/s wr, 481 op/s; 4460/255366 objects degraded (1.747%); 27063/255366 objects misplaced (10.598%); 7.9 MiB/s, 126 objects/s recovering 2024-03-24T12:30:18.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:17 smithi029 ceph-mon[29919]: pgmap v154: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 782 B/s rd, 0 B/s wr, 481 op/s; 4460/255366 objects degraded (1.747%); 27063/255366 objects misplaced (10.598%); 7.9 MiB/s, 126 objects/s recovering 2024-03-24T12:30:18.453 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:18.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:18.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:18.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:18.619 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:18.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:18.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:18.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:18.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:18.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:18.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:30:18.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:18.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:18.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:18.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:18.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:19.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:19.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:19.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:19.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:19.157 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:19.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:19.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:19.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:19.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:19.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:19.304 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:19.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:19.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:19.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:19.451 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:19.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:19.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:19.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:19.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:19.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:19.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:19.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:19.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:19.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:19.840 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:19.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:19.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:19.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:19.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:19.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:20.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:20.102 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:20.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:20.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:20.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:20.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:19 smithi029 ceph-mon[41500]: pgmap v155: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 655 op/s; 4460/249099 objects degraded (1.790%); 26450/249099 objects misplaced (10.618%); 8.3 MiB/s, 133 objects/s recovering 2024-03-24T12:30:20.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:19 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 4460/249099 objects degraded (1.790%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:20.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:19 smithi029 ceph-mon[29919]: pgmap v155: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 655 op/s; 4460/249099 objects degraded (1.790%); 26450/249099 objects misplaced (10.618%); 8.3 MiB/s, 133 objects/s recovering 2024-03-24T12:30:20.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:19 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 4460/249099 objects degraded (1.790%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:20.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:20.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:20.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:20.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:20.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:19 smithi190 ceph-mon[33748]: pgmap v155: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 655 op/s; 4460/249099 objects degraded (1.790%); 26450/249099 objects misplaced (10.618%); 8.3 MiB/s, 133 objects/s recovering 2024-03-24T12:30:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:19 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 4460/249099 objects degraded (1.790%), 1 pg degraded (PG_DEGRADED) 2024-03-24T12:30:20.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:20.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:20.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:20.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:20.612 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:20.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:20.668 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:20 smithi029 podman[92953]: 2024-03-24 12:30:20.223588145 +0000 UTC m=+27.475771825 volume create 09a600bc2c5d3918b47c613da5c4172a23c2fcea36f59df03f75ea1a21d11e07 2024-03-24T12:30:20.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:20.754 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:20.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:20.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:20.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:20.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:20.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:20.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:20.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:21.022 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:21.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:21.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:21.165 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:21.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:20 smithi029 ceph-mon[29919]: pgmap v156: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 647 op/s; 4268/246372 objects degraded (1.732%); 25581/246372 objects misplaced (10.383%); 12 MiB/s, 197 objects/s recovering 2024-03-24T12:30:21.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:20 smithi029 ceph-mon[41500]: pgmap v156: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 647 op/s; 4268/246372 objects degraded (1.732%); 25581/246372 objects misplaced (10.383%); 12 MiB/s, 197 objects/s recovering 2024-03-24T12:30:21.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:21.311 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:21.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:20 smithi190 ceph-mon[33748]: pgmap v156: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 647 op/s; 4268/246372 objects degraded (1.732%); 25581/246372 objects misplaced (10.383%); 12 MiB/s, 197 objects/s recovering 2024-03-24T12:30:21.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:21.442 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:21.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:21.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:21.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:21.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:21.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:21.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:21.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:21.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:21.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:21.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:21.946 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:21.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:22.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:22.074 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:22.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:22.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:22.224 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:22.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:22.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:22.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:22.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:22.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:22.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:22.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:22.491 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:22.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:22.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:22.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:22.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:22.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:22.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:22.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:22.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:22.887 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:22.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:22.989 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:23.014 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:23.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:23.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:23.140 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:23.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:23.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:23.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:23.270 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:23.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:23.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:23.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:23.401 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:23.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:23.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:23.418 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:22 smithi029 podman[92953]: 2024-03-24 12:30:22.960293199 +0000 UTC m=+30.212476919 container create d88a22cde729f6720b901e219a3336342391ca4779bc0e11785eeaa08598a689 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:30:23.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:23.543 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:23.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:23.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:23.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:23.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:23.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:23.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:23.815 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:23.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:23.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:23 smithi190 ceph-mon[33748]: pgmap v157: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 569 op/s; 4268/243552 objects degraded (1.752%); 25581/243552 objects misplaced (10.503%); 9.7 MiB/s, 154 objects/s recovering 2024-03-24T12:30:23.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:23.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:23 smithi029 ceph-mon[41500]: pgmap v157: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 569 op/s; 4268/243552 objects degraded (1.752%); 25581/243552 objects misplaced (10.503%); 9.7 MiB/s, 154 objects/s recovering 2024-03-24T12:30:23.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:23 smithi029 ceph-mon[29919]: pgmap v157: 158 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 569 op/s; 4268/243552 objects degraded (1.752%); 25581/243552 objects misplaced (10.503%); 9.7 MiB/s, 154 objects/s recovering 2024-03-24T12:30:23.944 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:23.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:23.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:24.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:24.074 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:24.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:24.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:24.203 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:24.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:24.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:24.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:24.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:24.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:24.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:24.454 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:24.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:24.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:24.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:24.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:24.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:24.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:24.721 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:24.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:24.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:24.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:24.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:24.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:24.997 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:24.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:24.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:25.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:25.127 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:25.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:25.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:25.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:25.252 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:25.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:25.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:25.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:25.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:25.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:25.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:25.522 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:25.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:25.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:25.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:25.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:25.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:25.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:25.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:25.773 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:25.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:25.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:25 smithi190 ceph-mon[33748]: pgmap v158: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 612 op/s; 4268/239706 objects degraded (1.781%); 24852/239706 objects misplaced (10.368%); 10 MiB/s, 160 objects/s recovering 2024-03-24T12:30:25.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:25 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 4268/239706 objects degraded (1.781%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:25.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:25.913 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:25.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:25.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:25 smithi029 ceph-mon[29919]: pgmap v158: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 612 op/s; 4268/239706 objects degraded (1.781%); 24852/239706 objects misplaced (10.368%); 10 MiB/s, 160 objects/s recovering 2024-03-24T12:30:25.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:25 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 4268/239706 objects degraded (1.781%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:25.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:25 smithi029 ceph-mon[41500]: pgmap v158: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 612 op/s; 4268/239706 objects degraded (1.781%); 24852/239706 objects misplaced (10.368%); 10 MiB/s, 160 objects/s recovering 2024-03-24T12:30:25.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:25 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 4268/239706 objects degraded (1.781%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:26.014 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:26.064 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:26.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:26.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:26.166 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:26.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:26.209 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:26.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:26.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:26.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:26.334 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:26.334 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:26.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:26.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:26.459 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:26.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:26.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:26.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:26.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:26.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:26.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:26.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:26.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:26.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:26.863 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:26.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:26.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:27.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:27.006 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:27.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:27.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:27.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:27.140 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:27.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:27.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:27.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:27.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:27.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:27.390 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:27.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:27.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:27.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:27.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:27.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:27.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:27.672 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:27.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:27.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:27.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:27.812 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:27.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:27.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:27 smithi190 ceph-mon[33748]: pgmap v159: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 887 B/s rd, 0 B/s wr, 527 op/s; 4048/237147 objects degraded (1.707%); 23965/237147 objects misplaced (10.106%); 8.6 MiB/s, 137 objects/s recovering 2024-03-24T12:30:27.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:30:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:30:27.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:27.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:27 smithi029 ceph-mon[29919]: pgmap v159: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 887 B/s rd, 0 B/s wr, 527 op/s; 4048/237147 objects degraded (1.707%); 23965/237147 objects misplaced (10.106%); 8.6 MiB/s, 137 objects/s recovering 2024-03-24T12:30:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:27 smithi029 ceph-mon[41500]: pgmap v159: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.8 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 887 B/s rd, 0 B/s wr, 527 op/s; 4048/237147 objects degraded (1.707%); 23965/237147 objects misplaced (10.106%); 8.6 MiB/s, 137 objects/s recovering 2024-03-24T12:30:27.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:30:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:30:27.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:27.954 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:27.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:28.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:28.079 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:28.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:28.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:28.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:28.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:28.313 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:28.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:28.341 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:28.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:28.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:28.484 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:28.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:28.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:28.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:28.627 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:28.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:28.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:28.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:28.775 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:28.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:28.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:28.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:28.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:28.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:28.920 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:28.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:29.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:29.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:29.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:29.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:29.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:29.196 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:29.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:29.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:29.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:29.340 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:29.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:29.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:29.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:29.470 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:29.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:29.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:29.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:29.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:29.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:29.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:29.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:29.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:29.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:29.868 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:29.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:29.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:29 smithi190 ceph-mon[33748]: pgmap v160: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 675 op/s; 4048/231084 objects degraded (1.752%); 23331/231084 objects misplaced (10.096%); 9.5 MiB/s, 151 objects/s recovering 2024-03-24T12:30:29.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:29 smithi029 ceph-mon[29919]: pgmap v160: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 675 op/s; 4048/231084 objects degraded (1.752%); 23331/231084 objects misplaced (10.096%); 9.5 MiB/s, 151 objects/s recovering 2024-03-24T12:30:29.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:29 smithi029 ceph-mon[41500]: pgmap v160: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 675 op/s; 4048/231084 objects degraded (1.752%); 23331/231084 objects misplaced (10.096%); 9.5 MiB/s, 151 objects/s recovering 2024-03-24T12:30:29.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:30.010 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:30.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:30.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:30.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:30.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:30.136 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:30.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:30.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:30.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:30.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:30.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:30.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:30.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:30.519 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:30.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:30.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:30.669 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:30.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:30.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:30.804 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:30.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:30.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:30.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:30 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 4048/231084 objects degraded (1.752%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:30.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:30.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:30 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 4048/231084 objects degraded (1.752%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:30.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:30 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 4048/231084 objects degraded (1.752%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:30.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:30.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:31.038 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:31.072 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:31.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:31.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:31.202 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:31.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:31.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:31.344 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:31.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:31.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:31.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:31.492 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:31.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:31.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:31.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:31.618 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:31.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:31.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:31.744 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:31.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:31.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:31.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:31.878 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:31.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:31.980 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:32.023 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:32.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:32.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:32.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:32.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:32.150 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:32.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:32.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:32.296 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:32.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:32.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:31 smithi190 ceph-mon[33748]: pgmap v161: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 571 op/s; 3890/228549 objects degraded (1.702%); 22452/228549 objects misplaced (9.824%); 11 MiB/s, 175 objects/s recovering 2024-03-24T12:30:32.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:32.397 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:32.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:31 smithi029 ceph-mon[29919]: pgmap v161: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 571 op/s; 3890/228549 objects degraded (1.702%); 22452/228549 objects misplaced (9.824%); 11 MiB/s, 175 objects/s recovering 2024-03-24T12:30:32.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:32.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:31 smithi029 ceph-mon[41500]: pgmap v161: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 571 op/s; 3890/228549 objects degraded (1.702%); 22452/228549 objects misplaced (9.824%); 11 MiB/s, 175 objects/s recovering 2024-03-24T12:30:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:32.442 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:32.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:32.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:32.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:32.572 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:32.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:32.573 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:32.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:32.703 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:32.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:32.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:32.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:32.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:32.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:32.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:32.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:32.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:32.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:33.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:33.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:33.092 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:33.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:33.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:33.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:33.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:33.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:33.359 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:33.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:32 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:30:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:32 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:30:33.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:32 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:30:33.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:32 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:30:33.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:32 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:30:33.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:32 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:30:33.419 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:32 smithi029 podman[92953]: 2024-03-24 12:30:32.955610602 +0000 UTC m=+40.207794310 container init d88a22cde729f6720b901e219a3336342391ca4779bc0e11785eeaa08598a689 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:30:33.419 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:32 smithi029 podman[92953]: 2024-03-24 12:30:32.972336679 +0000 UTC m=+40.224520356 container start d88a22cde729f6720b901e219a3336342391ca4779bc0e11785eeaa08598a689 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:30:33.419 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:32 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:30:32.993Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-24T12:30:33.419 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:32 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:30:32.994Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-24T12:30:33.419 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:32 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:30:32.997Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.29 port=9094 2024-03-24T12:30:33.419 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:32 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:30:32.999Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-24T12:30:33.420 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:30:33.055Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T12:30:33.420 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:30:33.056Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-24T12:30:33.420 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:30:33.061Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-24T12:30:33.420 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:30:33.061Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-24T12:30:33.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:33.509 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:33.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:33.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:33.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:33.645 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:33.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:33.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:33.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:33.776 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:33.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:33.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:33.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:33.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:33.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:33.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:34.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:34.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:34.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:34.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:34.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:34.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:34.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:34.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:34.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:34.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:34.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:33 smithi190 ceph-mon[33748]: pgmap v162: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 577 op/s; 3890/225609 objects degraded (1.724%); 22452/225609 objects misplaced (9.952%); 8.3 MiB/s, 132 objects/s recovering 2024-03-24T12:30:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:33 smithi190 ceph-mon[33748]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T12:30:34.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:34.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:33 smithi029 ceph-mon[29919]: pgmap v162: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 577 op/s; 3890/225609 objects degraded (1.724%); 22452/225609 objects misplaced (9.952%); 8.3 MiB/s, 132 objects/s recovering 2024-03-24T12:30:34.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:33 smithi029 ceph-mon[29919]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T12:30:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:33 smithi029 ceph-mon[41500]: pgmap v162: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 5 active+recovering+undersized+remapped, 150 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 577 op/s; 3890/225609 objects degraded (1.724%); 22452/225609 objects misplaced (9.952%); 8.3 MiB/s, 132 objects/s recovering 2024-03-24T12:30:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:33 smithi029 ceph-mon[41500]: osdmap e130: 8 total, 8 up, 8 in 2024-03-24T12:30:34.433 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:34.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:34.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:34.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:34.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:34.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:34.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:34.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:34.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:34.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:34.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:34.818 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:34.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:34.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:34.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:34.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:35.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:35.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:35.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:35.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:35.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:35.220 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:35.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:35.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:35.235 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:34 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:30:34.999Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000183873s 2024-03-24T12:30:35.236 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:34 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 3890/225609 objects degraded (1.724%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:35.236 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:34 smithi029 ceph-mon[41500]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T12:30:35.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:34 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 3890/225609 objects degraded (1.724%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:35.236 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:34 smithi029 ceph-mon[29919]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T12:30:35.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:35.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:35.346 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:35.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:34 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 3890/225609 objects degraded (1.724%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:34 smithi190 ceph-mon[33748]: osdmap e131: 8 total, 8 up, 8 in 2024-03-24T12:30:35.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:35.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:35.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:35.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:35.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:35.613 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:35.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:35.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:35.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:35.740 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:35.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:35.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:35.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:35.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:35.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:36.011 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:36.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:36.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:36.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:36.158 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:36.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:36.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:36.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:36.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:36.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:35 smithi190 ceph-mon[33748]: pgmap v164: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 591 op/s; 3890/221976 objects degraded (1.752%); 21714/221976 objects misplaced (9.782%); 10 MiB/s, 164 objects/s recovering 2024-03-24T12:30:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:35 smithi190 ceph-mon[33748]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T12:30:36.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:36.411 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:36.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:36.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:36.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:35 smithi029 ceph-mon[41500]: pgmap v164: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 591 op/s; 3890/221976 objects degraded (1.752%); 21714/221976 objects misplaced (9.782%); 10 MiB/s, 164 objects/s recovering 2024-03-24T12:30:36.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:35 smithi029 ceph-mon[41500]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T12:30:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:35 smithi029 ceph-mon[29919]: pgmap v164: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 591 op/s; 3890/221976 objects degraded (1.752%); 21714/221976 objects misplaced (9.782%); 10 MiB/s, 164 objects/s recovering 2024-03-24T12:30:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:35 smithi029 ceph-mon[29919]: osdmap e132: 8 total, 8 up, 8 in 2024-03-24T12:30:36.513 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:36.545 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:36.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:36.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:36.678 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:36.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:36.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:36.825 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:36.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:36.927 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:36.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:36.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:36.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:37.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:37.101 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:37.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:37.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:37.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:37.232 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:37.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:37.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:37.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:37.358 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:37.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:36 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:30:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:36 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:30:37.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:36 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:30:37.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:36 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:30:37.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:36 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:30:37.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:36 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:30:37.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:37.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:37.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:37.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:37.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:37.631 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:37.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:37.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:37.761 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:37.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:37.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:37.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:30:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:30:37.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:37.892 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:37.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:37.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:30:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:30:37.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:38.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:38.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:38.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:38.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:38.146 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:38.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:38.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:38.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:37 smithi190 ceph-mon[33748]: pgmap v167: 158 pgs: 1 active+undersized+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 512 op/s; 3713/219330 objects degraded (1.693%); 20809/219330 objects misplaced (9.488%); 9.4 MiB/s, 149 objects/s recovering 2024-03-24T12:30:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:37 smithi190 ceph-mon[33748]: 7.16 scrub starts 2024-03-24T12:30:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:37 smithi190 ceph-mon[33748]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T12:30:38.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:37 smithi190 ceph-mon[33748]: 7.16 scrub ok 2024-03-24T12:30:38.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:37 smithi029 ceph-mon[29919]: pgmap v167: 158 pgs: 1 active+undersized+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 512 op/s; 3713/219330 objects degraded (1.693%); 20809/219330 objects misplaced (9.488%); 9.4 MiB/s, 149 objects/s recovering 2024-03-24T12:30:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:38 smithi029 ceph-mon[29919]: 7.16 scrub starts 2024-03-24T12:30:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:38 smithi029 ceph-mon[29919]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T12:30:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:38 smithi029 ceph-mon[29919]: 7.16 scrub ok 2024-03-24T12:30:38.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:37 smithi029 ceph-mon[41500]: pgmap v167: 158 pgs: 1 active+undersized+remapped, 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 150 active+clean; 4.5 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 512 op/s; 3713/219330 objects degraded (1.693%); 20809/219330 objects misplaced (9.488%); 9.4 MiB/s, 149 objects/s recovering 2024-03-24T12:30:38.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:38 smithi029 ceph-mon[41500]: 7.16 scrub starts 2024-03-24T12:30:38.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:38 smithi029 ceph-mon[41500]: osdmap e133: 8 total, 8 up, 8 in 2024-03-24T12:30:38.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:38 smithi029 ceph-mon[41500]: 7.16 scrub ok 2024-03-24T12:30:38.436 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:38.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:38.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:38.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:38.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:38.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:38.692 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:38.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:38.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:38.820 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:38.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:38.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:38.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:38.945 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:38.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:38.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:39.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:39.075 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:39.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:39.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:39.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:38 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:39.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:38 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:39.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:39.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:39.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:39.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:39.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:39.333 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:39.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:38 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:39.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:39.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:39.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:39.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:39.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:39.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:39.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:39.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:39.725 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:39.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:39.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:39.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:39.876 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:39.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:39.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:39.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:40.007 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:40.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:40.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:40.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:40.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:40.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:40.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:40.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:40.315 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:39 smithi029 ceph-mon[29919]: pgmap v169: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 927 B/s rd, 0 op/s; 3713/212628 objects degraded (1.746%); 19512/212628 objects misplaced (9.177%) 2024-03-24T12:30:40.315 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:39 smithi029 ceph-mon[29919]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:30:40.315 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:39 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 3713/212628 objects degraded (1.746%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:40.315 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:39 smithi029 ceph-mon[41500]: pgmap v169: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 927 B/s rd, 0 op/s; 3713/212628 objects degraded (1.746%); 19512/212628 objects misplaced (9.177%) 2024-03-24T12:30:40.315 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:39 smithi029 ceph-mon[41500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:30:40.316 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:39 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 3713/212628 objects degraded (1.746%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:40.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:39 smithi190 ceph-mon[33748]: pgmap v169: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 927 B/s rd, 0 op/s; 3713/212628 objects degraded (1.746%); 19512/212628 objects misplaced (9.177%) 2024-03-24T12:30:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:39 smithi190 ceph-mon[33748]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:30:40.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:39 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 3713/212628 objects degraded (1.746%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:40.420 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:40.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:40.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:40.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:40.559 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:40.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:40.660 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:40.702 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:40.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:40.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:40.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:40.846 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:40.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:40.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:40.991 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:40.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:40.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:41.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:41.133 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:41.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:41.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:41.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:41.276 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:41.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:41.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:41.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:41 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:30:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:41 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:30:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:41 smithi190 ceph-mon[33748]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T12:30:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:41 smithi190 ceph-mon[33748]: pgmap v171: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 924 B/s rd, 0 op/s; 3532/210324 objects degraded (1.679%); 18773/210324 objects misplaced (8.926%) 2024-03-24T12:30:41.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:41 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:30:41.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:41 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:30:41.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:41 smithi029 ceph-mon[29919]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T12:30:41.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:41 smithi029 ceph-mon[29919]: pgmap v171: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 924 B/s rd, 0 op/s; 3532/210324 objects degraded (1.679%); 18773/210324 objects misplaced (8.926%) 2024-03-24T12:30:41.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:41 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:30:41.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:41 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:30:41.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:41 smithi029 ceph-mon[41500]: osdmap e134: 8 total, 8 up, 8 in 2024-03-24T12:30:41.422 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:41 smithi029 ceph-mon[41500]: pgmap v171: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 924 B/s rd, 0 op/s; 3532/210324 objects degraded (1.679%); 18773/210324 objects misplaced (8.926%) 2024-03-24T12:30:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:41.428 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:41.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:41.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:41.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:41.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:41.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:41.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:41.711 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:41.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:41.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:41.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:41.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:41.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:41.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:41.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:42.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:42.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:42.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:42.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:42.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:42.260 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:42.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:42.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:42.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:42.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:42.536 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:42.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:42.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:42.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:42.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:42.769 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:42.815 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:42.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:42.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:42.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:42.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:42.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:42.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:43.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:43.088 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:43.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:43.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:43.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:43.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:43.232 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:43.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:43.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:43.362 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:43.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:43.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:43.418 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:30:43.005Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.006482163s 2024-03-24T12:30:43.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:43.487 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:43.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:43.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:43.631 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:43.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:43.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:43.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:43.773 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:43.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:43.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:43.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:43 smithi190 ceph-mon[33748]: pgmap v172: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 14 op/s; 3532/206568 objects degraded (1.710%); 18773/206568 objects misplaced (9.088%) 2024-03-24T12:30:43.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:43 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:30:43.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:43 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:30:43.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:43.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:43.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:43 smithi029 ceph-mon[29919]: pgmap v172: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 14 op/s; 3532/206568 objects degraded (1.710%); 18773/206568 objects misplaced (9.088%) 2024-03-24T12:30:43.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:43 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:30:43.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:43 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:30:43.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:43 smithi029 ceph-mon[41500]: pgmap v172: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 14 op/s; 3532/206568 objects degraded (1.710%); 18773/206568 objects misplaced (9.088%) 2024-03-24T12:30:43.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:43 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:30:43.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:43 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:30:44.005 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:44.054 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:44.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:44.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:44.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:44.198 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:44.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:44.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:44.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:44.322 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:44.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:44.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:44.459 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:44.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:44.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:44.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:44.600 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:44.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:44.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:44.731 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:44.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:44.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:44.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:44.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:44.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:44.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:44.918 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:44 smithi029 bash[92953]: d88a22cde729f6720b901e219a3336342391ca4779bc0e11785eeaa08598a689 2024-03-24T12:30:44.918 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:30:44 smithi029 systemd[1]: Started Ceph alertmanager.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:30:44.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:44.980 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:44.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:44.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:45.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:45.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:45.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:45.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:45.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:45.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:45.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:45.383 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:45.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:45.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:45.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:45.509 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:45.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:45.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:45.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:45.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:45.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:45.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:45.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:45.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:45.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:45.899 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:45.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:46.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:46.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:46.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:46.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:46.060 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:45 smithi190 ceph-mon[33748]: pgmap v173: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 197 op/s; 3532/202116 objects degraded (1.748%); 17252/202116 objects misplaced (8.536%) 2024-03-24T12:30:46.060 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:45 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:46.060 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:45 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:46.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:45 smithi190 ceph-mon[33748]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T12:30:46.061 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:45 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 3532/202116 objects degraded (1.748%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:46.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:46.163 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:46.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:45 smithi029 ceph-mon[41500]: pgmap v173: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 197 op/s; 3532/202116 objects degraded (1.748%); 17252/202116 objects misplaced (8.536%) 2024-03-24T12:30:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:45 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:45 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:45 smithi029 ceph-mon[41500]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T12:30:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:45 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 3532/202116 objects degraded (1.748%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:46.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:45 smithi029 ceph-mon[29919]: pgmap v173: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 197 op/s; 3532/202116 objects degraded (1.748%); 17252/202116 objects misplaced (8.536%) 2024-03-24T12:30:46.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:45 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:46.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:45 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:46.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:45 smithi029 ceph-mon[29919]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-24T12:30:46.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:45 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 3532/202116 objects degraded (1.748%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:46.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:46.288 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:46.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:46.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:46.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:46.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:46.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:46.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:46.565 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:46.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:46.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:46.710 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:46.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:46.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:46.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:46.854 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:46.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:46.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:46.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:46.987 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:46.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:46.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:47.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:47.115 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:46 smithi190 ceph-mon[33748]: Reconfiguring daemon prometheus.a on smithi190 2024-03-24T12:30:47.119 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:47.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:47.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:47.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:46 smithi029 ceph-mon[29919]: Reconfiguring daemon prometheus.a on smithi190 2024-03-24T12:30:47.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:46 smithi029 ceph-mon[41500]: Reconfiguring daemon prometheus.a on smithi190 2024-03-24T12:30:47.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:47.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:47.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:47.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:47.381 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:47.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:47.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:47.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:47.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:47.525 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:47.525 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:47.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:47.650 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:47.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:47.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:47.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:47.795 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:47.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:47.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:47 smithi190 ceph-mon[33748]: pgmap v174: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 272 op/s; 3310/199101 objects degraded (1.662%); 16417/199101 objects misplaced (8.246%) 2024-03-24T12:30:47.823 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:47.823 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:30:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:30:47.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:47 smithi029 ceph-mon[29919]: pgmap v174: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 272 op/s; 3310/199101 objects degraded (1.662%); 16417/199101 objects misplaced (8.246%) 2024-03-24T12:30:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:47 smithi029 ceph-mon[41500]: pgmap v174: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 4.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 272 op/s; 3310/199101 objects degraded (1.662%); 16417/199101 objects misplaced (8.246%) 2024-03-24T12:30:47.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:30:47.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:30:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:30:47.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:47.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:47.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:48.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:48.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:48.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:48.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:48.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:48.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:48.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:48.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:48.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:48.325 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:48.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:48.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:48.452 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:48.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:48.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:48.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:48.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:48.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:48.709 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:48.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:48.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:48.857 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:48.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:48.878 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:48 smithi190 ceph-mon[33748]: 3.9 scrub starts 2024-03-24T12:30:48.878 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:48 smithi190 ceph-mon[33748]: 3.9 scrub ok 2024-03-24T12:30:48.879 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:48 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:48.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:48.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:48.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:48.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:49.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:49.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:49.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:49.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:48 smithi029 ceph-mon[29919]: 3.9 scrub starts 2024-03-24T12:30:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:48 smithi029 ceph-mon[29919]: 3.9 scrub ok 2024-03-24T12:30:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:48 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:49.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:48 smithi029 ceph-mon[41500]: 3.9 scrub starts 2024-03-24T12:30:49.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:48 smithi029 ceph-mon[41500]: 3.9 scrub ok 2024-03-24T12:30:49.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:48 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:49.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:49.250 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:49.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:49.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:49.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:49.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:49.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:49.522 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:49.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:49.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:49.581 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 systemd[1]: Stopping Ceph prometheus.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:30:49.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:49.672 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:49.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:49.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:49.808 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:49.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:49.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-mon[33748]: pgmap v175: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 699 op/s; 3310/191673 objects degraded (1.727%); 14880/191673 objects misplaced (7.763%); 13 MiB/s, 213 objects/s recovering 2024-03-24T12:30:49.868 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.584Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.584Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.584Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.584Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.584Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.584Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.584Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.584Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.584Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.585Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.585Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[64308]: ts=2024-03-24T12:30:49.585Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T12:30:49.869 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:49 smithi190 podman[78940]: 2024-03-24 12:30:49.595310786 +0000 UTC m=+0.148769596 container died 3c089fa4a41da5e73cc096a5228d811a557319979d32f7adf54dfc5834524240 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:30:49.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:49.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:49 smithi029 ceph-mon[41500]: pgmap v175: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 699 op/s; 3310/191673 objects degraded (1.727%); 14880/191673 objects misplaced (7.763%); 13 MiB/s, 213 objects/s recovering 2024-03-24T12:30:49.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:49 smithi029 ceph-mon[29919]: pgmap v175: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped, 151 active+clean; 3.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 699 op/s; 3310/191673 objects degraded (1.727%); 14880/191673 objects misplaced (7.763%); 13 MiB/s, 213 objects/s recovering 2024-03-24T12:30:49.934 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:49.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:50.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:50.077 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:50.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:50.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:50.133 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:50 smithi190 podman[78940]: 2024-03-24 12:30:50.071673576 +0000 UTC m=+0.625132386 container cleanup 3c089fa4a41da5e73cc096a5228d811a557319979d32f7adf54dfc5834524240 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:30:50.133 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:50 smithi190 bash[78940]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T12:30:50.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:50.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:50.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:50.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:50.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:50.342 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:50.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:50.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:50.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:50.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:50.468 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:50.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:50.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:50.617 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:50.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:50.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:50.633 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:50 smithi190 podman[78957]: 2024-03-24 12:30:50.382606415 +0000 UTC m=+0.787419698 container remove 3c089fa4a41da5e73cc096a5228d811a557319979d32f7adf54dfc5834524240 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:30:50.634 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:50 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-24T12:30:50.634 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:50 smithi190 systemd[1]: Stopped Ceph prometheus.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:30:50.634 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:50 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 1.471s CPU time. 2024-03-24T12:30:50.634 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:50 smithi190 systemd[1]: Starting Ceph prometheus.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:30:50.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:50.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:50.767 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:50.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:50.868 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:50.913 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:50.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:51.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:51.032 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:50 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 3310/191673 objects degraded (1.727%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:51.033 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:50 smithi190 podman[79099]: 2024-03-24 12:30:50.677383089 +0000 UTC m=+0.030259536 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-24T12:30:51.033 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:50 smithi190 podman[79099]: 2024-03-24 12:30:50.800582289 +0000 UTC m=+0.153458703 container create 46683ac055d1e2a14266996917c17018fbb5bb9208453c9c1431df51b0f85b2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:30:51.033 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 podman[79099]: 2024-03-24 12:30:51.036281681 +0000 UTC m=+0.389158092 container init 46683ac055d1e2a14266996917c17018fbb5bb9208453c9c1431df51b0f85b2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:30:51.040 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:51.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:51.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:51.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:51.176 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:50 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 3310/191673 objects degraded (1.727%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:51.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:50 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 3310/191673 objects degraded (1.727%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:51.183 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:51.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:51.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:51.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:51.313 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:51.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:51.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:51.384 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 podman[79099]: 2024-03-24 12:30:51.041779375 +0000 UTC m=+0.394655781 container start 46683ac055d1e2a14266996917c17018fbb5bb9208453c9c1431df51b0f85b2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:30:51.384 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.064Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-24T12:30:51.384 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.065Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.065Z caller=main.go:570 level=info host_details="(Linux 5.14.0-430.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Mar 14 17:54:49 UTC 2024 x86_64 smithi190 (none))" 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.065Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.065Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.066Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.066Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.067Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.067Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.069Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.069Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.413µs 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.069Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.071Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.071Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-24T12:30:51.385 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.071Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.194µs wal_replay_duration=1.850445ms wbl_replay_duration=99ns total_replay_duration=1.884631ms 2024-03-24T12:30:51.386 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.072Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-24T12:30:51.386 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.072Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-24T12:30:51.386 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.072Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-24T12:30:51.386 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.089Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.660249ms db_storage=676ns remote_storage=1.46µs web_handler=383ns query_engine=469ns scrape=185.593µs scrape_sd=84.208µs notify=9.102µs notify_sd=10.806µs rules=17.163266ms tracing=5.24µs 2024-03-24T12:30:51.386 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.089Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-24T12:30:51.386 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:30:51.089Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-24T12:30:51.386 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 bash[79099]: 46683ac055d1e2a14266996917c17018fbb5bb9208453c9c1431df51b0f85b2d 2024-03-24T12:30:51.386 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:30:51 smithi190 systemd[1]: Started Ceph prometheus.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:30:51.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:51.446 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:51.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:51.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:51.504 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STOPPING 2024-03-24T12:30:51.504 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:30:51.504 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STOPPED 2024-03-24T12:30:51.504 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STARTING 2024-03-24T12:30:51.505 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Serving on http://:::9283 2024-03-24T12:30:51.505 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STARTED 2024-03-24T12:30:51.505 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STOPPING 2024-03-24T12:30:51.505 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:30:51.505 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STOPPED 2024-03-24T12:30:51.505 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STARTING 2024-03-24T12:30:51.505 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Serving on http://:::9283 2024-03-24T12:30:51.505 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STARTED 2024-03-24T12:30:51.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:51.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:51.595 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:51.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:51.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:51.726 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:51.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:51.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:51.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:51.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:51.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:51.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:51.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STOPPING 2024-03-24T12:30:51.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:30:51.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STOPPED 2024-03-24T12:30:51.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STARTING 2024-03-24T12:30:51.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Serving on http://:::9283 2024-03-24T12:30:51.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:30:51] ENGINE Bus STARTED 2024-03-24T12:30:51.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:51.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:52.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:52.133 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:52.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:52.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:52.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:52.280 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:52.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:52.339 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: pgmap v176: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 976 B/s rd, 0 B/s wr, 675 op/s; 3078/189111 objects degraded (1.628%); 14035/189111 objects misplaced (7.422%); 13 MiB/s, 208 objects/s recovering 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.340 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:51 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:30:52.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:52.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:52.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:52.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: pgmap v176: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 976 B/s rd, 0 B/s wr, 675 op/s; 3078/189111 objects degraded (1.628%); 14035/189111 objects misplaced (7.422%); 13 MiB/s, 208 objects/s recovering 2024-03-24T12:30:52.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: pgmap v176: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 976 B/s rd, 0 B/s wr, 675 op/s; 3078/189111 objects degraded (1.628%); 14035/189111 objects misplaced (7.422%); 13 MiB/s, 208 objects/s recovering 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:52.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:51 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:30:52.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:52.537 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:52.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:52.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:52.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:52.663 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:52.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:52.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:52.807 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:52.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:52.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:52.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:52.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:52.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:53.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:53.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:53.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:53.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:53.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:53.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:53.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:53.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:53.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:53.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:53.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T12:30:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T12:30:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T12:30:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T12:30:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T12:30:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T12:30:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:30:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:30:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:30:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:30:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:52 smithi190 ceph-mon[33748]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T12:30:53.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T12:30:53.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T12:30:53.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T12:30:53.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T12:30:53.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T12:30:53.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T12:30:53.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:30:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:30:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:30:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:30:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[41500]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T12:30:53.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T12:30:53.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi029.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T12:30:53.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T12:30:53.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi190.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T12:30:53.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T12:30:53.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi190.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T12:30:53.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:30:53.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:30:53.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:30:53.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:30:53.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:52 smithi029 ceph-mon[29919]: osdmap e135: 8 total, 8 up, 8 in 2024-03-24T12:30:53.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:53.475 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:53.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:53.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:53.608 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:53.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:53.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:53.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:53.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:53.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:53.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:53.897 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:53.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:53.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:54.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:54.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:54.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:54.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:54.151 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:54.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:54.207 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:53 smithi029 ceph-mon[41500]: pgmap v178: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 696 op/s; 3078/185685 objects degraded (1.658%); 14035/185685 objects misplaced (7.558%); 14 MiB/s, 218 objects/s recovering 2024-03-24T12:30:54.207 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:54 smithi029 ceph-mon[41500]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T12:30:54.208 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:53 smithi029 ceph-mon[29919]: pgmap v178: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 696 op/s; 3078/185685 objects degraded (1.658%); 14035/185685 objects misplaced (7.558%); 14 MiB/s, 218 objects/s recovering 2024-03-24T12:30:54.208 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:54 smithi029 ceph-mon[29919]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T12:30:54.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:54.294 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:54.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:54.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:54.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:53 smithi190 ceph-mon[33748]: pgmap v178: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 151 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 696 op/s; 3078/185685 objects degraded (1.658%); 14035/185685 objects misplaced (7.558%); 14 MiB/s, 218 objects/s recovering 2024-03-24T12:30:54.354 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:53 smithi190 ceph-mon[33748]: osdmap e136: 8 total, 8 up, 8 in 2024-03-24T12:30:54.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:54.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:54.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:54.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:54.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:54.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:54.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:54.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:54.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:54.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:54.848 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:54.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:54.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:54.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:54.994 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:54.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:55.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:55.119 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:55.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:55.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:55.244 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:55.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:55.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:55.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:55.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:54 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 3078/185685 objects degraded (1.658%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:55.384 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:55.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:54 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 3078/185685 objects degraded (1.658%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:54 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 3078/185685 objects degraded (1.658%), 1 pg degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-24T12:30:55.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:55.531 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:55.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:55.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:55.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:55.678 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:55.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:55.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:55.802 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:55.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:55.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:55.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:55.946 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:55.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:55.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:56.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:56.078 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:56.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:56.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:55 smithi190 ceph-mon[33748]: pgmap v180: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 716 op/s; 3078/181929 objects degraded (1.692%); 12863/181929 objects misplaced (7.070%); 13 MiB/s, 211 objects/s recovering 2024-03-24T12:30:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:55 smithi190 ceph-mon[33748]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:30:56.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:55 smithi029 ceph-mon[29919]: pgmap v180: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 716 op/s; 3078/181929 objects degraded (1.692%); 12863/181929 objects misplaced (7.070%); 13 MiB/s, 211 objects/s recovering 2024-03-24T12:30:56.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:55 smithi029 ceph-mon[29919]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:30:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:55 smithi029 ceph-mon[41500]: pgmap v180: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 3.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 716 op/s; 3078/181929 objects degraded (1.692%); 12863/181929 objects misplaced (7.070%); 13 MiB/s, 211 objects/s recovering 2024-03-24T12:30:56.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:55 smithi029 ceph-mon[41500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:30:56.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:56.203 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:56.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:56.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:56.329 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:56.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:56.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:56.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:56.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:56.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:56.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:56.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:56.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:56.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:56.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:56.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:56.896 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:56.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:56.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:56.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:57.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:57.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:57.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:57.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:57.154 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:57.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:57.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:57.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:57.306 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:57.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:57.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:57.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:57.447 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:57.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:57.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:57.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:57.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:57.674 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:57.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:57.699 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:57.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:57.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:30:57.847 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:57.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:57.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:30:57.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:30:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:30:57.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:30:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:30:57.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:57.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:57.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:57.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:58.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:58.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:58.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:58.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:58.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:58.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:58.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:30:58.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:58.374 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:58.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:30:58.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:57 smithi190 ceph-mon[33748]: 6.1a scrub starts 2024-03-24T12:30:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:57 smithi190 ceph-mon[33748]: 6.1a scrub ok 2024-03-24T12:30:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:57 smithi190 ceph-mon[33748]: pgmap v181: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 509 op/s; 14824/179454 objects misplaced (8.261%); 10 MiB/s, 164 objects/s recovering 2024-03-24T12:30:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:57 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:57 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-mon[29919]: 6.1a scrub starts 2024-03-24T12:30:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-mon[29919]: 6.1a scrub ok 2024-03-24T12:30:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-mon[29919]: pgmap v181: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 509 op/s; 14824/179454 objects misplaced (8.261%); 10 MiB/s, 164 objects/s recovering 2024-03-24T12:30:58.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:58.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-mon[41500]: 6.1a scrub starts 2024-03-24T12:30:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-mon[41500]: 6.1a scrub ok 2024-03-24T12:30:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-mon[41500]: pgmap v181: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.6 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 509 op/s; 14824/179454 objects misplaced (8.261%); 10 MiB/s, 164 objects/s recovering 2024-03-24T12:30:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:57 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:30:58.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:58.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:58.508 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:58.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:58.609 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:58.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:58.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:58.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:58.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:30:58.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:58.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:58.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:30:58.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:58.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:58.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:59.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:59.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:59.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:58 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:30:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:58 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:30:59.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:59.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:30:59.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:58 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:30:59.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:59 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:30:59.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:58 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:30:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:59 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:30:59.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:30:59.200 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:59.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:30:59.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:30:59.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:59.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:59.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:30:59.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:59.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:59.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:59.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:59.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:30:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:59.584 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:59.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:30:59.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:30:59.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:59.716 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:30:59.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:30:59.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:30:59.860 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:30:59.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:30:59.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:30:59.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:00.002 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:00.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:00.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:00.127 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:00.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:00.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:00.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:00.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:00.260 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:00.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:59 smithi190 ceph-mon[33748]: pgmap v182: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 684 op/s; 14255/172713 objects misplaced (8.254%); 9.1 MiB/s, 144 objects/s recovering 2024-03-24T12:31:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:30:59 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-03-24T12:31:00.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:00.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:59 smithi029 ceph-mon[41500]: pgmap v182: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 684 op/s; 14255/172713 objects misplaced (8.254%); 9.1 MiB/s, 144 objects/s recovering 2024-03-24T12:31:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:30:59 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-03-24T12:31:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:59 smithi029 ceph-mon[29919]: pgmap v182: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 684 op/s; 14255/172713 objects misplaced (8.254%); 9.1 MiB/s, 144 objects/s recovering 2024-03-24T12:31:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:30:59 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-03-24T12:31:00.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:00.548 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:00.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:00.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:00.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:00.689 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:00.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:00.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:00.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:00.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:00.957 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:00.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:00.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:01.059 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:01.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:01.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:01.191 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:01.248 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:01.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:01.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:01.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:01.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:01.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:01.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:00 smithi190 ceph-mon[33748]: 7.b scrub starts 2024-03-24T12:31:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:00 smithi190 ceph-mon[33748]: 3.1a scrub starts 2024-03-24T12:31:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:00 smithi190 ceph-mon[33748]: 3.1a scrub ok 2024-03-24T12:31:01.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:00 smithi029 ceph-mon[29919]: 7.b scrub starts 2024-03-24T12:31:01.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:00 smithi029 ceph-mon[29919]: 3.1a scrub starts 2024-03-24T12:31:01.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:00 smithi029 ceph-mon[29919]: 3.1a scrub ok 2024-03-24T12:31:01.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:00 smithi029 ceph-mon[41500]: 7.b scrub starts 2024-03-24T12:31:01.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:00 smithi029 ceph-mon[41500]: 3.1a scrub starts 2024-03-24T12:31:01.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:00 smithi029 ceph-mon[41500]: 3.1a scrub ok 2024-03-24T12:31:01.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:01.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:01.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:01.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:01.636 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:01.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:01.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:01.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:01.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:01.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:01.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:01.904 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:01.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:02.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:02.051 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:02.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:02.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:02.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:02.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:02.325 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:02.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:02.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:01 smithi190 ceph-mon[33748]: pgmap v183: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 757 op/s; 13263/170658 objects misplaced (7.772%); 11 MiB/s, 183 objects/s recovering 2024-03-24T12:31:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:01 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:31:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:01 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:31:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:01 smithi190 ceph-mon[33748]: 7.b scrub ok 2024-03-24T12:31:02.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:01 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:01 smithi029 ceph-mon[41500]: pgmap v183: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 757 op/s; 13263/170658 objects misplaced (7.772%); 11 MiB/s, 183 objects/s recovering 2024-03-24T12:31:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:01 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:31:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:01 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:31:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:01 smithi029 ceph-mon[41500]: 7.b scrub ok 2024-03-24T12:31:02.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:01 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:02.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:01 smithi029 ceph-mon[29919]: pgmap v183: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 757 op/s; 13263/170658 objects misplaced (7.772%); 11 MiB/s, 183 objects/s recovering 2024-03-24T12:31:02.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:01 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:31:02.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:01 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:31:02.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:01 smithi029 ceph-mon[29919]: 7.b scrub ok 2024-03-24T12:31:02.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:01 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:02.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:02.460 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:02.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:02.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:02.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:02.586 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:02.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:02.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:02.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:02.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:02.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:02.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:02.840 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:02.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:02.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:02.973 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:02.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:03.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:03.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:03.106 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:03.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:03.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:03.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:03.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:03.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:03.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:03.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:03.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:03 smithi190 ceph-mon[33748]: pgmap v184: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 609 op/s; 13263/167430 objects misplaced (7.922%); 9.3 MiB/s, 148 objects/s recovering 2024-03-24T12:31:03.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:02 smithi029 ceph-mon[29919]: pgmap v184: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 609 op/s; 13263/167430 objects misplaced (7.922%); 9.3 MiB/s, 148 objects/s recovering 2024-03-24T12:31:03.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:03 smithi029 ceph-mon[41500]: pgmap v184: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 609 op/s; 13263/167430 objects misplaced (7.922%); 9.3 MiB/s, 148 objects/s recovering 2024-03-24T12:31:03.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:03.496 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:03.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:03.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:03.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:03.621 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:03.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:03.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:03.763 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:03.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:03.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:03.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:03.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:03.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:04.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:04.042 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:04.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:04.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:04.168 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:04.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:04.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:04.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:04.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:04.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:04 smithi190 ceph-mon[33748]: 5.1c scrub starts 2024-03-24T12:31:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:04 smithi190 ceph-mon[33748]: 5.1c scrub ok 2024-03-24T12:31:04.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:04 smithi029 ceph-mon[29919]: 5.1c scrub starts 2024-03-24T12:31:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:04 smithi029 ceph-mon[29919]: 5.1c scrub ok 2024-03-24T12:31:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:04 smithi029 ceph-mon[41500]: 5.1c scrub starts 2024-03-24T12:31:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:04 smithi029 ceph-mon[41500]: 5.1c scrub ok 2024-03-24T12:31:04.430 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:04.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:04.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:04.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:04.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:04.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:04.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:04.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:04.687 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:04.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:04.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:04.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:04.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:04.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:04.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:04.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:04.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:05.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:05.123 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:05.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:05.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:05.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:05.255 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:05.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:05.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:05.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:05.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:05 smithi190 ceph-mon[33748]: pgmap v185: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 919 B/s rd, 0 B/s wr, 601 op/s; 12736/163548 objects misplaced (7.787%); 7.0 MiB/s, 112 objects/s recovering 2024-03-24T12:31:05.389 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:05.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:05.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:05 smithi029 ceph-mon[29919]: pgmap v185: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 919 B/s rd, 0 B/s wr, 601 op/s; 12736/163548 objects misplaced (7.787%); 7.0 MiB/s, 112 objects/s recovering 2024-03-24T12:31:05.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:05 smithi029 ceph-mon[41500]: pgmap v185: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 919 B/s rd, 0 B/s wr, 601 op/s; 12736/163548 objects misplaced (7.787%); 7.0 MiB/s, 112 objects/s recovering 2024-03-24T12:31:05.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:05.532 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:05.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:05.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:05.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:05.676 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:05.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:05.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:05.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:05.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:05.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:05.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:05.936 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:05.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:06.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:06.063 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:06.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:06.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:06.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:06.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:06.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:06.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:06.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:06.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:06 smithi029 ceph-mon[29919]: 4.1 scrub starts 2024-03-24T12:31:06.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:06 smithi029 ceph-mon[29919]: 4.1 scrub ok 2024-03-24T12:31:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:06 smithi029 ceph-mon[41500]: 4.1 scrub starts 2024-03-24T12:31:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:06 smithi029 ceph-mon[41500]: 4.1 scrub ok 2024-03-24T12:31:06.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:06.468 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:06.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:06.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:06.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:06.593 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:06.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:06 smithi190 ceph-mon[33748]: 4.1 scrub starts 2024-03-24T12:31:06.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:06 smithi190 ceph-mon[33748]: 4.1 scrub ok 2024-03-24T12:31:06.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:06.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:06.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:06.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:06.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:06.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:07.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:07.013 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:07.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:07.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:07.138 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:07.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:07.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:07.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:07.264 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:07.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:07.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:07.366 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:07.390 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:07.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:07.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-mon[41500]: 6.4 scrub starts 2024-03-24T12:31:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-mon[41500]: 6.4 scrub ok 2024-03-24T12:31:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-mon[41500]: pgmap v186: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 581 op/s; 11725/161010 objects misplaced (7.282%); 7.4 MiB/s, 117 objects/s recovering 2024-03-24T12:31:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-mon[41500]: 5.1 scrub starts 2024-03-24T12:31:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-mon[41500]: 5.1 scrub ok 2024-03-24T12:31:07.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-mon[29919]: 6.4 scrub starts 2024-03-24T12:31:07.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-mon[29919]: 6.4 scrub ok 2024-03-24T12:31:07.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-mon[29919]: pgmap v186: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 581 op/s; 11725/161010 objects misplaced (7.282%); 7.4 MiB/s, 117 objects/s recovering 2024-03-24T12:31:07.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-mon[29919]: 5.1 scrub starts 2024-03-24T12:31:07.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-mon[29919]: 5.1 scrub ok 2024-03-24T12:31:07.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:07.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:07.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:07.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:07.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:07.625 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:07 smithi190 ceph-mon[33748]: 6.4 scrub starts 2024-03-24T12:31:07.625 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:07 smithi190 ceph-mon[33748]: 6.4 scrub ok 2024-03-24T12:31:07.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:07 smithi190 ceph-mon[33748]: pgmap v186: 158 pgs: 1 active+remapped+backfill_wait+backfill_toofull, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 581 op/s; 11725/161010 objects misplaced (7.282%); 7.4 MiB/s, 117 objects/s recovering 2024-03-24T12:31:07.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:07 smithi190 ceph-mon[33748]: 5.1 scrub starts 2024-03-24T12:31:07.626 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:07 smithi190 ceph-mon[33748]: 5.1 scrub ok 2024-03-24T12:31:07.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:07.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:07.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:07.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:07.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:07.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:07.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:31:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:31:07.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:07.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:07.913 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:07.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:07.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:31:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:31:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:31:08.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:08.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:08.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:08.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:08.200 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:08.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:08.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:08.341 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:08.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:08.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:08.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:08 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:31:08.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:08 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:31:08.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:08.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:08 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:31:08.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:08 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:31:08.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:08.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:08 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:31:08.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:08 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:31:08.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:08.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:08.467 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:08.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:08.468 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:08.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:08.593 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:08.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:08.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:08.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:08.737 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:08.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:08.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:08.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:08.879 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:08.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:08.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:08.980 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:09.004 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:09.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:09.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:09.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:09.130 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:09.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:09.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:09.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:09.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:09.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:09.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:09.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:09.390 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:09.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:09.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[41500]: 4.7 deep-scrub starts 2024-03-24T12:31:09.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[41500]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T12:31:09.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[41500]: 4.7 deep-scrub ok 2024-03-24T12:31:09.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[41500]: pgmap v188: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s; 11510/154584 objects misplaced (7.446%); 5.6 MiB/s, 89 objects/s recovering 2024-03-24T12:31:09.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[41500]: 4.1a scrub starts 2024-03-24T12:31:09.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[41500]: 4.1a scrub ok 2024-03-24T12:31:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[29919]: 4.7 deep-scrub starts 2024-03-24T12:31:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[29919]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T12:31:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[29919]: 4.7 deep-scrub ok 2024-03-24T12:31:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[29919]: pgmap v188: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s; 11510/154584 objects misplaced (7.446%); 5.6 MiB/s, 89 objects/s recovering 2024-03-24T12:31:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[29919]: 4.1a scrub starts 2024-03-24T12:31:09.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:09 smithi029 ceph-mon[29919]: 4.1a scrub ok 2024-03-24T12:31:09.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:09.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:09.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:09 smithi190 ceph-mon[33748]: 4.7 deep-scrub starts 2024-03-24T12:31:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:09 smithi190 ceph-mon[33748]: osdmap e137: 8 total, 8 up, 8 in 2024-03-24T12:31:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:09 smithi190 ceph-mon[33748]: 4.7 deep-scrub ok 2024-03-24T12:31:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:09 smithi190 ceph-mon[33748]: pgmap v188: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 4 active+recovering+undersized+remapped, 152 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 605 op/s; 11510/154584 objects misplaced (7.446%); 5.6 MiB/s, 89 objects/s recovering 2024-03-24T12:31:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:09 smithi190 ceph-mon[33748]: 4.1a scrub starts 2024-03-24T12:31:09.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:09 smithi190 ceph-mon[33748]: 4.1a scrub ok 2024-03-24T12:31:09.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:09.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:09.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:09.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:09.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:09.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:09.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:09.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:09.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:09.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:10.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:10.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:10.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:10.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:10.194 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:10.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:10.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:10.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:10.341 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:10.341 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:10.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:10.465 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:10.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:10.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:10.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:10 smithi190 ceph-mon[33748]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:31:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:10 smithi190 ceph-mon[33748]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T12:31:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:10 smithi190 ceph-mon[33748]: 4.d scrub starts 2024-03-24T12:31:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:10 smithi190 ceph-mon[33748]: 4.d scrub ok 2024-03-24T12:31:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:10 smithi190 ceph-mon[33748]: 5.12 deep-scrub starts 2024-03-24T12:31:10.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:10 smithi190 ceph-mon[33748]: 5.12 deep-scrub ok 2024-03-24T12:31:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[41500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:31:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[41500]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T12:31:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[41500]: 4.d scrub starts 2024-03-24T12:31:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[41500]: 4.d scrub ok 2024-03-24T12:31:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[41500]: 5.12 deep-scrub starts 2024-03-24T12:31:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[41500]: 5.12 deep-scrub ok 2024-03-24T12:31:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[29919]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:31:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[29919]: osdmap e138: 8 total, 8 up, 8 in 2024-03-24T12:31:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[29919]: 4.d scrub starts 2024-03-24T12:31:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[29919]: 4.d scrub ok 2024-03-24T12:31:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[29919]: 5.12 deep-scrub starts 2024-03-24T12:31:10.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:10 smithi029 ceph-mon[29919]: 5.12 deep-scrub ok 2024-03-24T12:31:10.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:10.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:10.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:10.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:10.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:10.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:10.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:10.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:10.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:11.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:11.006 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:11.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:11.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:11.132 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:11.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:11.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:11.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:11.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:11.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:11.403 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:11.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:11.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:11.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:11.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:11.551 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:11.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:11 smithi190 ceph-mon[33748]: pgmap v190: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 634 op/s; 10192/152202 objects misplaced (6.696%); 8.3 MiB/s, 132 objects/s recovering 2024-03-24T12:31:11.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:11.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:11 smithi029 ceph-mon[41500]: pgmap v190: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 634 op/s; 10192/152202 objects misplaced (6.696%); 8.3 MiB/s, 132 objects/s recovering 2024-03-24T12:31:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:11 smithi029 ceph-mon[29919]: pgmap v190: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 634 op/s; 10192/152202 objects misplaced (6.696%); 8.3 MiB/s, 132 objects/s recovering 2024-03-24T12:31:11.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:11.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:11.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:11.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:11.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:11.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:11.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:11.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:11.955 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:11.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:12.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:12.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:12.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:12.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:12.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:12.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:12.365 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:12.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:12.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:12.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:12.496 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:12.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:12.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:12.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:12.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:12 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-03-24T12:31:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:12 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:31:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:12 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:31:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:12 smithi190 ceph-mon[33748]: 4.2 deep-scrub starts 2024-03-24T12:31:12.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:12 smithi190 ceph-mon[33748]: 4.2 deep-scrub ok 2024-03-24T12:31:12.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:12 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-03-24T12:31:12.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:12 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:31:12.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:12 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:31:12.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:12 smithi029 ceph-mon[29919]: 4.2 deep-scrub starts 2024-03-24T12:31:12.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:12 smithi029 ceph-mon[29919]: 4.2 deep-scrub ok 2024-03-24T12:31:12.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:12 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-03-24T12:31:12.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:12 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:31:12.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:12 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:31:12.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:12 smithi029 ceph-mon[41500]: 4.2 deep-scrub starts 2024-03-24T12:31:12.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:12 smithi029 ceph-mon[41500]: 4.2 deep-scrub ok 2024-03-24T12:31:12.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:12.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:12.761 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:12.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:12.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:12.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:12.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:12.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:12.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:13.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:13.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:13.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:13.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:13.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:13.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:13.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:13.277 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:13.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:13.278 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:13.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:13.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:13.421 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:13.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:13.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:13.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:13.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:13.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:13 smithi190 ceph-mon[33748]: pgmap v191: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 601 op/s; 10192/149142 objects misplaced (6.834%); 6.9 MiB/s, 111 objects/s recovering 2024-03-24T12:31:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:13 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:31:13.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:13 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:31:13.657 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:13.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:13 smithi029 ceph-mon[29919]: pgmap v191: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 601 op/s; 10192/149142 objects misplaced (6.834%); 6.9 MiB/s, 111 objects/s recovering 2024-03-24T12:31:13.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:13 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:31:13.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:13 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:31:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:13 smithi029 ceph-mon[41500]: pgmap v191: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 601 op/s; 10192/149142 objects misplaced (6.834%); 6.9 MiB/s, 111 objects/s recovering 2024-03-24T12:31:13.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:13 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:31:13.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:13 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:31:13.698 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:13.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:13.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:13.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:13.844 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:13.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:13.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:13.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:13.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:13.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:13.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:14.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:14.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:14.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:14.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:14.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:14.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:14.241 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:14.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:14.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:14.373 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:14.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:14.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:14.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:14.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:14.521 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:14.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:14.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:14.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:14 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:31:14.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:14 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:31:14.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:14.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:14.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:14.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:14 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:31:14.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:14 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:31:14.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:14 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:31:14.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:14 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:31:14.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:14.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:14.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:14.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:14.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:14.927 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:14.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:15.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:15.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:15.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:15.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:15.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:15.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:15.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:15.303 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:15.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:15.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:15.443 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:15.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:15.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:15.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:15.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:15.583 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:15.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:15 smithi190 ceph-mon[33748]: pgmap v192: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 152 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 639 op/s; 9089/145689 objects misplaced (6.239%); 12 MiB/s, 190 objects/s recovering 2024-03-24T12:31:15.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:15 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:31:15.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:15 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:31:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:15 smithi029 ceph-mon[41500]: pgmap v192: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 152 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 639 op/s; 9089/145689 objects misplaced (6.239%); 12 MiB/s, 190 objects/s recovering 2024-03-24T12:31:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:15 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:31:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:15 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:31:15.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:15 smithi029 ceph-mon[29919]: pgmap v192: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 152 active+clean; 3.0 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 639 op/s; 9089/145689 objects misplaced (6.239%); 12 MiB/s, 190 objects/s recovering 2024-03-24T12:31:15.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:15 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:31:15.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:15 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:31:15.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:15.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:15.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:15.858 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:15.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:15.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:15.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:15.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:16.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:16.135 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:16.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:16.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:16.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:16.278 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:16.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:16.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:16.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:16.403 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:16.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:16.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:16.530 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:16.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:16.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:16.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:16.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:16.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:16.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:16.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:16.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:16.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:16.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:17.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:17.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:17.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:17.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:17.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:17.181 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:17.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:17.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:17.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:17.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:17.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:17.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:17.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:17.609 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:17.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:17.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:17.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:17.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:17.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:17.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:17.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:17.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:17.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:17 smithi190 ceph-mon[33748]: pgmap v193: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 617 B/s rd, 0 B/s wr, 598 op/s; 7927/142623 objects misplaced (5.558%); 14 MiB/s, 230 objects/s recovering 2024-03-24T12:31:17.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:17.884 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:31:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:31:17.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:17 smithi029 ceph-mon[29919]: pgmap v193: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 617 B/s rd, 0 B/s wr, 598 op/s; 7927/142623 objects misplaced (5.558%); 14 MiB/s, 230 objects/s recovering 2024-03-24T12:31:17.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:17 smithi029 ceph-mon[41500]: pgmap v193: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 2.9 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 617 B/s rd, 0 B/s wr, 598 op/s; 7927/142623 objects misplaced (5.558%); 14 MiB/s, 230 objects/s recovering 2024-03-24T12:31:17.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:17.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:31:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:31:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:31:17.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:18.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:18.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:18.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:18.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:18.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:18.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:18.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:18.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:18.275 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:18.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:18.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:18.416 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:18.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:18.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:18.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:18.565 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:18.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:18.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:18.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:18.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:18.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:18.827 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:18.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:18 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:31:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:18 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:31:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:18 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:31:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:18 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:31:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:18 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:31:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:18 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:31:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:18 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:31:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:18 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:31:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:18 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:31:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:18 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:31:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:18 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:31:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:18 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:31:18.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:18.953 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:18.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:19.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:19.082 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:19.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:19.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:19.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:19.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:19.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:19.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:19.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:19.338 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:19.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:19.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:19.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T12:31:19.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:19.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:19.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:19.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:19.568 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:19.590 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:19.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:19.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:19.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:19 smithi029 ceph-mon[29919]: pgmap v194: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 152 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 665 op/s; 7314/134649 objects misplaced (5.432%); 15 MiB/s, 237 objects/s recovering 2024-03-24T12:31:19.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:19 smithi029 ceph-mon[41500]: pgmap v194: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 152 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 665 op/s; 7314/134649 objects misplaced (5.432%); 15 MiB/s, 237 objects/s recovering 2024-03-24T12:31:19.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:19.721 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:19.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:19.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:19.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:19.870 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:19.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:19.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:19.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:19 smithi190 ceph-mon[33748]: pgmap v194: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 152 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 665 op/s; 7314/134649 objects misplaced (5.432%); 15 MiB/s, 237 objects/s recovering 2024-03-24T12:31:19.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:19 smithi190 ceph-mon[33748]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T12:31:19.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:19.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:20.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:20.120 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:20.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:20.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:20.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:19 smithi029 ceph-mon[29919]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T12:31:20.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:19 smithi029 ceph-mon[41500]: osdmap e139: 8 total, 8 up, 8 in 2024-03-24T12:31:20.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:20.268 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:20.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:20.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:20.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:20.402 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:20.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:20.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:20.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:20.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:20.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:20.656 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:20.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:20.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:20.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:20.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:20.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:20.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:20.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:20 smithi029 ceph-mon[29919]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T12:31:20.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:20 smithi029 ceph-mon[29919]: 3.14 scrub starts 2024-03-24T12:31:20.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:20 smithi029 ceph-mon[29919]: 3.14 scrub ok 2024-03-24T12:31:20.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:20 smithi029 ceph-mon[29919]: 6.1d scrub starts 2024-03-24T12:31:20.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:20 smithi029 ceph-mon[29919]: 6.1d scrub ok 2024-03-24T12:31:20.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:20 smithi029 ceph-mon[41500]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T12:31:20.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:20 smithi029 ceph-mon[41500]: 3.14 scrub starts 2024-03-24T12:31:20.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:20 smithi029 ceph-mon[41500]: 3.14 scrub ok 2024-03-24T12:31:20.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:20 smithi029 ceph-mon[41500]: 6.1d scrub starts 2024-03-24T12:31:20.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:20 smithi029 ceph-mon[41500]: 6.1d scrub ok 2024-03-24T12:31:20.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:20.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:20.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:21.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:21.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:21.059 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:21.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:21.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:20 smithi190 ceph-mon[33748]: osdmap e140: 8 total, 8 up, 8 in 2024-03-24T12:31:21.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:20 smithi190 ceph-mon[33748]: 3.14 scrub starts 2024-03-24T12:31:21.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:20 smithi190 ceph-mon[33748]: 3.14 scrub ok 2024-03-24T12:31:21.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:20 smithi190 ceph-mon[33748]: 6.1d scrub starts 2024-03-24T12:31:21.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:20 smithi190 ceph-mon[33748]: 6.1d scrub ok 2024-03-24T12:31:21.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:21.207 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:21.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:21.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:21.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:21.350 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:21.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:21.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:21.437 INFO:tasks.thrashosds.thrasher:j is {'epoch': 118, 'fsid': 'bc017092-e9d8-11ee-95cd-87774f69a715', 'created': '2024-03-24T12:20:18.712998+0000', 'modified': '2024-03-24T12:29:30.941018+0000', 'last_up_change': '2024-03-24T12:25:48.676565+0000', 'last_in_change': '2024-03-24T12:25:30.369664+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T12:23:41.491597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-24T12:26:01.465080+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T12:26:02.100021+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T12:26:03.473862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T12:26:05.573828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T12:26:07.645083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-24T12:28:04.290397+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '109', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4600000381469727, 'score_stable': 1.850000023841858, 'optimal_score': 1, 'raw_score_acting': 2.4600000381469727, 'raw_score_stable': 1.850000023841858, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0aa1c49b-7887-4f74-b03c-08cc0adec90b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6801', 'nonce': 2514818464}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6802', 'nonce': 2514818464}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6804', 'nonce': 2514818464}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6803', 'nonce': 2514818464}]}, 'public_addr': '172.21.15.29:6801/2514818464', 'cluster_addr': '172.21.15.29:6802/2514818464', 'heartbeat_back_addr': '172.21.15.29:6804/2514818464', 'heartbeat_front_addr': '172.21.15.29:6803/2514818464', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99d8ef8f-eb59-4882-ab51-b398e6b53bcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6805', 'nonce': 2923396572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6806', 'nonce': 2923396572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6808', 'nonce': 2923396572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6807', 'nonce': 2923396572}]}, 'public_addr': '172.21.15.29:6805/2923396572', 'cluster_addr': '172.21.15.29:6806/2923396572', 'heartbeat_back_addr': '172.21.15.29:6808/2923396572', 'heartbeat_front_addr': '172.21.15.29:6807/2923396572', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c3f83505-25ee-4217-b60a-081c36092e92', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6809', 'nonce': 126032915}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6810', 'nonce': 126032915}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6812', 'nonce': 126032915}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6811', 'nonce': 126032915}]}, 'public_addr': '172.21.15.29:6809/126032915', 'cluster_addr': '172.21.15.29:6810/126032915', 'heartbeat_back_addr': '172.21.15.29:6812/126032915', 'heartbeat_front_addr': '172.21.15.29:6811/126032915', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f5e479a8-459a-4315-a808-711a7b84bb96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6813', 'nonce': 1080171824}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6814', 'nonce': 1080171824}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6816', 'nonce': 1080171824}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6815', 'nonce': 1080171824}]}, 'public_addr': '172.21.15.29:6813/1080171824', 'cluster_addr': '172.21.15.29:6814/1080171824', 'heartbeat_back_addr': '172.21.15.29:6816/1080171824', 'heartbeat_front_addr': '172.21.15.29:6815/1080171824', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '5e420304-2798-4d11-9688-d0d89482008c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 3690258869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6801', 'nonce': 3690258869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6803', 'nonce': 3690258869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 3690258869}]}, 'public_addr': '172.21.15.190:6800/3690258869', 'cluster_addr': '172.21.15.190:6801/3690258869', 'heartbeat_back_addr': '172.21.15.190:6803/3690258869', 'heartbeat_front_addr': '172.21.15.190:6802/3690258869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1213431a-652c-4cbf-b32a-36335430c666', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 109, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 1327220488}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6805', 'nonce': 1327220488}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6807', 'nonce': 1327220488}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 1327220488}]}, 'public_addr': '172.21.15.190:6804/1327220488', 'cluster_addr': '172.21.15.190:6805/1327220488', 'heartbeat_back_addr': '172.21.15.190:6807/1327220488', 'heartbeat_front_addr': '172.21.15.190:6806/1327220488', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0712e511-0257-44f3-a82f-ee09ab957c0c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 1295064976}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6809', 'nonce': 1295064976}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6811', 'nonce': 1295064976}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 1295064976}]}, 'public_addr': '172.21.15.190:6808/1295064976', 'cluster_addr': '172.21.15.190:6809/1295064976', 'heartbeat_back_addr': '172.21.15.190:6811/1295064976', 'heartbeat_front_addr': '172.21.15.190:6810/1295064976', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7e36bde4-14d0-4e7e-9a82-2bb97859dbda', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 112, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 110963949}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6813', 'nonce': 110963949}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6815', 'nonce': 110963949}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 110963949}]}, 'public_addr': '172.21.15.190:6812/110963949', 'cluster_addr': '172.21.15.190:6813/110963949', 'heartbeat_back_addr': '172.21.15.190:6815/110963949', 'heartbeat_front_addr': '172.21.15.190:6814/110963949', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:28:57.006151+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:29:28.213452+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:28:42.762815+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:28:45.010123+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:29:30.861346+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:28:54.639476+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:29:21.054689+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:29:27.828822+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '7.10', 'osds': [3, 1]}, {'pgid': '7.11', 'osds': [3, 6, 4]}, {'pgid': '7.12', 'osds': [1, 2, 3]}, {'pgid': '7.13', 'osds': [2, 3, 4]}, {'pgid': '7.14', 'osds': [0, 2]}, {'pgid': '7.15', 'osds': [7, 2, 0]}, {'pgid': '7.16', 'osds': [0, 7]}, {'pgid': '7.17', 'osds': [0, 2]}, {'pgid': '7.18', 'osds': [2, 7]}, {'pgid': '7.19', 'osds': [0, 2]}], 'primary_temp': [], 'blocklist': {'172.21.15.29:0/2633022206': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2612078930': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2365028933': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4113609697': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4130171883': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/3697529912': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2929430696': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/327626505': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/3715448582': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/861146182': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2109454792': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/1631806715': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:6800/852212806': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/1823252077': '2024-03-25T12:21:15.151901+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:31:21.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:31:21.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:21.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:21.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:21.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:21.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:21.629 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:21.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:21.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:21.787 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:21.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:21.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:21.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:21.959 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:21.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:22.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:22.105 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:22.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:22.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:22.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:22.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:22.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:22.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:22.361 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:22.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:21 smithi190 ceph-mon[33748]: pgmap v197: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 725 op/s; 5256/131739 objects misplaced (3.990%); 20 MiB/s, 320 objects/s recovering 2024-03-24T12:31:22.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:21 smithi029 ceph-mon[41500]: pgmap v197: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 725 op/s; 5256/131739 objects misplaced (3.990%); 20 MiB/s, 320 objects/s recovering 2024-03-24T12:31:22.422 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:21 smithi029 ceph-mon[29919]: pgmap v197: 158 pgs: 1 active+clean+remapped, 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 152 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 725 op/s; 5256/131739 objects misplaced (3.990%); 20 MiB/s, 320 objects/s recovering 2024-03-24T12:31:22.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:22.503 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:22.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:22.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:22.647 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:22.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:22.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:22.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:22.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:22.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:22.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:22.904 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:22.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:22.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:23.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:23.038 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:23.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:23.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:23.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:23.172 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:23.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:23.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:23.305 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:23.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:22 smithi190 ceph-mon[33748]: 5.e deep-scrub starts 2024-03-24T12:31:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:22 smithi190 ceph-mon[33748]: 5.e deep-scrub ok 2024-03-24T12:31:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:22 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:31:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:22 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:31:23.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:22 smithi029 ceph-mon[29919]: 5.e deep-scrub starts 2024-03-24T12:31:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:22 smithi029 ceph-mon[29919]: 5.e deep-scrub ok 2024-03-24T12:31:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:22 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:31:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:22 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:31:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:22 smithi029 ceph-mon[41500]: 5.e deep-scrub starts 2024-03-24T12:31:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:22 smithi029 ceph-mon[41500]: 5.e deep-scrub ok 2024-03-24T12:31:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:22 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:31:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:22 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:31:23.452 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:23.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:23.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:23.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:23.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:23.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:23.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:23.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:23.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:23.845 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:23.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:23.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:23.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:23.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:23.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:23.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:24.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:24.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:24.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:24.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:24.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:24.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:24.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:24.356 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:23 smithi190 ceph-mon[33748]: pgmap v198: 158 pgs: 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 153 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 726 op/s; 5256/128277 objects misplaced (4.097%); 12 MiB/s, 198 objects/s recovering 2024-03-24T12:31:24.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:24.402 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:24.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:23 smithi029 ceph-mon[29919]: pgmap v198: 158 pgs: 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 153 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 726 op/s; 5256/128277 objects misplaced (4.097%); 12 MiB/s, 198 objects/s recovering 2024-03-24T12:31:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:23 smithi029 ceph-mon[41500]: pgmap v198: 158 pgs: 1 active+recovery_wait, 1 active+remapped+backfill_toofull, 3 active+recovering+undersized+remapped, 153 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 726 op/s; 5256/128277 objects misplaced (4.097%); 12 MiB/s, 198 objects/s recovering 2024-03-24T12:31:24.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:24.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:24.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:24.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:24.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:24.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:24.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:24.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:24.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:24.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:24.892 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:24.922 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:24.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:24.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:25.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:25.068 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:25.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:25.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:25.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:25.202 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:25.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:25.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:25.344 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:25.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:24 smithi190 ceph-mon[33748]: 6.1e scrub starts 2024-03-24T12:31:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:24 smithi190 ceph-mon[33748]: 6.1e scrub ok 2024-03-24T12:31:25.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:24 smithi029 ceph-mon[41500]: 6.1e scrub starts 2024-03-24T12:31:25.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:25 smithi029 ceph-mon[41500]: 6.1e scrub ok 2024-03-24T12:31:25.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:24 smithi029 ceph-mon[29919]: 6.1e scrub starts 2024-03-24T12:31:25.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:25 smithi029 ceph-mon[29919]: 6.1e scrub ok 2024-03-24T12:31:25.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:25.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:25.492 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:25.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:25.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:25.636 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:25.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:25.738 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:25.767 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:25.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:25.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:25.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:25.899 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:25.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:25.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:26.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:26.061 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:26.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:26.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:26.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:26.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:26.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:26.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:26.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:26.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:25 smithi190 ceph-mon[33748]: pgmap v199: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 153 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 736 op/s; 5058/124977 objects misplaced (4.047%); 9.8 MiB/s, 157 objects/s recovering 2024-03-24T12:31:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:25 smithi190 ceph-mon[33748]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:31:26.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:25 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:31:26.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:25 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:31:26.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:25 smithi029 ceph-mon[41500]: pgmap v199: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 153 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 736 op/s; 5058/124977 objects misplaced (4.047%); 9.8 MiB/s, 157 objects/s recovering 2024-03-24T12:31:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:25 smithi029 ceph-mon[41500]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:31:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:25 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:31:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:25 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:31:26.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:25 smithi029 ceph-mon[29919]: pgmap v199: 158 pgs: 1 active+remapped+backfilling, 1 active+recovery_wait, 3 active+recovering+undersized+remapped, 153 active+clean; 2.5 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 736 op/s; 5058/124977 objects misplaced (4.047%); 9.8 MiB/s, 157 objects/s recovering 2024-03-24T12:31:26.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:25 smithi029 ceph-mon[29919]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:31:26.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:25 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:31:26.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:25 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:31:26.451 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:26.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:26.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:26.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:26.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:26.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:26.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:26.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:26.704 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:26.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:26.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:26.833 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:26.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:26.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:26.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:27.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:27.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:27.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:27.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:27.244 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:27.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:27.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:27 smithi190 ceph-mon[33748]: 4.9 deep-scrub starts 2024-03-24T12:31:27.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:27 smithi190 ceph-mon[33748]: 4.9 deep-scrub ok 2024-03-24T12:31:27.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:27 smithi190 ceph-mon[33748]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T12:31:27.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:27 smithi190 ceph-mon[33748]: pgmap v201: 158 pgs: 1 active+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 154 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 539 op/s; 3615/121809 objects misplaced (2.968%); 9.7 MiB/s, 154 objects/s recovering 2024-03-24T12:31:27.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:27.369 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:27.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:27.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:27 smithi029 ceph-mon[29919]: 4.9 deep-scrub starts 2024-03-24T12:31:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:27 smithi029 ceph-mon[29919]: 4.9 deep-scrub ok 2024-03-24T12:31:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:27 smithi029 ceph-mon[29919]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T12:31:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:27 smithi029 ceph-mon[29919]: pgmap v201: 158 pgs: 1 active+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 154 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 539 op/s; 3615/121809 objects misplaced (2.968%); 9.7 MiB/s, 154 objects/s recovering 2024-03-24T12:31:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:27 smithi029 ceph-mon[41500]: 4.9 deep-scrub starts 2024-03-24T12:31:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:27 smithi029 ceph-mon[41500]: 4.9 deep-scrub ok 2024-03-24T12:31:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:27 smithi029 ceph-mon[41500]: osdmap e141: 8 total, 8 up, 8 in 2024-03-24T12:31:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:27 smithi029 ceph-mon[41500]: pgmap v201: 158 pgs: 1 active+undersized+remapped, 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 154 active+clean; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 539 op/s; 3615/121809 objects misplaced (2.968%); 9.7 MiB/s, 154 objects/s recovering 2024-03-24T12:31:27.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:27.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:27.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:27.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:27.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:27.621 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:27.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:27.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:27.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:27.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:27.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:27.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:27.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:27.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:27.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:31:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:31:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:31:27.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:28.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:28.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:28.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:28.027 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:31:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:31:28.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:28.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:28.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:28.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:28.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:28.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:28.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:28.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:28 smithi190 ceph-mon[33748]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T12:31:28.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:28.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:28.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:28.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:28.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:28 smithi029 ceph-mon[41500]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T12:31:28.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:28.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:28 smithi029 ceph-mon[29919]: osdmap e142: 8 total, 8 up, 8 in 2024-03-24T12:31:28.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:28.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:28.521 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:28.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:28.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:28.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:28.651 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:28.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:28.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:28.779 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:28.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:28.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:28.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:28.904 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:28.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:28.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:29.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:29.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:29.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:29.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:29.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:29.155 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:29.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:29.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:29 smithi029 ceph-mon[29919]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T12:31:29.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:29 smithi029 ceph-mon[41500]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T12:31:29.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:29.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:29.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:29.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:29 smithi190 ceph-mon[33748]: osdmap e143: 8 total, 8 up, 8 in 2024-03-24T12:31:29.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:29 smithi190 ceph-mon[33748]: pgmap v204: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 155 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 673 op/s; 3615/116160 objects misplaced (3.112%); 5.6 MiB/s, 90 objects/s recovering 2024-03-24T12:31:29.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:29.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:29.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:29.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:29.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:29.555 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:29.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:29.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:29.657 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:29.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:29 smithi029 ceph-mon[41500]: pgmap v204: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 155 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 673 op/s; 3615/116160 objects misplaced (3.112%); 5.6 MiB/s, 90 objects/s recovering 2024-03-24T12:31:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:29 smithi029 ceph-mon[29919]: pgmap v204: 158 pgs: 1 active+remapped+backfilling, 2 active+recovering+undersized+remapped, 155 active+clean; 2.4 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 673 op/s; 3615/116160 objects misplaced (3.112%); 5.6 MiB/s, 90 objects/s recovering 2024-03-24T12:31:29.680 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:29.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:29.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:29.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:29.816 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:29.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:29.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:29.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:29.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:29.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:29.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:30.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:30.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:30.086 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:30.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:30.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:30.211 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:30.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:30.212 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:30.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:30.336 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:30.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:30.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:30 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:31:30.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:30 smithi190 ceph-mon[33748]: 4.11 scrub ok 2024-03-24T12:31:30.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:30 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-03-24T12:31:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:30 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1063844259' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:31:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:30 smithi190 ceph-mon[33748]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T12:31:30.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:30 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:31:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:31:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[41500]: 4.11 scrub ok 2024-03-24T12:31:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-03-24T12:31:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1063844259' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:31:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[41500]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T12:31:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:31:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:31:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[29919]: 4.11 scrub ok 2024-03-24T12:31:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 2 pgs undersized (PG_DEGRADED) 2024-03-24T12:31:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1063844259' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:31:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[29919]: osdmap e144: 8 total, 8 up, 8 in 2024-03-24T12:31:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:30 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:31:30.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:30.474 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:30.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:30.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:30.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:30.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:30.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:30.731 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:30.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:30.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:30.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:30.860 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:30.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:30.961 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:30.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:30.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:30.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:31.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:31.121 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:31.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:31.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:31.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:31.263 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:31.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:31.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:31 smithi190 ceph-mon[33748]: 6.13 deep-scrub starts 2024-03-24T12:31:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:31 smithi190 ceph-mon[33748]: 6.13 deep-scrub ok 2024-03-24T12:31:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:31 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:31:31.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:31 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:31:31.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:31 smithi190 ceph-mon[33748]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T12:31:31.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:31 smithi190 ceph-mon[33748]: pgmap v207: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 743 op/s; 1628/112905 objects misplaced (1.442%); 12 MiB/s, 190 objects/s recovering 2024-03-24T12:31:31.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:31.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:31.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[41500]: 6.13 deep-scrub starts 2024-03-24T12:31:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[41500]: 6.13 deep-scrub ok 2024-03-24T12:31:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:31:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:31:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[41500]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T12:31:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[41500]: pgmap v207: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 743 op/s; 1628/112905 objects misplaced (1.442%); 12 MiB/s, 190 objects/s recovering 2024-03-24T12:31:31.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[29919]: 6.13 deep-scrub starts 2024-03-24T12:31:31.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[29919]: 6.13 deep-scrub ok 2024-03-24T12:31:31.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:31:31.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:31:31.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[29919]: osdmap e145: 8 total, 8 up, 8 in 2024-03-24T12:31:31.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:31 smithi029 ceph-mon[29919]: pgmap v207: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.3 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 743 op/s; 1628/112905 objects misplaced (1.442%); 12 MiB/s, 190 objects/s recovering 2024-03-24T12:31:31.434 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:31:31.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:31.538 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:31.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:31.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:31.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:31.680 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:31.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:31.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:31.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:31.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:31.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:31.952 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:31.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:32.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:32.091 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:32.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:32.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:32.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:32.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:32.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:32.360 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:32.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:32.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:32.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:32.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:32.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:32.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:32.632 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:32.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:32.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:32.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:32.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:31:32.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:32.899 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:32.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:32.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:33.001 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:33.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:33.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:33.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:33.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:32 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:31:33.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:32 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:33.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:33.168 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:33.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:32 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:31:33.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:32 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:33.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:32 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:31:33.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:32 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:33.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:33.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:33.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:33.303 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:33.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:33.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:33.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:33.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:33.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:33.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:33.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:33.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:33.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:33.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:33.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:33.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:33.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:34.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:34.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:34.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:34.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:34.114 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:33 smithi029 ceph-mon[41500]: pgmap v208: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.2 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 761 op/s; 1628/109395 objects misplaced (1.488%); 8.7 MiB/s, 139 objects/s recovering 2024-03-24T12:31:34.115 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:33 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:31:34.115 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:33 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:31:34.115 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:33 smithi029 ceph-mon[41500]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T12:31:34.115 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:33 smithi029 ceph-mon[29919]: pgmap v208: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.2 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 761 op/s; 1628/109395 objects misplaced (1.488%); 8.7 MiB/s, 139 objects/s recovering 2024-03-24T12:31:34.115 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:33 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:31:34.115 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:33 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:31:34.115 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:33 smithi029 ceph-mon[29919]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T12:31:34.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:33 smithi190 ceph-mon[33748]: pgmap v208: 158 pgs: 2 active+recovering+undersized+remapped, 156 active+clean; 2.2 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 761 op/s; 1628/109395 objects misplaced (1.488%); 8.7 MiB/s, 139 objects/s recovering 2024-03-24T12:31:34.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:33 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:31:34.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:33 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:31:34.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:33 smithi190 ceph-mon[33748]: osdmap e146: 8 total, 8 up, 8 in 2024-03-24T12:31:34.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:34.160 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:34.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:34.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:34.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:34.315 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:34.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:34.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:34.462 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:34.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:34.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:34.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:34.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:34.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:34.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:34.739 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:34.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:34.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:34.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:34.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:34.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:34.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:35.002 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:35.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:31:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:31:35.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:35.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:34 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:31:35.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:34 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:31:35.155 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:35.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:35.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:35.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:34 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:31:35.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:34 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:31:35.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:34 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:31:35.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:34 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:31:35.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:35.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:35.306 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:35.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:35.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:35.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:35.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:35.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:35.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:31:35.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:35.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:35.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:35.679 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:35.743 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:35.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:35.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:35.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:35.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:35 smithi029 ceph-mon[29919]: pgmap v210: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 592 op/s; 986/105513 objects misplaced (0.934%); 10 MiB/s, 162 objects/s recovering 2024-03-24T12:31:35.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:35 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-24T12:31:35.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:35 smithi029 ceph-mon[29919]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T12:31:35.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:35.913 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:35.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:36.014 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:36.059 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:36.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:36.060 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:35 smithi190 ceph-mon[33748]: pgmap v210: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 592 op/s; 986/105513 objects misplaced (0.934%); 10 MiB/s, 162 objects/s recovering 2024-03-24T12:31:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:35 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-24T12:31:36.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:35 smithi190 ceph-mon[33748]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T12:31:36.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:35 smithi029 ceph-mon[41500]: pgmap v210: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.1 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 592 op/s; 986/105513 objects misplaced (0.934%); 10 MiB/s, 162 objects/s recovering 2024-03-24T12:31:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:35 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-24T12:31:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:35 smithi029 ceph-mon[41500]: osdmap e147: 8 total, 8 up, 8 in 2024-03-24T12:31:36.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:36.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:36.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:36.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:36.316 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:36.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:36.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:36.447 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:36.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:36.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:36.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:36.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:36.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:36.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:36.712 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:36.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:36.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:36.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:36.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:36.853 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:36.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:36.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:36.995 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:36.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:37.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:37.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:37.129 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:37.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:37.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:37.254 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:37.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:37.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:37.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:37.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:37.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:37.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:37.514 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:37.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:37.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:37.639 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:37.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:37.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:37.766 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:37.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:37.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:37.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:37.883 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:31:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:31:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:31:37.894 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:37.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:37.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:38.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:38.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:38.121 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:38.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:37 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:31:38.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:37 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:31:38.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:37 smithi190 ceph-mon[33748]: pgmap v212: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 709 op/s; 147/101433 objects misplaced (0.145%); 12 MiB/s, 199 objects/s recovering 2024-03-24T12:31:38.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:37 smithi190 ceph-mon[33748]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T12:31:38.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:31:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:31:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:38.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:38.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:37 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:31:38.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:37 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:31:38.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:37 smithi029 ceph-mon[41500]: pgmap v212: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 709 op/s; 147/101433 objects misplaced (0.145%); 12 MiB/s, 199 objects/s recovering 2024-03-24T12:31:38.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:37 smithi029 ceph-mon[41500]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T12:31:38.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:37 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:31:38.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:37 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:31:38.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:37 smithi029 ceph-mon[29919]: pgmap v212: 158 pgs: 1 active+recovering+undersized+remapped, 157 active+clean; 2.1 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 709 op/s; 147/101433 objects misplaced (0.145%); 12 MiB/s, 199 objects/s recovering 2024-03-24T12:31:38.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:37 smithi029 ceph-mon[29919]: osdmap e148: 8 total, 8 up, 8 in 2024-03-24T12:31:38.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:38.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:38.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:38.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:38.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:38.437 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:38.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:38.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:38.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:38.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:38.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:38.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:38.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:38.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:38.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:38.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:38.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:38.980 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:38.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:39.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:39.127 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:39.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:39.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:38 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:39.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:38 smithi029 ceph-mon[41500]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T12:31:39.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:38 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:39.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:38 smithi029 ceph-mon[29919]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T12:31:39.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:39.251 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:39.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:39.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:39.381 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:39.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:39.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:38 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:38 smithi190 ceph-mon[33748]: osdmap e149: 8 total, 8 up, 8 in 2024-03-24T12:31:39.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:39.517 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:39.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:39.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:39.659 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:39.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:39.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:39.785 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:39.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:39.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:39.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:39.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:39.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:40.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:40.054 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:40.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:40.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:40.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:39 smithi029 ceph-mon[41500]: pgmap v215: 158 pgs: 1 creating+peering, 157 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 911 B/s rd, 0 B/s wr, 963 op/s; 6.0 MiB/s, 96 objects/s recovering 2024-03-24T12:31:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:39 smithi029 ceph-mon[41500]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T12:31:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:39 smithi029 ceph-mon[41500]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-24T12:31:40.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:39 smithi029 ceph-mon[29919]: pgmap v215: 158 pgs: 1 creating+peering, 157 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 911 B/s rd, 0 B/s wr, 963 op/s; 6.0 MiB/s, 96 objects/s recovering 2024-03-24T12:31:40.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:39 smithi029 ceph-mon[29919]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T12:31:40.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:39 smithi029 ceph-mon[29919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-24T12:31:40.182 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:40.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:40.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:40.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:40.308 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:40.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:39 smithi190 ceph-mon[33748]: pgmap v215: 158 pgs: 1 creating+peering, 157 active+clean; 1.9 GiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 911 B/s rd, 0 B/s wr, 963 op/s; 6.0 MiB/s, 96 objects/s recovering 2024-03-24T12:31:40.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:39 smithi190 ceph-mon[33748]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T12:31:40.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:39 smithi190 ceph-mon[33748]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-24T12:31:40.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:40.444 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:40.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:40.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:40.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:40.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:40.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:40.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:40.700 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:40.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:40.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:40.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:40.831 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:40.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:40.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:40.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:40.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:40.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:41.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:41.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:41.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:41.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:41.240 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:41.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:41.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:41.374 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:41.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:41.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:41.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:41.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:41.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:41.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:41.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:41.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:41.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:41.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:41.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:41.904 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:41.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:41.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:42.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:42.035 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:42.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:42.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:42.166 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:42.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:42.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:41 smithi029 ceph-mon[29919]: pgmap v216: 158 pgs: 1 creating+peering, 157 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 872 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-03-24T12:31:42.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:41 smithi029 ceph-mon[41500]: pgmap v216: 158 pgs: 1 creating+peering, 157 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 872 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-03-24T12:31:42.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:42.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:42.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:42.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:41 smithi190 ceph-mon[33748]: pgmap v216: 158 pgs: 1 creating+peering, 157 active+clean; 1.8 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 872 op/s; 3.4 MiB/s, 54 objects/s recovering 2024-03-24T12:31:42.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:42.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:42.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:42.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:42.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:42.558 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:42.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:42.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:42.660 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:42.688 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:42.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:42.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:42.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:42.829 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:42.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:42.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:42.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:42.967 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:42.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:42.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:43.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:43.107 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:43.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:43.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:43.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:43.246 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:43.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:43.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:43.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:43.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:43.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:43.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:42 smithi190 ceph-mon[33748]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T12:31:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:42 smithi029 ceph-mon[41500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T12:31:43.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:42 smithi029 ceph-mon[29919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T12:31:43.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:43.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:43.523 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:43.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:43.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:43.656 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:43.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:43.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:43.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:43.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:43.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:43.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:43.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:43.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:44.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:44.080 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:44.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:44.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:44.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:44.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:44.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:44.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:44.343 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:44.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:44.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:44.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:43 smithi190 ceph-mon[33748]: pgmap v217: 158 pgs: 158 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 2.7 MiB/s, 43 objects/s recovering 2024-03-24T12:31:44.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:43 smithi029 ceph-mon[41500]: pgmap v217: 158 pgs: 158 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 2.7 MiB/s, 43 objects/s recovering 2024-03-24T12:31:44.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:43 smithi029 ceph-mon[29919]: pgmap v217: 158 pgs: 158 active+clean; 1.6 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.08k op/s; 2.7 MiB/s, 43 objects/s recovering 2024-03-24T12:31:44.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:44.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:44.478 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:44.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:44.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:44.610 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:44.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:44.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:44.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:44.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:44.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:44.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:44.890 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:44.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:44.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:45.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:45.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:45.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:45.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:45.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:45.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:45.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:45.289 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:45.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:45.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:45.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:45.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:45.430 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:45.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:45.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:45.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:45.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:45.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:45.704 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:45.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:45.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:45.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:45.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:45.845 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:45.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:45.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:45.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:45.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:46.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:46.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:46.120 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:46.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:46.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:46.248 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:46.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:46.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:46.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:46.376 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:46.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:46.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:46.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:45 smithi190 ceph-mon[33748]: pgmap v218: 158 pgs: 158 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 392 KiB/s, 6 objects/s recovering 2024-03-24T12:31:46.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:45 smithi029 ceph-mon[41500]: pgmap v218: 158 pgs: 158 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 392 KiB/s, 6 objects/s recovering 2024-03-24T12:31:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:45 smithi029 ceph-mon[29919]: pgmap v218: 158 pgs: 158 active+clean; 1.5 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s; 392 KiB/s, 6 objects/s recovering 2024-03-24T12:31:46.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:46.514 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:46.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:46.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:46.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:46.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:46.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:46.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:46.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:46.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:46.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:46.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:46.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:47.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:47.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:47.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:47.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:47.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:47.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:47.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:47.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:47.289 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:47.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:47.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:47.430 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:47.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:47.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:47.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:47.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:47.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:47.701 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:47.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:47.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:47.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:47.842 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:47.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:47.856 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:31:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:31:47.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:47.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:47.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:48.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:48.119 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:48.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:48.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:47 smithi190 ceph-mon[33748]: pgmap v219: 158 pgs: 158 active+clean; 1.4 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s; 327 KiB/s, 5 objects/s recovering 2024-03-24T12:31:48.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:31:48.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:48.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:47 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:31:48.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:47 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:31:48.135 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:31:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:31:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-mon[29919]: pgmap v219: 158 pgs: 158 active+clean; 1.4 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s; 327 KiB/s, 5 objects/s recovering 2024-03-24T12:31:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:31:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:31:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:31:48.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-mon[41500]: pgmap v219: 158 pgs: 158 active+clean; 1.4 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.03k op/s; 327 KiB/s, 5 objects/s recovering 2024-03-24T12:31:48.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:31:48.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:31:48.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:31:48.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:47 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:31:48.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:48.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:48.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:48.402 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:48.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:48.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:48.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:48.535 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:48.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:48.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:48.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:48.676 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:48.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:48.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:48.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:48.817 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:48.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:48.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:48.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:48.956 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:48.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:49.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:49.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:49.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:49.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:49.182 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:49.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:49.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:49.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:49.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:49.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:49.358 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:49.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:49.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:48 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:49.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:48 smithi190 ceph-mon[33748]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T12:31:49.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:48 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:49.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:48 smithi029 ceph-mon[41500]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T12:31:49.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:48 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:49.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:48 smithi029 ceph-mon[29919]: osdmap e150: 8 total, 8 up, 8 in 2024-03-24T12:31:49.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:49.484 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:49.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:49.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:49.611 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:49.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:49.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:49.735 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:49.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:49.838 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:49.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:49.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:49.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:50.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:50.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:50.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:50.158 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:50.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:50.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:50.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:49 smithi029 ceph-mon[29919]: pgmap v221: 158 pgs: 158 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T12:31:50.237 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:49 smithi029 ceph-mon[41500]: pgmap v221: 158 pgs: 158 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T12:31:50.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:50.309 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:50.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:50.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:49 smithi190 ceph-mon[33748]: pgmap v221: 158 pgs: 158 active+clean; 1.2 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T12:31:50.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:50.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:50.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:50.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:50.576 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:50.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:50.678 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:50.715 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:50.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:50.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:50.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:50.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:50.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:50.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:50.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:50.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:51.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:51.112 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:51.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:51.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:51.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:51.249 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:51.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:51.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:51.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:51.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:51.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:51.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:51.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:51.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:51.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:51.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:51.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:51.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:51.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:51.807 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:51.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:51.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:51.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:51.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:51.950 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:52.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:52.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:52.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:52.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:52.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:52.232 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:52.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:52.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:52.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:52.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:52.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:51 smithi190 ceph-mon[33748]: pgmap v222: 158 pgs: 158 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:31:52.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:51 smithi029 ceph-mon[41500]: pgmap v222: 158 pgs: 158 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:31:52.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:51 smithi029 ceph-mon[29919]: pgmap v222: 158 pgs: 158 active+clean; 1.2 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:31:52.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:52.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:52.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:52.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:52.590 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:52.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:52.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:52.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:52.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:52.748 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:52.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:52.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:52.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:52.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:52.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:52.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:52.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:53.013 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:53.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:53.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:53.151 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:53.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:53.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:53.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:53.284 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:53.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:53.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:53.408 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:53.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:53.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:53.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:53.545 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:53.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:53.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:53.685 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:53.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:53.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:53.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:53.822 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:53.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:53.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:53.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:53.955 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:53.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:53.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:54.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:54.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:54.082 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:54.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:54.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:54.225 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:54.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:54.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:54.359 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:54.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:54.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:53 smithi190 ceph-mon[33748]: pgmap v223: 158 pgs: 158 active+clean; 998 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:31:54.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:53 smithi029 ceph-mon[29919]: pgmap v223: 158 pgs: 158 active+clean; 998 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:31:54.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:53 smithi029 ceph-mon[41500]: pgmap v223: 158 pgs: 158 active+clean; 998 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:31:54.460 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:54.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:54.499 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:54.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:54.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:54.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:54.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:54.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:54.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:54.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:54.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:54.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:54.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:54.884 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:54.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:54.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:54.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:55.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:55.031 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:55.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:55.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:55.171 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:55.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:55.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:55.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:55.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:55.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:55.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:55.448 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:55.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:55.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:55.585 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:55.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:55.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:55.727 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:55.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:55.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:55.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:55.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:55.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:55.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:56.022 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:56.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:56.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:56.156 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:56.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:56.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:56.282 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:56.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:56.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:56.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:55 smithi190 ceph-mon[33748]: pgmap v224: 158 pgs: 158 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T12:31:56.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:55 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:31:56.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:55 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:31:56.415 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:56.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:56.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:56.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:55 smithi029 ceph-mon[41500]: pgmap v224: 158 pgs: 158 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T12:31:56.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:55 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:31:56.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:55 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:31:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:55 smithi029 ceph-mon[29919]: pgmap v224: 158 pgs: 158 active+clean; 897 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T12:31:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:55 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:31:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:55 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:31:56.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:56.551 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:56.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:56.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:56.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:56.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:56.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:56.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:56.810 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:56.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:56.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:56.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:56.935 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:56.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:56.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:57.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:57.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:57.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:57.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:57.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:57.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:57.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:57.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:57 smithi190 ceph-mon[33748]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T12:31:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:57 smithi190 ceph-mon[33748]: pgmap v226: 158 pgs: 158 active+clean; 831 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 600 B/s rd, 0 B/s wr, 907 op/s 2024-03-24T12:31:57.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:57.342 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:57.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:57.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:57.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:57 smithi029 ceph-mon[41500]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T12:31:57.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:57 smithi029 ceph-mon[41500]: pgmap v226: 158 pgs: 158 active+clean; 831 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 600 B/s rd, 0 B/s wr, 907 op/s 2024-03-24T12:31:57.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:57 smithi029 ceph-mon[29919]: osdmap e151: 8 total, 8 up, 8 in 2024-03-24T12:31:57.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:57 smithi029 ceph-mon[29919]: pgmap v226: 158 pgs: 158 active+clean; 831 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 600 B/s rd, 0 B/s wr, 907 op/s 2024-03-24T12:31:57.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:57.481 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:57.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:57.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:57.625 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:57.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:57.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:57.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:57.750 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:57.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:57.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:57.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:57.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:57.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:31:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:31:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:31:57.979 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:58.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:31:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:31:58.007 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:58.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:58.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:58.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:31:58.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:58.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:31:58.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:58.286 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:58.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:58.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:58.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:58.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:31:58.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:58.414 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:58.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:31:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:58 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:58.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:58 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:31:58.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:31:58.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:58.556 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:58.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:31:58.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:31:58.692 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:58.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:31:58.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:58.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:58.819 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:58.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:58.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:58.963 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:58.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:58.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:59.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:59.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:59.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:59.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:59.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:31:59.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:59.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:59.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:31:59.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:31:59.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:31:59.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:59.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:31:59.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:31:59 smithi190 ceph-mon[33748]: pgmap v227: 158 pgs: 158 active+clean; 644 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T12:31:59.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:31:59 smithi029 ceph-mon[41500]: pgmap v227: 158 pgs: 158 active+clean; 644 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T12:31:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:31:59 smithi029 ceph-mon[29919]: pgmap v227: 158 pgs: 158 active+clean; 644 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T12:31:59.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:59.502 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:59.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:59.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:59.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:59.635 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:59.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:31:59.736 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:31:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:59.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:31:59.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:31:59.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:31:59.907 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:31:59.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:00.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:00.040 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:00.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:00.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:00.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:00.177 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:00.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:00.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:00.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:00.402 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:00.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:00.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:00 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:32:00.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:00 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:32:00.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:00 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:32:00.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:00 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:32:00.403 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:00 smithi190 ceph-mon[33748]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T12:32:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:00 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:32:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:00 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:32:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:00 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:32:00.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:00 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:32:00.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:00 smithi029 ceph-mon[41500]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T12:32:00.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:00 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:32:00.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:00 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:32:00.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:00 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:32:00.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:00 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:32:00.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:00 smithi029 ceph-mon[29919]: osdmap e152: 8 total, 8 up, 8 in 2024-03-24T12:32:00.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:00.540 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:00.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:00.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:00.681 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:00.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:00.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:00.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:00.821 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:00.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:00.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:00.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:00.953 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:00.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:01.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:01.091 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:01.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:01.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:01.225 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:01.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:01.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:01.362 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:01.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:01.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:01 smithi190 ceph-mon[33748]: pgmap v229: 158 pgs: 158 active+clean; 583 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s 2024-03-24T12:32:01.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:01 smithi029 ceph-mon[41500]: pgmap v229: 158 pgs: 158 active+clean; 583 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s 2024-03-24T12:32:01.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:01 smithi029 ceph-mon[29919]: pgmap v229: 158 pgs: 158 active+clean; 583 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 829 op/s 2024-03-24T12:32:01.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:01.502 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:01.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:01.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:01.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:01.645 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:01.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:01.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:01.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:01.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:01.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:01.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:01.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:01.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:02.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:02.029 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:02.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:02.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:02.174 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:02.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:02.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:02.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:02.314 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:02.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:02.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:02 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:02.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:02 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:02.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:02 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:02.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:02.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:02.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:02.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:02.570 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:02.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:02.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:02.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:02.708 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:02.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:02.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:02.840 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:02.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:02.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:02.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:02.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:03.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:03.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:03.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:03.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:03.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:03.242 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:03.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:03.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:03.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:03 smithi190 ceph-mon[33748]: pgmap v230: 158 pgs: 158 active+clean; 381 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:32:03.388 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:03.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:03.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:03 smithi029 ceph-mon[29919]: pgmap v230: 158 pgs: 158 active+clean; 381 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:32:03.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:03 smithi029 ceph-mon[41500]: pgmap v230: 158 pgs: 158 active+clean; 381 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:32:03.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:03.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:03.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:03.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:03.646 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:03.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:03.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:03.781 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:03.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:03.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:03.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:03.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:03.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:04.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:04.058 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:04.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:04.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:04.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:04.183 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:04.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:04.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:04.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:04.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:04.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:04.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:04.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:04.456 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:04.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:04.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:04.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:04.602 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:04.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:04.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:04.742 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:04.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:04.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:04.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:04.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:04.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:04.977 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:05.018 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:05.019 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:05.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:05.160 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:05.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:05.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:05.284 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:05.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:05.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:05.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:05.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:05.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:05.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:05.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:05.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:05.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:05.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:05.689 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:05.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:05.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:05.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:05.827 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:05.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:05.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:05.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:05.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:05.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:06.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:06.106 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:06.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:06.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:05 smithi029 ceph-mon[41500]: pgmap v231: 158 pgs: 158 active+clean; 280 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:32:06.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:05 smithi029 ceph-mon[29919]: pgmap v231: 158 pgs: 158 active+clean; 280 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:32:06.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:06.238 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Removed 100760 objects 2024-03-24T12:32:06.238 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Clean up completed and total clean up time :141.349 2024-03-24T12:32:06.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:06.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:06.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:06.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:06.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:06.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:06.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:05 smithi190 ceph-mon[33748]: pgmap v231: 158 pgs: 158 active+clean; 280 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:32:06.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:06.496 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:06.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:06.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:06.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:06.623 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:06.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:06.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:06.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:06.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:06.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:06.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:06.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:06.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:06.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:07.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:07.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:07.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:07.130 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:07.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:07.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:07.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:07.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:07.257 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:07.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:07.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:07.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:07.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:07.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:07.514 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:07.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:07.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:07.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:07.645 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:07.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:07.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:07.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:07.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:07.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:07.872 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:32:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:32:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:32:07.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:07.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:07.904 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:07.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:07.943 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:32:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:32:08.005 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:08.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:08.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:08.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:08.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:08.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:07 smithi029 ceph-mon[29919]: pgmap v232: 158 pgs: 158 active+clean; 205 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-24T12:32:08.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:07 smithi029 ceph-mon[41500]: pgmap v232: 158 pgs: 158 active+clean; 205 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-24T12:32:08.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:08.292 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:08.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:08.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:08.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:07 smithi190 ceph-mon[33748]: pgmap v232: 158 pgs: 158 active+clean; 205 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-24T12:32:08.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:08.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:08.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:08.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:08.546 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:08.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:08.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:08.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:08.672 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:08.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:08.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:08.797 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:08.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:08.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:08.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:08.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:09.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:09.051 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:09.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:09.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:09.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:09.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:09.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:09.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:09.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:09.305 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:09.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:09.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:09.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:09.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:09.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:09.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:09.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:09.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:09.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:09.555 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:09.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:09.657 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:09.680 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:09.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:09.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:09.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:09.805 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:09.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:09.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:09.936 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:09.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:10.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:10.061 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:10.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:10.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:09 smithi029 ceph-mon[41500]: pgmap v233: 158 pgs: 158 active+clean; 60 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 938 op/s 2024-03-24T12:32:10.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:09 smithi029 ceph-mon[29919]: pgmap v233: 158 pgs: 158 active+clean; 60 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 938 op/s 2024-03-24T12:32:10.186 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:10.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:10.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:10.313 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:10.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:10.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:09 smithi190 ceph-mon[33748]: pgmap v233: 158 pgs: 158 active+clean; 60 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 938 op/s 2024-03-24T12:32:10.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:10.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:10.439 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:10.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:10.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:10.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:10.569 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:10.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:10.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:10.695 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:10.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:10.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:10.824 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:10.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:10.926 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:10.953 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:10.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:11.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:11.082 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:11.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:11.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:11.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:11.213 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:11.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:11.314 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:11.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:11.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:11.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:11.471 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:11.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:11.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:11.552 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-24T12:32:11.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-24T12:32:11.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:11.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:11.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:11.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:11.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:11.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:11.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:11.855 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:11.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:11.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:11.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:11.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:11.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:12.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:12.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:12.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:12.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:12.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:12.238 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:12.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:12.339 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:12.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:12.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:12.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:11 smithi190 ceph-mon[33748]: pgmap v234: 158 pgs: 158 active+clean; 25 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 781 op/s 2024-03-24T12:32:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:11 smithi029 ceph-mon[41500]: pgmap v234: 158 pgs: 158 active+clean; 25 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 781 op/s 2024-03-24T12:32:12.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:11 smithi029 ceph-mon[29919]: pgmap v234: 158 pgs: 158 active+clean; 25 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 892 B/s rd, 0 B/s wr, 781 op/s 2024-03-24T12:32:12.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:12.497 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:12.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:12.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:12.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:12.626 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:12.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:12.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:12.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:12.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:12.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:12.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:12.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:12.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:12.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:12.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:12.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:13.014 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:13.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:13.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:13.115 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:13.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:13.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:13.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:13.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:13.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:13.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:13.402 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:13.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:13.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:13.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:13.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:13.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:13.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:13.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:13.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:13.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:13.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:13.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:13.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:13.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:13.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:13.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:13.919 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:13.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:13.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:14.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:14.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:14.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:14.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:14.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:14.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:14.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:14.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:14.309 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:14.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:13 smithi190 ceph-mon[33748]: pgmap v235: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 763 op/s 2024-03-24T12:32:14.412 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:14.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:13 smithi029 ceph-mon[41500]: pgmap v235: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 763 op/s 2024-03-24T12:32:14.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:13 smithi029 ceph-mon[29919]: pgmap v235: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 763 op/s 2024-03-24T12:32:14.439 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:14.440 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:14.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:14.568 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:14.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:14.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:14.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:14.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:14.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:14.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:14.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:14.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:14.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:15.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:15.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:15.088 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:15.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:15.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:15.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:15.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:15.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:15.318 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:15.346 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:15.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:15.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:15.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:15.477 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:15.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:15.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:15.604 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:15.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:15.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:15.733 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:15.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:15.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:15.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:15.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:15.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:15.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:15.979 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:15 smithi029 ceph-mon[41500]: pgmap v236: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 494 op/s 2024-03-24T12:32:15.979 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:15 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2867151398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:32:15.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:15 smithi029 ceph-mon[29919]: pgmap v236: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 494 op/s 2024-03-24T12:32:15.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:15 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2867151398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:32:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:15.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:15.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:16.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:16.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:16.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:16.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:16.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:16.245 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:16.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:16.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:16.371 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:16.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:15 smithi190 ceph-mon[33748]: pgmap v236: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 494 op/s 2024-03-24T12:32:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:15 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2867151398' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:32:16.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:16.497 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:16.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:16.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:16.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:16.624 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:16.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:16.726 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:16.750 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:16.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:16.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:16.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:16.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:17.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:17.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:17.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:17.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:17.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:17.232 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:32:17.232 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:32:17.232 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:32:17.232 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:32:17.232 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:32:17.232 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:32:17.233 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:32:17.233 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:32:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:32:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:32:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:32:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:32:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:32:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:32:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:32:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:32:17.234 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:17.234 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:17.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:17.264 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:17.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:17.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:17.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:17.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:17.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:17.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:17.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:17.521 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:17.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:17.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:17.649 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:17.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:17.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:17.776 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:17.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:17.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:17.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:17.904 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:17.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:17.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:17.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:32:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:32:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:32:17.962 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:32:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:32:18.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:18.035 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:18.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:18.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:18.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:18.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:18.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:18.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:18.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:18.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:18.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:17 smithi190 ceph-mon[33748]: pgmap v237: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 358 op/s 2024-03-24T12:32:18.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:17 smithi029 ceph-mon[41500]: pgmap v237: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 358 op/s 2024-03-24T12:32:18.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:17 smithi029 ceph-mon[29919]: pgmap v237: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 358 op/s 2024-03-24T12:32:18.423 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:18.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:18.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:18.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:18.553 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:18.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:18.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:18.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:18.682 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:18.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:18.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:18.811 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:18.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:18.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:18.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:18.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:18.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:19.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:19.070 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:19.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:19.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:19.200 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:19.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:19.223 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:19.223 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:19.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:19.327 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:19.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:19.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:19.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:19.428 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:19.455 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:19.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:19.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:19.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:19.584 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:19.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:19.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:19.711 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:19.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:19.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:19.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:19.836 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:19.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:19.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:19.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:19.966 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:19.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:19.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:20.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:20.092 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:20.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:20.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:19 smithi029 ceph-mon[41500]: pgmap v238: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 259 op/s 2024-03-24T12:32:20.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:19 smithi029 ceph-mon[29919]: pgmap v238: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 259 op/s 2024-03-24T12:32:20.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:20.221 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:20.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:20.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:20.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:20.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:20.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:20.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:19 smithi190 ceph-mon[33748]: pgmap v238: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 259 op/s 2024-03-24T12:32:20.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:20.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:20.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:20.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:20.611 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:20.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:20.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:20.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:20.741 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:20.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:20.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:20.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:20.870 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:20.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:20.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:20.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:20.999 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:21.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:21.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:21.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:21.127 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:21.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:21.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:21.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:21.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:21.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:21.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:21.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:21.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:21.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:21.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:21.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:21.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:21.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:21.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:21.645 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:21.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:21.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:21.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:21.775 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:21.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:21.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:21.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:21.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:22.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:22.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:22.037 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:22.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:22.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:22.165 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:22.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:22.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:22.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:21 smithi029 ceph-mon[41500]: pgmap v239: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 63 op/s 2024-03-24T12:32:22.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:21 smithi029 ceph-mon[29919]: pgmap v239: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 63 op/s 2024-03-24T12:32:22.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:22.296 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:22.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:22.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:21 smithi190 ceph-mon[33748]: pgmap v239: 158 pgs: 158 active+clean; 13 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 63 op/s 2024-03-24T12:32:22.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:22.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:22.424 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:22.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:22.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:22.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:22.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:22.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:22.611 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:22.617 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:22.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:22.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:22.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:22.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:22.809 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:22.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:22.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:22.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:22.935 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:22.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:23.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:23.061 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:23.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:23.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:23.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:23.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:23.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:23.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:23.316 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:23.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:23.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:23.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:23.419 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:32:23.419 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:32:23.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:23.443 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:23.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:23.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:23.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:23.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:23.674 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:23.699 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:23.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:23.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:23.829 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:23.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:23.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:23.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:23.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:23.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:24.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:24.084 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:24.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:24.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:24.162 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:23 smithi029 ceph-mon[29919]: pgmap v240: 158 pgs: 158 active+clean; 13 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T12:32:24.163 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:23 smithi029 ceph-mon[41500]: pgmap v240: 158 pgs: 158 active+clean; 13 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T12:32:24.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:24.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T12:32:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:24.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:24.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:32:24.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:24.375 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:24.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:23 smithi190 ceph-mon[33748]: pgmap v240: 158 pgs: 158 active+clean; 13 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-24T12:32:24.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:24.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:24.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:24.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:24.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:24.629 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:24.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:24.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:24.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:24.755 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:24.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:24.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:24.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:24.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:24.880 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:24.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:24.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:25.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:25.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:25.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:25.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:25.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:25.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:25.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:25.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:25.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:25.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:25.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:25.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:25.523 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:25.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:25.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:25.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:25.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:25.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:25.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:25.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:25.778 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:25.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:25.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:25.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:25.904 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:25.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:26.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:26.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:26.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:26.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:26.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:26.159 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:26.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:26.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:26.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:25 smithi029 ceph-mon[41500]: pgmap v241: 158 pgs: 158 active+clean; 13 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:26.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:25 smithi029 ceph-mon[29919]: pgmap v241: 158 pgs: 158 active+clean; 13 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:26.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:26.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:26.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:26.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:25 smithi190 ceph-mon[33748]: pgmap v241: 158 pgs: 158 active+clean; 13 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:26.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:26.420 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:26.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:26.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:26.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:26.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:26.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:26.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:26.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:26.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:26.679 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:26.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:26.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:26.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:26.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:26.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:26.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:26.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:26.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:27.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:27.068 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:27.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:27.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:27.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:27.196 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:27.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:27.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:27.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:27.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:27.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:27.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:27.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:27.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:27.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:27.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:27.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:27.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:27.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:27.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:27.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:27.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:27.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:27.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:27.835 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:27.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:27.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:32:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:32:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:32:27.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:27.961 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:27.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:27.977 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:32:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:32:28.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:28.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:28.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:28.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:28.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:28.216 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:28.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:28.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:28.303 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:27 smithi029 ceph-mon[41500]: pgmap v242: 158 pgs: 158 active+clean; 13 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:28.303 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:27 smithi029 ceph-mon[29919]: pgmap v242: 158 pgs: 158 active+clean; 13 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:28.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:28.341 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:28.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:28.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:28.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:27 smithi190 ceph-mon[33748]: pgmap v242: 158 pgs: 158 active+clean; 13 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:28.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:28.449 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:28.473 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:28.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:28.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:28.575 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:28.598 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:28.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:28.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:28.691 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:28.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:28.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:28.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:28.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:28.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:28.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:28.854 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:28.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:28.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:28.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:28.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:28.984 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:29.001 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'unique_pool_0' removed 2024-03-24T12:32:29.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:29.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:29.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:29.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:29.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:28 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2784708421' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:32:29.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:29.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:28 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2784708421' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:32:29.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:29.236 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:29.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:29.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:29.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:29.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:29.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:29.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:29.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:28 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2784708421' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:32:29.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:29.513 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:29.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:29.556 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:32:29.558 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:32:29.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:29.638 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:29.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:29.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:29.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:29.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:29.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:29.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:29.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:29.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:32:29.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:30.016 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:30.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:30.118 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:30.141 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:30.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:30.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:30.267 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:30.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:30.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:30.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:30.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:29 smithi190 ceph-mon[33748]: pgmap v243: 158 pgs: 158 active+clean; 13 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:32:30.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:29 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2784708421' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:32:30.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:29 smithi190 ceph-mon[33748]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T12:32:30.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:29 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3692929178' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:32:30.392 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:30.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:30.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:29 smithi029 ceph-mon[41500]: pgmap v243: 158 pgs: 158 active+clean; 13 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:32:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:29 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2784708421' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:32:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:29 smithi029 ceph-mon[41500]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T12:32:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:29 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3692929178' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:32:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:29 smithi029 ceph-mon[29919]: pgmap v243: 158 pgs: 158 active+clean; 13 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:32:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:29 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2784708421' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:32:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:29 smithi029 ceph-mon[29919]: osdmap e153: 8 total, 8 up, 8 in 2024-03-24T12:32:30.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:29 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3692929178' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:32:30.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:30.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:30.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:30.561 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-03-24T12:32:28.985634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1311,"num_read_kb":1157,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":666,"ondisk_log_size":666,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":481,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1706364,"kb_used_data":411580,"kb_used_omap":12,"kb_used_meta":1294707,"kb_avail":748254852,"statfs":{"total":767960285184,"available":766212968448,"internally_reserved":0,"allocated":421457920,"data_stored":418632114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1325780529},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-12582912,"num_objects":-193,"num_object_clones":0,"num_object_copies":-579,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-193,"num_whiteouts":0,"num_read":9,"num_read_kb":9,"num_write":-169665,"num_write_kb":-4469953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-22853,"num_bytes_recovered":-1497628672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.489708"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027748+0000","last_change":"2024-03-24T12:30:56.424651+0000","last_active":"2024-03-24T12:31:59.027748+0000","last_peered":"2024-03-24T12:31:59.027748+0000","last_clean":"2024-03-24T12:31:59.027748+0000","last_became_active":"2024-03-24T12:26:09.350521+0000","last_became_peered":"2024-03-24T12:26:09.350521+0000","last_unstale":"2024-03-24T12:31:59.027748+0000","last_undegraded":"2024-03-24T12:31:59.027748+0000","last_fullsized":"2024-03-24T12:31:59.027748+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:30:56.424569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:30:56.424569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:07.293059+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034857+0000","last_change":"2024-03-24T12:29:17.258327+0000","last_active":"2024-03-24T12:31:59.034857+0000","last_peered":"2024-03-24T12:31:59.034857+0000","last_clean":"2024-03-24T12:31:59.034857+0000","last_became_active":"2024-03-24T12:26:07.335517+0000","last_became_peered":"2024-03-24T12:26:07.335517+0000","last_unstale":"2024-03-24T12:31:59.034857+0000","last_undegraded":"2024-03-24T12:31:59.034857+0000","last_fullsized":"2024-03-24T12:31:59.034857+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:17.258079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:29:17.258079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":192,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.026884+0000","last_change":"2024-03-24T12:29:00.447969+0000","last_active":"2024-03-24T12:31:59.026884+0000","last_peered":"2024-03-24T12:31:59.026884+0000","last_clean":"2024-03-24T12:31:59.026884+0000","last_became_active":"2024-03-24T12:26:03.170822+0000","last_became_peered":"2024-03-24T12:26:03.170822+0000","last_unstale":"2024-03-24T12:31:59.026884+0000","last_undegraded":"2024-03-24T12:31:59.026884+0000","last_fullsized":"2024-03-24T12:31:59.026884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:00.447769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:00.447769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:14.919444+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":211,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027614+0000","last_change":"2024-03-24T12:30:51.375278+0000","last_active":"2024-03-24T12:31:59.027614+0000","last_peered":"2024-03-24T12:31:59.027614+0000","last_clean":"2024-03-24T12:31:59.027614+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:31:59.027614+0000","last_undegraded":"2024-03-24T12:31:59.027614+0000","last_fullsized":"2024-03-24T12:31:59.027614+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:30:51.374990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:30:51.374990+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:20.262442+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":179,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.029044+0000","last_change":"2024-03-24T12:27:35.494737+0000","last_active":"2024-03-24T12:31:59.029044+0000","last_peered":"2024-03-24T12:31:59.029044+0000","last_clean":"2024-03-24T12:31:59.029044+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:31:59.029044+0000","last_undegraded":"2024-03-24T12:31:59.029044+0000","last_fullsized":"2024-03-24T12:31:59.029044+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:35.494635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:35.494635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.515636+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027722+0000","last_change":"2024-03-24T12:29:19.397393+0000","last_active":"2024-03-24T12:31:59.027722+0000","last_peered":"2024-03-24T12:31:59.027722+0000","last_clean":"2024-03-24T12:31:59.027722+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:31:59.027722+0000","last_undegraded":"2024-03-24T12:31:59.027722+0000","last_fullsized":"2024-03-24T12:31:59.027722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:19.397265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:29:19.397265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":192,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.036460+0000","last_change":"2024-03-24T12:27:37.470895+0000","last_active":"2024-03-24T12:31:59.036460+0000","last_peered":"2024-03-24T12:31:59.036460+0000","last_clean":"2024-03-24T12:31:59.036460+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:31:59.036460+0000","last_undegraded":"2024-03-24T12:31:59.036460+0000","last_fullsized":"2024-03-24T12:31:59.036460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:37.470809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:37.470809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"90'22","reported_seq":227,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.026485+0000","last_change":"2024-03-24T12:29:01.369083+0000","last_active":"2024-03-24T12:31:59.026485+0000","last_peered":"2024-03-24T12:31:59.026485+0000","last_clean":"2024-03-24T12:31:59.026485+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:31:59.026485+0000","last_undegraded":"2024-03-24T12:31:59.026485+0000","last_fullsized":"2024-03-24T12:31:59.026485+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:29:01.368806+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:29:01.368806+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.516243+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025473+0000","last_change":"2024-03-24T12:28:43.131837+0000","last_active":"2024-03-24T12:31:59.025473+0000","last_peered":"2024-03-24T12:31:59.025473+0000","last_clean":"2024-03-24T12:31:59.025473+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:31:59.025473+0000","last_undegraded":"2024-03-24T12:31:59.025473+0000","last_fullsized":"2024-03-24T12:31:59.025473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:43.131717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:28:43.131717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":183,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.030186+0000","last_change":"2024-03-24T12:27:44.241377+0000","last_active":"2024-03-24T12:31:59.030186+0000","last_peered":"2024-03-24T12:31:59.030186+0000","last_clean":"2024-03-24T12:31:59.030186+0000","last_became_active":"2024-03-24T12:26:07.331562+0000","last_became_peered":"2024-03-24T12:26:07.331562+0000","last_unstale":"2024-03-24T12:31:59.030186+0000","last_undegraded":"2024-03-24T12:31:59.030186+0000","last_fullsized":"2024-03-24T12:31:59.030186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:44.241291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:44.241291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067382+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.030618+0000","last_change":"2024-03-24T12:29:09.487195+0000","last_active":"2024-03-24T12:31:59.030618+0000","last_peered":"2024-03-24T12:31:59.030618+0000","last_clean":"2024-03-24T12:31:59.030618+0000","last_became_active":"2024-03-24T12:26:03.161544+0000","last_became_peered":"2024-03-24T12:26:03.161544+0000","last_unstale":"2024-03-24T12:31:59.030618+0000","last_undegraded":"2024-03-24T12:31:59.030618+0000","last_fullsized":"2024-03-24T12:31:59.030618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:09.487112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:09.487112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067882+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":211,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027908+0000","last_change":"2024-03-24T12:31:08.707454+0000","last_active":"2024-03-24T12:31:59.027908+0000","last_peered":"2024-03-24T12:31:59.027908+0000","last_clean":"2024-03-24T12:31:59.027908+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:31:59.027908+0000","last_undegraded":"2024-03-24T12:31:59.027908+0000","last_fullsized":"2024-03-24T12:31:59.027908+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:31:08.707365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:31:08.707365+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:36.762040+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.029641+0000","last_change":"2024-03-24T12:28:55.399548+0000","last_active":"2024-03-24T12:31:59.029641+0000","last_peered":"2024-03-24T12:31:59.029641+0000","last_clean":"2024-03-24T12:31:59.029641+0000","last_became_active":"2024-03-24T12:26:09.350721+0000","last_became_peered":"2024-03-24T12:26:09.350721+0000","last_unstale":"2024-03-24T12:31:59.029641+0000","last_undegraded":"2024-03-24T12:31:59.029641+0000","last_fullsized":"2024-03-24T12:31:59.029641+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:55.399469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:28:55.399469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067744+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022074+0000","last_change":"2024-03-24T12:29:03.114986+0000","last_active":"2024-03-24T12:31:59.022074+0000","last_peered":"2024-03-24T12:31:59.022074+0000","last_clean":"2024-03-24T12:31:59.022074+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:31:59.022074+0000","last_undegraded":"2024-03-24T12:31:59.022074+0000","last_fullsized":"2024-03-24T12:31:59.022074+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:03.114851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:29:03.114851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.239151+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.031290+0000","last_change":"2024-03-24T12:28:48.277769+0000","last_active":"2024-03-24T12:31:59.031290+0000","last_peered":"2024-03-24T12:31:59.031290+0000","last_clean":"2024-03-24T12:31:59.031290+0000","last_became_active":"2024-03-24T12:26:03.159997+0000","last_became_peered":"2024-03-24T12:26:03.159997+0000","last_unstale":"2024-03-24T12:31:59.031290+0000","last_undegraded":"2024-03-24T12:31:59.031290+0000","last_fullsized":"2024-03-24T12:31:59.031290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:48.277660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:28:48.277660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067775+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":205,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028894+0000","last_change":"2024-03-24T12:31:00.500630+0000","last_active":"2024-03-24T12:31:59.028894+0000","last_peered":"2024-03-24T12:31:59.028894+0000","last_clean":"2024-03-24T12:31:59.028894+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:31:59.028894+0000","last_undegraded":"2024-03-24T12:31:59.028894+0000","last_fullsized":"2024-03-24T12:31:59.028894+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T12:31:00.500492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:31:00.500492+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:26.112455+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":196,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028518+0000","last_change":"2024-03-24T12:31:23.766176+0000","last_active":"2024-03-24T12:31:59.028518+0000","last_peered":"2024-03-24T12:31:59.028518+0000","last_clean":"2024-03-24T12:31:59.028518+0000","last_became_active":"2024-03-24T12:26:09.643671+0000","last_became_peered":"2024-03-24T12:26:09.643671+0000","last_unstale":"2024-03-24T12:31:59.028518+0000","last_undegraded":"2024-03-24T12:31:59.028518+0000","last_fullsized":"2024-03-24T12:31:59.028518+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:23.766078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:31:23.766078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:49.606869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034604+0000","last_change":"2024-03-24T12:31:22.058451+0000","last_active":"2024-03-24T12:31:59.034604+0000","last_peered":"2024-03-24T12:31:59.034604+0000","last_clean":"2024-03-24T12:31:59.034604+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:31:59.034604+0000","last_undegraded":"2024-03-24T12:31:59.034604+0000","last_fullsized":"2024-03-24T12:31:59.034604+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:22.057948+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:31:22.057948+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:31.842021+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025979+0000","last_change":"2024-03-24T12:27:28.361014+0000","last_active":"2024-03-24T12:31:59.025979+0000","last_peered":"2024-03-24T12:31:59.025979+0000","last_clean":"2024-03-24T12:31:59.025979+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:31:59.025979+0000","last_undegraded":"2024-03-24T12:31:59.025979+0000","last_fullsized":"2024-03-24T12:31:59.025979+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:27:28.360926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":220,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027574+0000","last_change":"2024-03-24T12:28:46.726536+0000","last_active":"2024-03-24T12:31:59.027574+0000","last_peered":"2024-03-24T12:31:59.027574+0000","last_clean":"2024-03-24T12:31:59.027574+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:31:59.027574+0000","last_undegraded":"2024-03-24T12:31:59.027574+0000","last_fullsized":"2024-03-24T12:31:59.027574+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:28:46.726391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:28:46.726391+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:00.568276+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":188,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.036788+0000","last_change":"2024-03-24T12:29:13.161639+0000","last_active":"2024-03-24T12:31:59.036788+0000","last_peered":"2024-03-24T12:31:59.036788+0000","last_clean":"2024-03-24T12:31:59.036788+0000","last_became_active":"2024-03-24T12:26:09.643738+0000","last_became_peered":"2024-03-24T12:26:09.643738+0000","last_unstale":"2024-03-24T12:31:59.036788+0000","last_undegraded":"2024-03-24T12:31:59.036788+0000","last_fullsized":"2024-03-24T12:31:59.036788+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:13.161510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:13.161510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028147+0000","last_change":"2024-03-24T12:31:02.557895+0000","last_active":"2024-03-24T12:31:59.028147+0000","last_peered":"2024-03-24T12:31:59.028147+0000","last_clean":"2024-03-24T12:31:59.028147+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:31:59.028147+0000","last_undegraded":"2024-03-24T12:31:59.028147+0000","last_fullsized":"2024-03-24T12:31:59.028147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:02.557739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:31:02.557739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:10.891849+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027935+0000","last_change":"2024-03-24T12:30:59.635977+0000","last_active":"2024-03-24T12:31:59.027935+0000","last_peered":"2024-03-24T12:31:59.027935+0000","last_clean":"2024-03-24T12:31:59.027935+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:31:59.027935+0000","last_undegraded":"2024-03-24T12:31:59.027935+0000","last_fullsized":"2024-03-24T12:31:59.027935+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:30:59.635892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:30:59.635892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:23.200986+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":210,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.035312+0000","last_change":"2024-03-24T12:27:32.464080+0000","last_active":"2024-03-24T12:31:59.035312+0000","last_peered":"2024-03-24T12:31:59.035312+0000","last_clean":"2024-03-24T12:31:59.035312+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:31:59.035312+0000","last_undegraded":"2024-03-24T12:31:59.035312+0000","last_fullsized":"2024-03-24T12:31:59.035312+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:32.463972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:32.463972+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.515770+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":180,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022247+0000","last_change":"2024-03-24T12:27:34.409721+0000","last_active":"2024-03-24T12:31:59.022247+0000","last_peered":"2024-03-24T12:31:59.022247+0000","last_clean":"2024-03-24T12:31:59.022247+0000","last_became_active":"2024-03-24T12:26:09.343786+0000","last_became_peered":"2024-03-24T12:26:09.343786+0000","last_unstale":"2024-03-24T12:31:59.022247+0000","last_undegraded":"2024-03-24T12:31:59.022247+0000","last_fullsized":"2024-03-24T12:31:59.022247+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:27:34.409577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:34.409577+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.238908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":280,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:22.455526+0000","last_change":"2024-03-24T12:28:51.745977+0000","last_active":"2024-03-24T12:32:22.455526+0000","last_peered":"2024-03-24T12:32:22.455526+0000","last_clean":"2024-03-24T12:32:22.455526+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:32:22.455526+0000","last_undegraded":"2024-03-24T12:32:22.455526+0000","last_fullsized":"2024-03-24T12:32:22.455526+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:28:51.745850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:28:51.745850+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:27.798419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":208,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034755+0000","last_change":"2024-03-24T12:31:25.051613+0000","last_active":"2024-03-24T12:31:59.034755+0000","last_peered":"2024-03-24T12:31:59.034755+0000","last_clean":"2024-03-24T12:31:59.034755+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:31:59.034755+0000","last_undegraded":"2024-03-24T12:31:59.034755+0000","last_fullsized":"2024-03-24T12:31:59.034755+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:31:25.051473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:31:25.051473+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:28.890055+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":211,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034443+0000","last_change":"2024-03-24T12:29:11.474277+0000","last_active":"2024-03-24T12:31:59.034443+0000","last_peered":"2024-03-24T12:31:59.034443+0000","last_clean":"2024-03-24T12:31:59.034443+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:31:59.034443+0000","last_undegraded":"2024-03-24T12:31:59.034443+0000","last_fullsized":"2024-03-24T12:31:59.034443+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:29:11.474132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:29:11.474132+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:14.919377+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":195,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034788+0000","last_change":"2024-03-24T12:31:20.022167+0000","last_active":"2024-03-24T12:31:59.034788+0000","last_peered":"2024-03-24T12:31:59.034788+0000","last_clean":"2024-03-24T12:31:59.034788+0000","last_became_active":"2024-03-24T12:26:09.348355+0000","last_became_peered":"2024-03-24T12:26:09.348355+0000","last_unstale":"2024-03-24T12:31:59.034788+0000","last_undegraded":"2024-03-24T12:31:59.034788+0000","last_fullsized":"2024-03-24T12:31:59.034788+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:20.022091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:31:20.022091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:20.767162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":184,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034859+0000","last_change":"2024-03-24T12:28:46.227145+0000","last_active":"2024-03-24T12:31:59.034859+0000","last_peered":"2024-03-24T12:31:59.034859+0000","last_clean":"2024-03-24T12:31:59.034859+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:31:59.034859+0000","last_undegraded":"2024-03-24T12:31:59.034859+0000","last_fullsized":"2024-03-24T12:31:59.034859+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:46.227003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:28:46.227003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028273+0000","last_change":"2024-03-24T12:28:57.028578+0000","last_active":"2024-03-24T12:31:59.028273+0000","last_peered":"2024-03-24T12:31:59.028273+0000","last_clean":"2024-03-24T12:31:59.028273+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:31:59.028273+0000","last_undegraded":"2024-03-24T12:31:59.028273+0000","last_fullsized":"2024-03-24T12:31:59.028273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:57.028457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:28:57.028457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.516112+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":215,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025703+0000","last_change":"2024-03-24T12:28:49.749828+0000","last_active":"2024-03-24T12:31:59.025703+0000","last_peered":"2024-03-24T12:31:59.025703+0000","last_clean":"2024-03-24T12:31:59.025703+0000","last_became_active":"2024-03-24T12:26:05.317587+0000","last_became_peered":"2024-03-24T12:26:05.317587+0000","last_unstale":"2024-03-24T12:31:59.025703+0000","last_undegraded":"2024-03-24T12:31:59.025703+0000","last_fullsized":"2024-03-24T12:31:59.025703+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:28:49.749615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:28:49.749615+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":179,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.021818+0000","last_change":"2024-03-24T12:27:55.392212+0000","last_active":"2024-03-24T12:31:59.021818+0000","last_peered":"2024-03-24T12:31:59.021818+0000","last_clean":"2024-03-24T12:31:59.021818+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:31:59.021818+0000","last_undegraded":"2024-03-24T12:31:59.021818+0000","last_fullsized":"2024-03-24T12:31:59.021818+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:55.392098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:55.392098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.238941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025812+0000","last_change":"2024-03-24T12:29:11.632402+0000","last_active":"2024-03-24T12:31:59.025812+0000","last_peered":"2024-03-24T12:31:59.025812+0000","last_clean":"2024-03-24T12:31:59.025812+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:31:59.025812+0000","last_undegraded":"2024-03-24T12:31:59.025812+0000","last_fullsized":"2024-03-24T12:31:59.025812+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:11.632288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:29:11.632288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.026990+0000","last_change":"2024-03-24T12:27:59.975825+0000","last_active":"2024-03-24T12:31:59.026990+0000","last_peered":"2024-03-24T12:31:59.026990+0000","last_clean":"2024-03-24T12:31:59.026990+0000","last_became_active":"2024-03-24T12:26:03.170395+0000","last_became_peered":"2024-03-24T12:26:03.170395+0000","last_unstale":"2024-03-24T12:31:59.026990+0000","last_undegraded":"2024-03-24T12:31:59.026990+0000","last_fullsized":"2024-03-24T12:31:59.026990+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:59.975745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:59.975745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":201,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028366+0000","last_change":"2024-03-24T12:28:54.970495+0000","last_active":"2024-03-24T12:31:59.028366+0000","last_peered":"2024-03-24T12:31:59.028366+0000","last_clean":"2024-03-24T12:31:59.028366+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:31:59.028366+0000","last_undegraded":"2024-03-24T12:31:59.028366+0000","last_fullsized":"2024-03-24T12:31:59.028366+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:28:54.970319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:28:54.970319+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.516084+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.029922+0000","last_change":"2024-03-24T12:31:29.996318+0000","last_active":"2024-03-24T12:31:59.029922+0000","last_peered":"2024-03-24T12:31:59.029922+0000","last_clean":"2024-03-24T12:31:59.029922+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:31:59.029922+0000","last_undegraded":"2024-03-24T12:31:59.029922+0000","last_fullsized":"2024-03-24T12:31:59.029922+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:31:29.996197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:57.692075+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022038+0000","last_change":"2024-03-24T12:29:10.991629+0000","last_active":"2024-03-24T12:31:59.022038+0000","last_peered":"2024-03-24T12:31:59.022038+0000","last_clean":"2024-03-24T12:31:59.022038+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:31:59.022038+0000","last_undegraded":"2024-03-24T12:31:59.022038+0000","last_fullsized":"2024-03-24T12:31:59.022038+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:10.991511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:29:10.991511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.239089+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":205,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.033388+0000","last_change":"2024-03-24T12:29:11.579113+0000","last_active":"2024-03-24T12:31:59.033388+0000","last_peered":"2024-03-24T12:31:59.033388+0000","last_clean":"2024-03-24T12:31:59.033388+0000","last_became_active":"2024-03-24T12:26:03.165083+0000","last_became_peered":"2024-03-24T12:26:03.165083+0000","last_unstale":"2024-03-24T12:31:59.033388+0000","last_undegraded":"2024-03-24T12:31:59.033388+0000","last_fullsized":"2024-03-24T12:31:59.033388+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:29:11.579030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:11.579030+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067853+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":214,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.026379+0000","last_change":"2024-03-24T12:31:29.027987+0000","last_active":"2024-03-24T12:31:59.026379+0000","last_peered":"2024-03-24T12:31:59.026379+0000","last_clean":"2024-03-24T12:31:59.026379+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:31:59.026379+0000","last_undegraded":"2024-03-24T12:31:59.026379+0000","last_fullsized":"2024-03-24T12:31:59.026379+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:31:29.027885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:31:29.027885+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:42.355576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":179,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025224+0000","last_change":"2024-03-24T12:27:53.059036+0000","last_active":"2024-03-24T12:31:59.025224+0000","last_peered":"2024-03-24T12:31:59.025224+0000","last_clean":"2024-03-24T12:31:59.025224+0000","last_became_active":"2024-03-24T12:26:09.350656+0000","last_became_peered":"2024-03-24T12:26:09.350656+0000","last_unstale":"2024-03-24T12:31:59.025224+0000","last_undegraded":"2024-03-24T12:31:59.025224+0000","last_fullsized":"2024-03-24T12:31:59.025224+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:27:53.058946+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028955+0000","last_change":"2024-03-24T12:28:56.010757+0000","last_active":"2024-03-24T12:31:59.028955+0000","last_peered":"2024-03-24T12:31:59.028955+0000","last_clean":"2024-03-24T12:31:59.028955+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:31:59.028955+0000","last_undegraded":"2024-03-24T12:31:59.028955+0000","last_fullsized":"2024-03-24T12:31:59.028955+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:56.010642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_clean_scrub_stamp":"2024-03-24T12:28:56.010642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.516142+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022473+0000","last_change":"2024-03-24T12:28:50.243855+0000","last_active":"2024-03-24T12:31:59.022473+0000","last_peered":"2024-03-24T12:31:59.022473+0000","last_clean":"2024-03-24T12:31:59.022473+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:31:59.022473+0000","last_undegraded":"2024-03-24T12:31:59.022473+0000","last_fullsized":"2024-03-24T12:31:59.022473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:50.243722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:50.243722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.239038+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":181,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.029792+0000","last_change":"2024-03-24T12:27:43.194108+0000","last_active":"2024-03-24T12:31:59.029792+0000","last_peered":"2024-03-24T12:31:59.029792+0000","last_clean":"2024-03-24T12:31:59.029792+0000","last_became_active":"2024-03-24T12:26:16.874170+0000","last_became_peered":"2024-03-24T12:26:16.874170+0000","last_unstale":"2024-03-24T12:31:59.029792+0000","last_undegraded":"2024-03-24T12:31:59.029792+0000","last_fullsized":"2024-03-24T12:31:59.029792+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:27:43.194004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:43.194004+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067491+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.026043+0000","last_change":"2024-03-24T12:29:05.266028+0000","last_active":"2024-03-24T12:31:59.026043+0000","last_peered":"2024-03-24T12:31:59.026043+0000","last_clean":"2024-03-24T12:31:59.026043+0000","last_became_active":"2024-03-24T12:26:09.343296+0000","last_became_peered":"2024-03-24T12:26:09.343296+0000","last_unstale":"2024-03-24T12:31:59.026043+0000","last_undegraded":"2024-03-24T12:31:59.026043+0000","last_fullsized":"2024-03-24T12:31:59.026043+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:05.265893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:29:05.265893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.516273+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034101+0000","last_change":"2024-03-24T12:31:10.144227+0000","last_active":"2024-03-24T12:31:59.034101+0000","last_peered":"2024-03-24T12:31:59.034101+0000","last_clean":"2024-03-24T12:31:59.034101+0000","last_became_active":"2024-03-24T12:26:07.330811+0000","last_became_peered":"2024-03-24T12:26:07.330811+0000","last_unstale":"2024-03-24T12:31:59.034101+0000","last_undegraded":"2024-03-24T12:31:59.034101+0000","last_fullsized":"2024-03-24T12:31:59.034101+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:31:10.144101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:35.556107+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":207,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028021+0000","last_change":"2024-03-24T12:31:19.731766+0000","last_active":"2024-03-24T12:31:59.028021+0000","last_peered":"2024-03-24T12:31:59.028021+0000","last_clean":"2024-03-24T12:31:59.028021+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:31:59.028021+0000","last_undegraded":"2024-03-24T12:31:59.028021+0000","last_fullsized":"2024-03-24T12:31:59.028021+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:19.731645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:31:19.731645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:20.301412+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"88'16","reported_seq":227,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.029035+0000","last_change":"2024-03-24T12:31:11.661404+0000","last_active":"2024-03-24T12:31:59.029035+0000","last_peered":"2024-03-24T12:31:59.029035+0000","last_clean":"2024-03-24T12:31:59.029035+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:31:59.029035+0000","last_undegraded":"2024-03-24T12:31:59.029035+0000","last_fullsized":"2024-03-24T12:31:59.029035+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:31:11.661310+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:31:11.661310+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:34.660740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":180,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027028+0000","last_change":"2024-03-24T12:28:47.193654+0000","last_active":"2024-03-24T12:31:59.027028+0000","last_peered":"2024-03-24T12:31:59.027028+0000","last_clean":"2024-03-24T12:31:59.027028+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:31:59.027028+0000","last_undegraded":"2024-03-24T12:31:59.027028+0000","last_fullsized":"2024-03-24T12:31:59.027028+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:28:47.193522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:28:47.193522+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":268,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:18.283738+0000","last_change":"2024-03-24T12:27:46.279895+0000","last_active":"2024-03-24T12:32:18.283738+0000","last_peered":"2024-03-24T12:32:18.283738+0000","last_clean":"2024-03-24T12:32:18.283738+0000","last_became_active":"2024-03-24T12:26:07.329487+0000","last_became_peered":"2024-03-24T12:26:07.329487+0000","last_unstale":"2024-03-24T12:32:18.283738+0000","last_undegraded":"2024-03-24T12:32:18.283738+0000","last_fullsized":"2024-03-24T12:32:18.283738+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:46.279789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:46.279789+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:26.301878+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.021927+0000","last_change":"2024-03-24T12:27:49.334495+0000","last_active":"2024-03-24T12:31:59.021927+0000","last_peered":"2024-03-24T12:31:59.021927+0000","last_clean":"2024-03-24T12:31:59.021927+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:31:59.021927+0000","last_undegraded":"2024-03-24T12:31:59.021927+0000","last_fullsized":"2024-03-24T12:31:59.021927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:49.334399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:49.334399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.238973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":210,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028341+0000","last_change":"2024-03-24T12:29:15.283980+0000","last_active":"2024-03-24T12:31:59.028341+0000","last_peered":"2024-03-24T12:31:59.028341+0000","last_clean":"2024-03-24T12:31:59.028341+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:31:59.028341+0000","last_undegraded":"2024-03-24T12:31:59.028341+0000","last_fullsized":"2024-03-24T12:31:59.028341+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:29:15.283868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:29:15.283868+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.516371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028651+0000","last_change":"2024-03-24T12:29:13.301053+0000","last_active":"2024-03-24T12:31:59.028651+0000","last_peered":"2024-03-24T12:31:59.028651+0000","last_clean":"2024-03-24T12:31:59.028651+0000","last_became_active":"2024-03-24T12:26:09.346062+0000","last_became_peered":"2024-03-24T12:26:09.346062+0000","last_unstale":"2024-03-24T12:31:59.028651+0000","last_undegraded":"2024-03-24T12:31:59.028651+0000","last_fullsized":"2024-03-24T12:31:59.028651+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:13.300936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:13.300936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"93'9","reported_seq":270,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:23.284330+0000","last_change":"2024-03-24T12:27:51.379024+0000","last_active":"2024-03-24T12:32:23.284330+0000","last_peered":"2024-03-24T12:32:23.284330+0000","last_clean":"2024-03-24T12:32:23.284330+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:32:23.284330+0000","last_undegraded":"2024-03-24T12:32:23.284330+0000","last_fullsized":"2024-03-24T12:32:23.284330+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'9","last_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:27:51.378939+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:28.463181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025793+0000","last_change":"2024-03-24T12:29:05.327318+0000","last_active":"2024-03-24T12:31:59.025793+0000","last_peered":"2024-03-24T12:31:59.025793+0000","last_clean":"2024-03-24T12:31:59.025793+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:31:59.025793+0000","last_undegraded":"2024-03-24T12:31:59.025793+0000","last_fullsized":"2024-03-24T12:31:59.025793+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:05.327071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:05.327071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'13","reported_seq":215,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.030947+0000","last_change":"2024-03-24T12:28:47.305482+0000","last_active":"2024-03-24T12:31:59.030947+0000","last_peered":"2024-03-24T12:31:59.030947+0000","last_clean":"2024-03-24T12:31:59.030947+0000","last_became_active":"2024-03-24T12:26:05.358213+0000","last_became_peered":"2024-03-24T12:26:05.358213+0000","last_unstale":"2024-03-24T12:31:59.030947+0000","last_undegraded":"2024-03-24T12:31:59.030947+0000","last_fullsized":"2024-03-24T12:31:59.030947+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:28:47.305346+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:28:47.305346+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067600+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027824+0000","last_change":"2024-03-24T12:28:49.752038+0000","last_active":"2024-03-24T12:31:59.027824+0000","last_peered":"2024-03-24T12:31:59.027824+0000","last_clean":"2024-03-24T12:31:59.027824+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:31:59.027824+0000","last_undegraded":"2024-03-24T12:31:59.027824+0000","last_fullsized":"2024-03-24T12:31:59.027824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:49.751864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:28:49.751864+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:00.568394+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":183,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.036613+0000","last_change":"2024-03-24T12:27:39.473015+0000","last_active":"2024-03-24T12:31:59.036613+0000","last_peered":"2024-03-24T12:31:59.036613+0000","last_clean":"2024-03-24T12:31:59.036613+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:31:59.036613+0000","last_undegraded":"2024-03-24T12:31:59.036613+0000","last_fullsized":"2024-03-24T12:31:59.036613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:39.472900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:39.472900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.515889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022109+0000","last_change":"2024-03-24T12:28:48.188864+0000","last_active":"2024-03-24T12:31:59.022109+0000","last_peered":"2024-03-24T12:31:59.022109+0000","last_clean":"2024-03-24T12:31:59.022109+0000","last_became_active":"2024-03-24T12:26:03.166149+0000","last_became_peered":"2024-03-24T12:26:03.166149+0000","last_unstale":"2024-03-24T12:31:59.022109+0000","last_undegraded":"2024-03-24T12:31:59.022109+0000","last_fullsized":"2024-03-24T12:31:59.022109+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:48.188782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:48.188782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.239017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":211,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034661+0000","last_change":"2024-03-24T12:29:15.371058+0000","last_active":"2024-03-24T12:31:59.034661+0000","last_peered":"2024-03-24T12:31:59.034661+0000","last_clean":"2024-03-24T12:31:59.034661+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:31:59.034661+0000","last_undegraded":"2024-03-24T12:31:59.034661+0000","last_fullsized":"2024-03-24T12:31:59.034661+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:29:15.370927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:29:15.370927+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":179,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022364+0000","last_change":"2024-03-24T12:27:42.333953+0000","last_active":"2024-03-24T12:31:59.022364+0000","last_peered":"2024-03-24T12:31:59.022364+0000","last_clean":"2024-03-24T12:31:59.022364+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:31:59.022364+0000","last_undegraded":"2024-03-24T12:31:59.022364+0000","last_fullsized":"2024-03-24T12:31:59.022364+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:42.333813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:42.333813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.238957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":183,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.033404+0000","last_change":"2024-03-24T12:27:47.315049+0000","last_active":"2024-03-24T12:31:59.033404+0000","last_peered":"2024-03-24T12:31:59.033404+0000","last_clean":"2024-03-24T12:31:59.033404+0000","last_became_active":"2024-03-24T12:26:07.329117+0000","last_became_peered":"2024-03-24T12:26:07.329117+0000","last_unstale":"2024-03-24T12:31:59.033404+0000","last_undegraded":"2024-03-24T12:31:59.033404+0000","last_fullsized":"2024-03-24T12:31:59.033404+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:47.314935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:47.314935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.024156+0000","last_change":"2024-03-24T12:28:43.710187+0000","last_active":"2024-03-24T12:31:59.024156+0000","last_peered":"2024-03-24T12:31:59.024156+0000","last_clean":"2024-03-24T12:31:59.024156+0000","last_became_active":"2024-03-24T12:26:03.161414+0000","last_became_peered":"2024-03-24T12:26:03.161414+0000","last_unstale":"2024-03-24T12:31:59.024156+0000","last_undegraded":"2024-03-24T12:31:59.024156+0000","last_fullsized":"2024-03-24T12:31:59.024156+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:43.710103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:43.710103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":196,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.036864+0000","last_change":"2024-03-24T12:27:50.392580+0000","last_active":"2024-03-24T12:31:59.036864+0000","last_peered":"2024-03-24T12:31:59.036864+0000","last_clean":"2024-03-24T12:31:59.036864+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:31:59.036864+0000","last_undegraded":"2024-03-24T12:31:59.036864+0000","last_fullsized":"2024-03-24T12:31:59.036864+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T12:27:50.392486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:50.392486+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.515702+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.032121+0000","last_change":"2024-03-24T12:29:15.585891+0000","last_active":"2024-03-24T12:31:59.032121+0000","last_peered":"2024-03-24T12:31:59.032121+0000","last_clean":"2024-03-24T12:31:59.032121+0000","last_became_active":"2024-03-24T12:26:09.643602+0000","last_became_peered":"2024-03-24T12:26:09.643602+0000","last_unstale":"2024-03-24T12:31:59.032121+0000","last_undegraded":"2024-03-24T12:31:59.032121+0000","last_fullsized":"2024-03-24T12:31:59.032121+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:15.585805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:15.585805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":211,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022515+0000","last_change":"2024-03-24T12:29:13.102973+0000","last_active":"2024-03-24T12:31:59.022515+0000","last_peered":"2024-03-24T12:31:59.022515+0000","last_clean":"2024-03-24T12:31:59.022515+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:31:59.022515+0000","last_undegraded":"2024-03-24T12:31:59.022515+0000","last_fullsized":"2024-03-24T12:31:59.022515+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:29:13.102846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:13.102846+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.239200+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":220,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.029599+0000","last_change":"2024-03-24T12:28:51.218158+0000","last_active":"2024-03-24T12:31:59.029599+0000","last_peered":"2024-03-24T12:31:59.029599+0000","last_clean":"2024-03-24T12:31:59.029599+0000","last_became_active":"2024-03-24T12:26:05.358217+0000","last_became_peered":"2024-03-24T12:26:05.358217+0000","last_unstale":"2024-03-24T12:31:59.029599+0000","last_undegraded":"2024-03-24T12:31:59.029599+0000","last_fullsized":"2024-03-24T12:31:59.029599+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:28:51.218026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:28:51.218026+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067809+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":279,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:18.283770+0000","last_change":"2024-03-24T12:29:07.944407+0000","last_active":"2024-03-24T12:32:18.283770+0000","last_peered":"2024-03-24T12:32:18.283770+0000","last_clean":"2024-03-24T12:32:18.283770+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:32:18.283770+0000","last_undegraded":"2024-03-24T12:32:18.283770+0000","last_fullsized":"2024-03-24T12:32:18.283770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:29:07.944285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:29:07.944285+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":179,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.029718+0000","last_change":"2024-03-24T12:27:52.426045+0000","last_active":"2024-03-24T12:31:59.029718+0000","last_peered":"2024-03-24T12:31:59.029718+0000","last_clean":"2024-03-24T12:31:59.029718+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:31:59.029718+0000","last_undegraded":"2024-03-24T12:31:59.029718+0000","last_fullsized":"2024-03-24T12:31:59.029718+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:52.425959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:52.425959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.515957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022452+0000","last_change":"2024-03-24T12:27:51.367179+0000","last_active":"2024-03-24T12:31:59.022452+0000","last_peered":"2024-03-24T12:31:59.022452+0000","last_clean":"2024-03-24T12:31:59.022452+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:31:59.022452+0000","last_undegraded":"2024-03-24T12:31:59.022452+0000","last_fullsized":"2024-03-24T12:31:59.022452+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:51.367099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:51.367099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.238925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"140'17","reported_seq":231,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028081+0000","last_change":"2024-03-24T12:31:25.769891+0000","last_active":"2024-03-24T12:31:59.028081+0000","last_peered":"2024-03-24T12:31:59.028081+0000","last_clean":"2024-03-24T12:31:59.028081+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:31:59.028081+0000","last_undegraded":"2024-03-24T12:31:59.028081+0000","last_fullsized":"2024-03-24T12:31:59.028081+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'17","last_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:31:25.769706+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:35.219831+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027941+0000","last_change":"2024-03-24T12:28:50.783336+0000","last_active":"2024-03-24T12:31:59.027941+0000","last_peered":"2024-03-24T12:31:59.027941+0000","last_clean":"2024-03-24T12:31:59.027941+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:31:59.027941+0000","last_undegraded":"2024-03-24T12:31:59.027941+0000","last_fullsized":"2024-03-24T12:31:59.027941+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:50.783207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:28:50.783207+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:00.568503+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":179,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.021666+0000","last_change":"2024-03-24T12:27:53.392878+0000","last_active":"2024-03-24T12:31:59.021666+0000","last_peered":"2024-03-24T12:31:59.021666+0000","last_clean":"2024-03-24T12:31:59.021666+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:31:59.021666+0000","last_undegraded":"2024-03-24T12:31:59.021666+0000","last_fullsized":"2024-03-24T12:31:59.021666+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:53.392787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:53.392787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.238988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028075+0000","last_change":"2024-03-24T12:29:07.739781+0000","last_active":"2024-03-24T12:31:59.028075+0000","last_peered":"2024-03-24T12:31:59.028075+0000","last_clean":"2024-03-24T12:31:59.028075+0000","last_became_active":"2024-03-24T12:26:07.331416+0000","last_became_peered":"2024-03-24T12:26:07.331416+0000","last_unstale":"2024-03-24T12:31:59.028075+0000","last_undegraded":"2024-03-24T12:31:59.028075+0000","last_fullsized":"2024-03-24T12:31:59.028075+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:07.739663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:29:07.739663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:00.568451+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022656+0000","last_change":"2024-03-24T12:29:12.028957+0000","last_active":"2024-03-24T12:31:59.022656+0000","last_peered":"2024-03-24T12:31:59.022656+0000","last_clean":"2024-03-24T12:31:59.022656+0000","last_became_active":"2024-03-24T12:26:03.165214+0000","last_became_peered":"2024-03-24T12:26:03.165214+0000","last_unstale":"2024-03-24T12:31:59.022656+0000","last_undegraded":"2024-03-24T12:31:59.022656+0000","last_fullsized":"2024-03-24T12:31:59.022656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:12.028882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:12.028882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.239121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":226,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025142+0000","last_change":"2024-03-24T12:29:01.707758+0000","last_active":"2024-03-24T12:31:59.025142+0000","last_peered":"2024-03-24T12:31:59.025142+0000","last_clean":"2024-03-24T12:31:59.025142+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:31:59.025142+0000","last_undegraded":"2024-03-24T12:31:59.025142+0000","last_fullsized":"2024-03-24T12:31:59.025142+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:29:01.707630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:29:01.707630+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":179,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.026313+0000","last_change":"2024-03-24T12:28:58.163795+0000","last_active":"2024-03-24T12:31:59.026313+0000","last_peered":"2024-03-24T12:31:59.026313+0000","last_clean":"2024-03-24T12:31:59.026313+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:31:59.026313+0000","last_undegraded":"2024-03-24T12:31:59.026313+0000","last_fullsized":"2024-03-24T12:31:59.026313+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:58.163632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:28:58.163632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.029475+0000","last_change":"2024-03-24T12:29:06.786509+0000","last_active":"2024-03-24T12:31:59.029475+0000","last_peered":"2024-03-24T12:31:59.029475+0000","last_clean":"2024-03-24T12:31:59.029475+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:31:59.029475+0000","last_undegraded":"2024-03-24T12:31:59.029475+0000","last_fullsized":"2024-03-24T12:31:59.029475+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:06.785836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:29:06.785836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:00.568529+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":206,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.031380+0000","last_change":"2024-03-24T12:28:46.292958+0000","last_active":"2024-03-24T12:31:59.031380+0000","last_peered":"2024-03-24T12:31:59.031380+0000","last_clean":"2024-03-24T12:31:59.031380+0000","last_became_active":"2024-03-24T12:26:03.162562+0000","last_became_peered":"2024-03-24T12:26:03.162562+0000","last_unstale":"2024-03-24T12:31:59.031380+0000","last_undegraded":"2024-03-24T12:31:59.031380+0000","last_fullsized":"2024-03-24T12:31:59.031380+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:28:46.292811+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:28:46.292811+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067562+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":219,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025773+0000","last_change":"2024-03-24T12:28:57.147331+0000","last_active":"2024-03-24T12:31:59.025773+0000","last_peered":"2024-03-24T12:31:59.025773+0000","last_clean":"2024-03-24T12:31:59.025773+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:31:59.025773+0000","last_undegraded":"2024-03-24T12:31:59.025773+0000","last_fullsized":"2024-03-24T12:31:59.025773+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:28:57.147200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:28:57.147200+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":179,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.036722+0000","last_change":"2024-03-24T12:27:24.439386+0000","last_active":"2024-03-24T12:31:59.036722+0000","last_peered":"2024-03-24T12:31:59.036722+0000","last_clean":"2024-03-24T12:31:59.036722+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:31:59.036722+0000","last_undegraded":"2024-03-24T12:31:59.036722+0000","last_fullsized":"2024-03-24T12:31:59.036722+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:27:24.439296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.515667+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":201,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.032268+0000","last_change":"2024-03-24T12:27:51.237927+0000","last_active":"2024-03-24T12:31:59.032268+0000","last_peered":"2024-03-24T12:31:59.032268+0000","last_clean":"2024-03-24T12:31:59.032268+0000","last_became_active":"2024-03-24T12:26:03.165771+0000","last_became_peered":"2024-03-24T12:26:03.165771+0000","last_unstale":"2024-03-24T12:31:59.032268+0000","last_undegraded":"2024-03-24T12:31:59.032268+0000","last_fullsized":"2024-03-24T12:31:59.032268+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:27:51.237812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:51.237812+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:26.301936+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":200,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028926+0000","last_change":"2024-03-24T12:30:42.485102+0000","last_active":"2024-03-24T12:31:59.028926+0000","last_peered":"2024-03-24T12:31:59.028926+0000","last_clean":"2024-03-24T12:31:59.028926+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:31:59.028926+0000","last_undegraded":"2024-03-24T12:31:59.028926+0000","last_fullsized":"2024-03-24T12:31:59.028926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:30:42.484993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:30:42.484993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:26.064355+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"90'38","reported_seq":331,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:22.637363+0000","last_change":"2024-03-24T12:29:15.190699+0000","last_active":"2024-03-24T12:32:22.637363+0000","last_peered":"2024-03-24T12:32:22.637363+0000","last_clean":"2024-03-24T12:32:22.637363+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:32:22.637363+0000","last_undegraded":"2024-03-24T12:32:22.637363+0000","last_fullsized":"2024-03-24T12:32:22.637363+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'38","last_scrub_stamp":"2024-03-24T12:29:15.190546+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:29:15.190546+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:23.386980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":183,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.026575+0000","last_change":"2024-03-24T12:27:55.077766+0000","last_active":"2024-03-24T12:31:59.026575+0000","last_peered":"2024-03-24T12:31:59.026575+0000","last_clean":"2024-03-24T12:31:59.026575+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:31:59.026575+0000","last_undegraded":"2024-03-24T12:31:59.026575+0000","last_fullsized":"2024-03-24T12:31:59.026575+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:55.077683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:55.077683+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":188,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022736+0000","last_change":"2024-03-24T12:29:09.997581+0000","last_active":"2024-03-24T12:31:59.022736+0000","last_peered":"2024-03-24T12:31:59.022736+0000","last_clean":"2024-03-24T12:31:59.022736+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:31:59.022736+0000","last_undegraded":"2024-03-24T12:31:59.022736+0000","last_fullsized":"2024-03-24T12:31:59.022736+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:09.997481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:09.997481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:31.209337+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":208,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034635+0000","last_change":"2024-03-24T12:31:13.036831+0000","last_active":"2024-03-24T12:31:59.034635+0000","last_peered":"2024-03-24T12:31:59.034635+0000","last_clean":"2024-03-24T12:31:59.034635+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:31:59.034635+0000","last_undegraded":"2024-03-24T12:31:59.034635+0000","last_fullsized":"2024-03-24T12:31:59.034635+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:13.036316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:31:13.036316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:34.832491+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":207,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028532+0000","last_change":"2024-03-24T12:28:48.746836+0000","last_active":"2024-03-24T12:31:59.028532+0000","last_peered":"2024-03-24T12:31:59.028532+0000","last_clean":"2024-03-24T12:31:59.028532+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:31:59.028532+0000","last_undegraded":"2024-03-24T12:31:59.028532+0000","last_fullsized":"2024-03-24T12:31:59.028532+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:48.746693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:48.746693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:22.741848+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":235,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.035068+0000","last_change":"2024-03-24T12:31:08.237565+0000","last_active":"2024-03-24T12:31:59.035068+0000","last_peered":"2024-03-24T12:31:59.035068+0000","last_clean":"2024-03-24T12:31:59.035068+0000","last_became_active":"2024-03-24T12:26:05.315639+0000","last_became_peered":"2024-03-24T12:26:05.315639+0000","last_unstale":"2024-03-24T12:31:59.035068+0000","last_undegraded":"2024-03-24T12:31:59.035068+0000","last_fullsized":"2024-03-24T12:31:59.035068+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:31:08.237387+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:31:08.237387+0000","last_clean_scrub_stamp":"2024-03-24T12:31:08.237387+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:18.578301+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027897+0000","last_change":"2024-03-24T12:28:45.725377+0000","last_active":"2024-03-24T12:31:59.027897+0000","last_peered":"2024-03-24T12:31:59.027897+0000","last_clean":"2024-03-24T12:31:59.027897+0000","last_became_active":"2024-03-24T12:26:07.343010+0000","last_became_peered":"2024-03-24T12:26:07.343010+0000","last_unstale":"2024-03-24T12:31:59.027897+0000","last_undegraded":"2024-03-24T12:31:59.027897+0000","last_fullsized":"2024-03-24T12:31:59.027897+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:45.724986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:28:45.724986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:00.568477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.031248+0000","last_change":"2024-03-24T12:28:49.270274+0000","last_active":"2024-03-24T12:31:59.031248+0000","last_peered":"2024-03-24T12:31:59.031248+0000","last_clean":"2024-03-24T12:31:59.031248+0000","last_became_active":"2024-03-24T12:26:09.347821+0000","last_became_peered":"2024-03-24T12:26:09.347821+0000","last_unstale":"2024-03-24T12:31:59.031248+0000","last_undegraded":"2024-03-24T12:31:59.031248+0000","last_fullsized":"2024-03-24T12:31:59.031248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:49.270187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:28:49.270187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067715+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":453,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.021863+0000","last_change":"2024-03-24T12:29:02.081702+0000","last_active":"2024-03-24T12:31:59.021863+0000","last_peered":"2024-03-24T12:31:59.021863+0000","last_clean":"2024-03-24T12:31:59.021863+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:31:59.021863+0000","last_undegraded":"2024-03-24T12:31:59.021863+0000","last_fullsized":"2024-03-24T12:31:59.021863+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:29:02.081615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:29:02.081615+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.239056+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":200,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.026851+0000","last_change":"2024-03-24T12:28:45.877854+0000","last_active":"2024-03-24T12:31:59.026851+0000","last_peered":"2024-03-24T12:31:59.026851+0000","last_clean":"2024-03-24T12:31:59.026851+0000","last_became_active":"2024-03-24T12:26:03.166467+0000","last_became_peered":"2024-03-24T12:26:03.166467+0000","last_unstale":"2024-03-24T12:31:59.026851+0000","last_undegraded":"2024-03-24T12:31:59.026851+0000","last_fullsized":"2024-03-24T12:31:59.026851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:45.877722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:45.877722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:23.461266+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":295,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:22.455484+0000","last_change":"2024-03-24T12:29:05.673965+0000","last_active":"2024-03-24T12:32:22.455484+0000","last_peered":"2024-03-24T12:32:22.455484+0000","last_clean":"2024-03-24T12:32:22.455484+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:32:22.455484+0000","last_undegraded":"2024-03-24T12:32:22.455484+0000","last_fullsized":"2024-03-24T12:32:22.455484+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:29:05.673830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:29:05.673830+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:27.798381+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":184,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022328+0000","last_change":"2024-03-24T12:27:46.341475+0000","last_active":"2024-03-24T12:31:59.022328+0000","last_peered":"2024-03-24T12:31:59.022328+0000","last_clean":"2024-03-24T12:31:59.022328+0000","last_became_active":"2024-03-24T12:26:07.337548+0000","last_became_peered":"2024-03-24T12:26:07.337548+0000","last_unstale":"2024-03-24T12:31:59.022328+0000","last_undegraded":"2024-03-24T12:31:59.022328+0000","last_fullsized":"2024-03-24T12:31:59.022328+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:46.341393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:46.341393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:31.209195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":196,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.033829+0000","last_change":"2024-03-24T12:31:06.163566+0000","last_active":"2024-03-24T12:31:59.033829+0000","last_peered":"2024-03-24T12:31:59.033829+0000","last_clean":"2024-03-24T12:31:59.033829+0000","last_became_active":"2024-03-24T12:26:09.349435+0000","last_became_peered":"2024-03-24T12:26:09.349435+0000","last_unstale":"2024-03-24T12:31:59.033829+0000","last_undegraded":"2024-03-24T12:31:59.033829+0000","last_fullsized":"2024-03-24T12:31:59.033829+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:06.163425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:31:06.163425+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":200,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.026161+0000","last_change":"2024-03-24T12:29:10.434582+0000","last_active":"2024-03-24T12:31:59.026161+0000","last_peered":"2024-03-24T12:31:59.026161+0000","last_clean":"2024-03-24T12:31:59.026161+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:31:59.026161+0000","last_undegraded":"2024-03-24T12:31:59.026161+0000","last_fullsized":"2024-03-24T12:31:59.026161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:10.434471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:10.434471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:14.919460+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":587,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:25.875684+0000","last_change":"2024-03-24T12:28:53.153811+0000","last_active":"2024-03-24T12:32:25.875684+0000","last_peered":"2024-03-24T12:32:25.875684+0000","last_clean":"2024-03-24T12:32:25.875684+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:32:25.875684+0000","last_undegraded":"2024-03-24T12:32:25.875684+0000","last_fullsized":"2024-03-24T12:32:25.875684+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:28:53.153687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:53.153687+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:31.209303+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":377,"num_read_kb":372,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":205,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027431+0000","last_change":"2024-03-24T12:27:38.328555+0000","last_active":"2024-03-24T12:31:59.027431+0000","last_peered":"2024-03-24T12:31:59.027431+0000","last_clean":"2024-03-24T12:31:59.027431+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:31:59.027431+0000","last_undegraded":"2024-03-24T12:31:59.027431+0000","last_fullsized":"2024-03-24T12:31:59.027431+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:27:38.328458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:27:38.328458+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.031542+0000","last_change":"2024-03-24T12:28:53.346229+0000","last_active":"2024-03-24T12:31:59.031542+0000","last_peered":"2024-03-24T12:31:59.031542+0000","last_clean":"2024-03-24T12:31:59.031542+0000","last_became_active":"2024-03-24T12:26:07.326457+0000","last_became_peered":"2024-03-24T12:26:07.326457+0000","last_unstale":"2024-03-24T12:31:59.031542+0000","last_undegraded":"2024-03-24T12:31:59.031542+0000","last_fullsized":"2024-03-24T12:31:59.031542+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:53.346108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:28:53.346108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067630+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":188,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.035028+0000","last_change":"2024-03-24T12:29:06.146523+0000","last_active":"2024-03-24T12:31:59.035028+0000","last_peered":"2024-03-24T12:31:59.035028+0000","last_clean":"2024-03-24T12:31:59.035028+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:31:59.035028+0000","last_undegraded":"2024-03-24T12:31:59.035028+0000","last_fullsized":"2024-03-24T12:31:59.035028+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:06.146149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:06.146149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":200,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034034+0000","last_change":"2024-03-24T12:29:05.113311+0000","last_active":"2024-03-24T12:31:59.034034+0000","last_peered":"2024-03-24T12:31:59.034034+0000","last_clean":"2024-03-24T12:31:59.034034+0000","last_became_active":"2024-03-24T12:26:03.167244+0000","last_became_peered":"2024-03-24T12:26:03.167244+0000","last_unstale":"2024-03-24T12:31:59.034034+0000","last_undegraded":"2024-03-24T12:31:59.034034+0000","last_fullsized":"2024-03-24T12:31:59.034034+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:05.112703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:05.112703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":224,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025928+0000","last_change":"2024-03-24T12:29:04.340617+0000","last_active":"2024-03-24T12:31:59.025928+0000","last_peered":"2024-03-24T12:31:59.025928+0000","last_clean":"2024-03-24T12:31:59.025928+0000","last_became_active":"2024-03-24T12:26:05.356745+0000","last_became_peered":"2024-03-24T12:26:05.356745+0000","last_unstale":"2024-03-24T12:31:59.025928+0000","last_undegraded":"2024-03-24T12:31:59.025928+0000","last_fullsized":"2024-03-24T12:31:59.025928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:29:04.340452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:29:04.340452+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:26.777041+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":192,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025261+0000","last_change":"2024-03-24T12:28:48.763223+0000","last_active":"2024-03-24T12:31:59.025261+0000","last_peered":"2024-03-24T12:31:59.025261+0000","last_clean":"2024-03-24T12:31:59.025261+0000","last_became_active":"2024-03-24T12:26:07.337303+0000","last_became_peered":"2024-03-24T12:26:07.337303+0000","last_unstale":"2024-03-24T12:31:59.025261+0000","last_undegraded":"2024-03-24T12:31:59.025261+0000","last_fullsized":"2024-03-24T12:31:59.025261+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:48.763090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:28:48.763090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":180,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.025371+0000","last_change":"2024-03-24T12:28:48.157327+0000","last_active":"2024-03-24T12:31:59.025371+0000","last_peered":"2024-03-24T12:31:59.025371+0000","last_clean":"2024-03-24T12:31:59.025371+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:31:59.025371+0000","last_undegraded":"2024-03-24T12:31:59.025371+0000","last_fullsized":"2024-03-24T12:31:59.025371+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:48.157115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:28:48.157115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":200,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.030848+0000","last_change":"2024-03-24T12:28:57.653033+0000","last_active":"2024-03-24T12:31:59.030848+0000","last_peered":"2024-03-24T12:31:59.030848+0000","last_clean":"2024-03-24T12:31:59.030848+0000","last_became_active":"2024-03-24T12:26:03.168184+0000","last_became_peered":"2024-03-24T12:26:03.168184+0000","last_unstale":"2024-03-24T12:31:59.030848+0000","last_undegraded":"2024-03-24T12:31:59.030848+0000","last_fullsized":"2024-03-24T12:31:59.030848+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:57.652930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:57.652930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:26.302009+0000","scrub_duration":287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"138'15","reported_seq":224,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034248+0000","last_change":"2024-03-24T12:31:12.053763+0000","last_active":"2024-03-24T12:31:59.034248+0000","last_peered":"2024-03-24T12:31:59.034248+0000","last_clean":"2024-03-24T12:31:59.034248+0000","last_became_active":"2024-03-24T12:26:05.316739+0000","last_became_peered":"2024-03-24T12:26:05.316739+0000","last_unstale":"2024-03-24T12:31:59.034248+0000","last_undegraded":"2024-03-24T12:31:59.034248+0000","last_fullsized":"2024-03-24T12:31:59.034248+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:31:12.053497+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:26.056840+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":268,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:18.283726+0000","last_change":"2024-03-24T12:27:46.193858+0000","last_active":"2024-03-24T12:32:18.283726+0000","last_peered":"2024-03-24T12:32:18.283726+0000","last_clean":"2024-03-24T12:32:18.283726+0000","last_became_active":"2024-03-24T12:26:07.336080+0000","last_became_peered":"2024-03-24T12:26:07.336080+0000","last_unstale":"2024-03-24T12:32:18.283726+0000","last_undegraded":"2024-03-24T12:32:18.283726+0000","last_fullsized":"2024-03-24T12:32:18.283726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:27:46.193732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:27:46.193732+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:26.776837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":188,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.022229+0000","last_change":"2024-03-24T12:29:05.078366+0000","last_active":"2024-03-24T12:31:59.022229+0000","last_peered":"2024-03-24T12:31:59.022229+0000","last_clean":"2024-03-24T12:31:59.022229+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:31:59.022229+0000","last_undegraded":"2024-03-24T12:31:59.022229+0000","last_fullsized":"2024-03-24T12:31:59.022229+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:05.078288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:05.078288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":200,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027540+0000","last_change":"2024-03-24T12:29:08.601851+0000","last_active":"2024-03-24T12:31:59.027540+0000","last_peered":"2024-03-24T12:31:59.027540+0000","last_clean":"2024-03-24T12:31:59.027540+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:31:59.027540+0000","last_undegraded":"2024-03-24T12:31:59.027540+0000","last_fullsized":"2024-03-24T12:31:59.027540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:29:08.601720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:08.601720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"93'21","reported_seq":224,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027955+0000","last_change":"2024-03-24T12:31:05.621115+0000","last_active":"2024-03-24T12:31:59.027955+0000","last_peered":"2024-03-24T12:31:59.027955+0000","last_clean":"2024-03-24T12:31:59.027955+0000","last_became_active":"2024-03-24T12:26:05.315454+0000","last_became_peered":"2024-03-24T12:26:05.315454+0000","last_unstale":"2024-03-24T12:31:59.027955+0000","last_undegraded":"2024-03-24T12:31:59.027955+0000","last_fullsized":"2024-03-24T12:31:59.027955+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'21","last_scrub_stamp":"2024-03-24T12:31:05.620984+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:31:05.620984+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:30.748285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":278,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:23.284321+0000","last_change":"2024-03-24T12:28:52.872991+0000","last_active":"2024-03-24T12:32:23.284321+0000","last_peered":"2024-03-24T12:32:23.284321+0000","last_clean":"2024-03-24T12:32:23.284321+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:32:23.284321+0000","last_undegraded":"2024-03-24T12:32:23.284321+0000","last_fullsized":"2024-03-24T12:32:23.284321+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:28:52.872867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:28:52.872867+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:28.463417+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":188,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027848+0000","last_change":"2024-03-24T12:31:13.677850+0000","last_active":"2024-03-24T12:31:59.027848+0000","last_peered":"2024-03-24T12:31:59.027848+0000","last_clean":"2024-03-24T12:31:59.027848+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:31:59.027848+0000","last_undegraded":"2024-03-24T12:31:59.027848+0000","last_fullsized":"2024-03-24T12:31:59.027848+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:13.677718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:31:13.677718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:37.697993+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.036385+0000","last_change":"2024-03-24T12:27:36.482571+0000","last_active":"2024-03-24T12:31:59.036385+0000","last_peered":"2024-03-24T12:31:59.036385+0000","last_clean":"2024-03-24T12:31:59.036385+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:31:59.036385+0000","last_undegraded":"2024-03-24T12:31:59.036385+0000","last_fullsized":"2024-03-24T12:31:59.036385+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:36.482466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:36.482466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.515859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":228,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028602+0000","last_change":"2024-03-24T12:28:53.983340+0000","last_active":"2024-03-24T12:31:59.028602+0000","last_peered":"2024-03-24T12:31:59.028602+0000","last_clean":"2024-03-24T12:31:59.028602+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:31:59.028602+0000","last_undegraded":"2024-03-24T12:31:59.028602+0000","last_fullsized":"2024-03-24T12:31:59.028602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T12:28:53.982595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:28:53.982595+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:14.490348+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":192,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027772+0000","last_change":"2024-03-24T12:31:06.719999+0000","last_active":"2024-03-24T12:31:59.027772+0000","last_peered":"2024-03-24T12:31:59.027772+0000","last_clean":"2024-03-24T12:31:59.027772+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:31:59.027772+0000","last_undegraded":"2024-03-24T12:31:59.027772+0000","last_fullsized":"2024-03-24T12:31:59.027772+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:06.719898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:31:06.719898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:26.064433+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.030528+0000","last_change":"2024-03-24T12:29:12.576453+0000","last_active":"2024-03-24T12:31:59.030528+0000","last_peered":"2024-03-24T12:31:59.030528+0000","last_clean":"2024-03-24T12:31:59.030528+0000","last_became_active":"2024-03-24T12:26:09.351061+0000","last_became_peered":"2024-03-24T12:26:09.351061+0000","last_unstale":"2024-03-24T12:31:59.030528+0000","last_undegraded":"2024-03-24T12:31:59.030528+0000","last_fullsized":"2024-03-24T12:31:59.030528+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:12.576328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:12.576328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067973+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"140'9","reported_seq":285,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:18.283749+0000","last_change":"2024-03-24T12:31:21.743353+0000","last_active":"2024-03-24T12:32:18.283749+0000","last_peered":"2024-03-24T12:32:18.283749+0000","last_clean":"2024-03-24T12:32:18.283749+0000","last_became_active":"2024-03-24T12:26:07.328595+0000","last_became_peered":"2024-03-24T12:26:07.328595+0000","last_unstale":"2024-03-24T12:32:18.283749+0000","last_undegraded":"2024-03-24T12:32:18.283749+0000","last_fullsized":"2024-03-24T12:32:18.283749+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'9","last_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:31:21.743220+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:40.414244+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028463+0000","last_change":"2024-03-24T12:27:42.431823+0000","last_active":"2024-03-24T12:31:59.028463+0000","last_peered":"2024-03-24T12:31:59.028463+0000","last_clean":"2024-03-24T12:31:59.028463+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:31:59.028463+0000","last_undegraded":"2024-03-24T12:31:59.028463+0000","last_fullsized":"2024-03-24T12:31:59.028463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:42.431729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:42.431729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.515800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"92'22","reported_seq":202,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027407+0000","last_change":"2024-03-24T12:28:56.811659+0000","last_active":"2024-03-24T12:31:59.027407+0000","last_peered":"2024-03-24T12:31:59.027407+0000","last_clean":"2024-03-24T12:31:59.027407+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:31:59.027407+0000","last_undegraded":"2024-03-24T12:31:59.027407+0000","last_fullsized":"2024-03-24T12:31:59.027407+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-24T12:28:56.811508+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:28:56.811508+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:00.568424+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":179,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028235+0000","last_change":"2024-03-24T12:27:40.439811+0000","last_active":"2024-03-24T12:31:59.028235+0000","last_peered":"2024-03-24T12:31:59.028235+0000","last_clean":"2024-03-24T12:31:59.028235+0000","last_became_active":"2024-03-24T12:26:09.643739+0000","last_became_peered":"2024-03-24T12:26:09.643739+0000","last_unstale":"2024-03-24T12:31:59.028235+0000","last_undegraded":"2024-03-24T12:31:59.028235+0000","last_fullsized":"2024-03-24T12:31:59.028235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:40.439695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:27:40.439695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.515927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.031488+0000","last_change":"2024-03-24T12:28:44.206074+0000","last_active":"2024-03-24T12:31:59.031488+0000","last_peered":"2024-03-24T12:31:59.031488+0000","last_clean":"2024-03-24T12:31:59.031488+0000","last_became_active":"2024-03-24T12:26:07.332766+0000","last_became_peered":"2024-03-24T12:26:07.332766+0000","last_unstale":"2024-03-24T12:31:59.031488+0000","last_undegraded":"2024-03-24T12:31:59.031488+0000","last_fullsized":"2024-03-24T12:31:59.031488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:44.205995+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:28:44.205995+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:03.067686+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.027756+0000","last_change":"2024-03-24T12:30:47.396816+0000","last_active":"2024-03-24T12:31:59.027756+0000","last_peered":"2024-03-24T12:31:59.027756+0000","last_clean":"2024-03-24T12:31:59.027756+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:31:59.027756+0000","last_undegraded":"2024-03-24T12:31:59.027756+0000","last_fullsized":"2024-03-24T12:31:59.027756+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:30:47.396639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:30:47.396639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:17.055708+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":214,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028440+0000","last_change":"2024-03-24T12:30:57.447665+0000","last_active":"2024-03-24T12:31:59.028440+0000","last_peered":"2024-03-24T12:31:59.028440+0000","last_clean":"2024-03-24T12:31:59.028440+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:31:59.028440+0000","last_undegraded":"2024-03-24T12:31:59.028440+0000","last_fullsized":"2024-03-24T12:31:59.028440+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:30:57.447568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:30:57.447568+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:01.902912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":187,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.029215+0000","last_change":"2024-03-24T12:28:51.782648+0000","last_active":"2024-03-24T12:31:59.029215+0000","last_peered":"2024-03-24T12:31:59.029215+0000","last_clean":"2024-03-24T12:31:59.029215+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:31:59.029215+0000","last_undegraded":"2024-03-24T12:31:59.029215+0000","last_fullsized":"2024-03-24T12:31:59.029215+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:51.782516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:28:51.782516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:00.568309+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.034225+0000","last_change":"2024-03-24T12:31:07.301409+0000","last_active":"2024-03-24T12:31:59.034225+0000","last_peered":"2024-03-24T12:31:59.034225+0000","last_clean":"2024-03-24T12:31:59.034225+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:31:59.034225+0000","last_undegraded":"2024-03-24T12:31:59.034225+0000","last_fullsized":"2024-03-24T12:31:59.034225+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:07.301287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:31:07.301287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:29.974955+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":199,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.024993+0000","last_change":"2024-03-24T12:28:55.717795+0000","last_active":"2024-03-24T12:31:59.024993+0000","last_peered":"2024-03-24T12:31:59.024993+0000","last_clean":"2024-03-24T12:31:59.024993+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:31:59.024993+0000","last_undegraded":"2024-03-24T12:31:59.024993+0000","last_fullsized":"2024-03-24T12:31:59.024993+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:55.717693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:55.717693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"94'25","reported_seq":231,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028272+0000","last_change":"2024-03-24T12:31:09.668493+0000","last_active":"2024-03-24T12:31:59.028272+0000","last_peered":"2024-03-24T12:31:59.028272+0000","last_clean":"2024-03-24T12:31:59.028272+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:31:59.028272+0000","last_undegraded":"2024-03-24T12:31:59.028272+0000","last_fullsized":"2024-03-24T12:31:59.028272+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'25","last_scrub_stamp":"2024-03-24T12:31:09.668392+0000","last_deep_scrub":"94'25","last_deep_scrub_stamp":"2024-03-24T12:28:02.436198+0000","last_clean_scrub_stamp":"2024-03-24T12:31:09.668392+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:38.834880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":195,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028596+0000","last_change":"2024-03-24T12:31:14.702284+0000","last_active":"2024-03-24T12:31:59.028596+0000","last_peered":"2024-03-24T12:31:59.028596+0000","last_clean":"2024-03-24T12:31:59.028596+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:31:59.028596+0000","last_undegraded":"2024-03-24T12:31:59.028596+0000","last_fullsized":"2024-03-24T12:31:59.028596+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:14.702197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:31:14.702197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:19.590963+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":277,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:32:23.284364+0000","last_change":"2024-03-24T12:29:01.800781+0000","last_active":"2024-03-24T12:32:23.284364+0000","last_peered":"2024-03-24T12:32:23.284364+0000","last_clean":"2024-03-24T12:32:23.284364+0000","last_became_active":"2024-03-24T12:26:07.334798+0000","last_became_peered":"2024-03-24T12:26:07.334798+0000","last_unstale":"2024-03-24T12:32:23.284364+0000","last_undegraded":"2024-03-24T12:32:23.284364+0000","last_fullsized":"2024-03-24T12:32:23.284364+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:29:01.799887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:29:01.799887+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:28.669443+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":191,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028857+0000","last_change":"2024-03-24T12:27:49.399753+0000","last_active":"2024-03-24T12:31:59.028857+0000","last_peered":"2024-03-24T12:31:59.028857+0000","last_clean":"2024-03-24T12:31:59.028857+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:31:59.028857+0000","last_undegraded":"2024-03-24T12:31:59.028857+0000","last_fullsized":"2024-03-24T12:31:59.028857+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:27:49.399671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:27:49.399671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:31:59.516021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":218,"reported_epoch":152,"state":"active+clean","last_fresh":"2024-03-24T12:31:59.028143+0000","last_change":"2024-03-24T12:31:17.834752+0000","last_active":"2024-03-24T12:31:59.028143+0000","last_peered":"2024-03-24T12:31:59.028143+0000","last_clean":"2024-03-24T12:31:59.028143+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:31:59.028143+0000","last_undegraded":"2024-03-24T12:31:59.028143+0000","last_fullsized":"2024-03-24T12:31:59.028143+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:31:17.834622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:31:17.834622+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:32:47.421907+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":377,"num_read_kb":372,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037841,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181012,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":179710,"kb_avail":93564140,"statfs":{"total":95995035648,"available":95809679360,"internally_reserved":0,"allocated":1323008,"data_stored":965799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":184023498},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332183,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":167100,"kb_used_data":1268,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93578052,"statfs":{"total":95995035648,"available":95823925248,"internally_reserved":0,"allocated":1298432,"data_stored":965211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":291756,"kb_used_data":148132,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":93453396,"statfs":{"total":95995035648,"available":95696277504,"internally_reserved":0,"allocated":151687168,"data_stored":151305229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":199060,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":93546092,"statfs":{"total":95995035648,"available":95791198208,"internally_reserved":0,"allocated":1323008,"data_stored":965143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":202504645},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411436,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":208184,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":207486,"kb_avail":93536968,"statfs":{"total":95995035648,"available":95781855232,"internally_reserved":0,"allocated":704512,"data_stored":374303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":212466117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117093,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":226272,"kb_used_data":54872,"kb_used_omap":1,"kb_used_meta":171390,"kb_avail":93518880,"statfs":{"total":95995035648,"available":95763333120,"internally_reserved":0,"allocated":56188928,"data_stored":55819203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":175503815},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920865,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":324752,"kb_used_data":203336,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93420400,"statfs":{"total":95995035648,"available":95662489600,"internally_reserved":0,"allocated":208216064,"data_stored":207861549,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626524,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108228,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636924,"statfs":{"total":95995035648,"available":95884210176,"internally_reserved":0,"allocated":716800,"data_stored":375677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:32:30.568 INFO:tasks.thrashosds.thrasher:Setting 3.11 to [3, 7, 6, 5, 2, 1] 2024-03-24T12:32:30.568 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.11', '3', '7', '6', '5', '2', '1'] 2024-03-24T12:32:30.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pg-upmap-items 3.11 3 7 6 5 2 1 2024-03-24T12:32:30.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:30.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:30.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:30.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:30.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:30.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:30.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:30.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:30.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:30.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:30.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:30.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:31.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:31.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:31.148 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:31.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:31.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:30 smithi029 ceph-mon[29919]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:32:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:30 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:32:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:30 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3692929178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:32:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:30 smithi029 ceph-mon[29919]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T12:32:31.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:30 smithi029 ceph-mon[41500]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:32:31.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:30 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:32:31.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:30 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3692929178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:32:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:30 smithi029 ceph-mon[41500]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T12:32:31.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:31.274 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:31.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:31.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:31.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:30 smithi190 ceph-mon[33748]: from='client.14871 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:32:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:30 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:32:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:30 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3692929178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:32:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:30 smithi190 ceph-mon[33748]: osdmap e154: 8 total, 8 up, 8 in 2024-03-24T12:32:31.400 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:31.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:31.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:31.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:31.526 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:31.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:31.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:31.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:31.705 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:31.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:31.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:31.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:31.832 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:31.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:31.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:31.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:31.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:32.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:32.084 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:32.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:32.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:32.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:32.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:32.209 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:32.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:32.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:32.334 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:32.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:32.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:31 smithi190 ceph-mon[33748]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:32:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:31 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:32.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:31 smithi029 ceph-mon[41500]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:32:32.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:32.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:32.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:31 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:32.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:31 smithi029 ceph-mon[29919]: pgmap v246: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:32:32.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:32.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:32.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:31 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:32.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:32.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:32.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:32.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:32.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:32.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:32.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:32.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:32.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:32.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:32.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:32.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:32.848 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:32.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:32.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:32.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:32.978 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:32.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:32.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:33.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:33.103 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:33.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:33.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:32 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:32:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:32 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:32:33.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:32 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:32:33.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:32 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:32:33.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:33.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:33.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:33.330 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:33.354 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:33.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:32 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:32:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:32 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:32:33.401 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T12:32:33.402 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T12:32:33.402 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-24T12:32:33.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-24T12:32:33.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:33.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:33.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:33.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:33.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:33.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:33.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:33.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:33.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:33.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:33.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:33.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:33.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:33.989 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:33.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:34.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:34.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:34.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:34.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:34.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:33 smithi029 ceph-mon[29919]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:32:34.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:33 smithi029 ceph-mon[29919]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T12:32:34.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:32:34.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:32:34.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:34.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:33 smithi029 ceph-mon[41500]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:32:34.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:33 smithi029 ceph-mon[41500]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T12:32:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:32:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:32:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:34.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:34.245 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:34.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:34.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:34.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:34.370 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:34.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:34.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:33 smithi190 ceph-mon[33748]: pgmap v247: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:32:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:33 smithi190 ceph-mon[33748]: osdmap e155: 8 total, 8 up, 8 in 2024-03-24T12:32:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:32:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:32:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:34.441 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:32:34.441 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:32:34.441 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:32:34.441 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:32:34.441 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:32:34.441 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:32:34.441 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:32:34.441 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:32:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:32:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:32:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:32:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:32:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:32:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:32:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:32:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:32:34.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:34.496 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:34.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:34.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:34.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:34.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:34.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:34.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:34.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:34.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:34.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:34.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:34.882 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:34.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:34.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:34.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:35.010 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:35.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:35.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:35.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:35.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:35.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:35.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:35.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:35.270 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:35.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:35.279 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:35 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3252766337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:32:35.279 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:35 smithi029 ceph-mon[29919]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T12:32:35.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:35 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3252766337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:32:35.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:35 smithi029 ceph-mon[41500]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T12:32:35.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:35 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3252766337' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:32:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:35 smithi190 ceph-mon[33748]: pgmap v249: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T12:32:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:35.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:35.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:35.519 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:35.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:35.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:35.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:35.651 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:35.651 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:35.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:35.774 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:35.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:35.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:35.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:35.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:35.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:35.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:36.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:36.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:36.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:36.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:36.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:36.162 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:36.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:36.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:36.288 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:36.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:36.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:36.414 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:36.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:36.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:36.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:36.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:36.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:36.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:36.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:36.663 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:36.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:36.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:36.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:36.788 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:36.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:36.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:36.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:36.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:37.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:37.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:37.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:37.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:37.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:32:37.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:37.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:37 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:32:37.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:37 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:32:37.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:37 smithi029 ceph-mon[29919]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T12:32:37.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:37 smithi029 ceph-mon[29919]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 786 B/s rd, 0 op/s 2024-03-24T12:32:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:37 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:32:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:37 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:32:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:37 smithi029 ceph-mon[41500]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T12:32:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:37 smithi029 ceph-mon[41500]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 786 B/s rd, 0 op/s 2024-03-24T12:32:37.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:37.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:37.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:37.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:37.303 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:37.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:37 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:32:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:37 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:32:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:37 smithi190 ceph-mon[33748]: osdmap e156: 8 total, 8 up, 8 in 2024-03-24T12:32:37.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:37 smithi190 ceph-mon[33748]: pgmap v251: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 786 B/s rd, 0 op/s 2024-03-24T12:32:37.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:37.433 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:37.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:37.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:37.559 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:37.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:37.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:37.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:32:37.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:37.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:37.690 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:37.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:37.739 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:32:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:32:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:32:37.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:37.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:37.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:37.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:37.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:37.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:37.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:37.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:38.006 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:32:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:32:38.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:38.074 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:38.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:38.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:38.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:38.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:38.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:38.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:38.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:38.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:38.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:38.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:38.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:38 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:32:38.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:38 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:32:38.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:38 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:38.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:38 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:32:38.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:38 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:32:38.361 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:38 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:38 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:32:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:38 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:32:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:38 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:38.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:38.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:38.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:38.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:38.590 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:38.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:38.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:38.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:38.719 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:38.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:38.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:38.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:38.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:38.873 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:32:38.873 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:32:38.873 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-03-24T12:32:38.873 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:32:38.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:38.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:38.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:38.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:39.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:39.101 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:39.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:39.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:39.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:39.227 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:39.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:39.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:39.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:39.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:39.352 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:39.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:39 smithi190 ceph-mon[33748]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T12:32:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:39 smithi190 ceph-mon[33748]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:39.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:39 smithi029 ceph-mon[41500]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T12:32:39.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:39 smithi029 ceph-mon[41500]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:39.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:39 smithi029 ceph-mon[29919]: osdmap e157: 8 total, 8 up, 8 in 2024-03-24T12:32:39.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:39 smithi029 ceph-mon[29919]: pgmap v253: 132 pgs: 132 active+clean; 582 KiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:39.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:39.477 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:39.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:39.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:39.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:39.603 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:39.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:39.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:39.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:39.729 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:39.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:39.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:39.858 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:39.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:39.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:39.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T12:32:39.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:39.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:39.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:39.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'unique_pool_1' created 2024-03-24T12:32:40.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:40.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:40.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:40.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:40 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3903005744' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T12:32:40.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:40 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T12:32:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:40 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3903005744' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T12:32:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:40 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T12:32:40.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:40.239 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:40.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:40.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:40.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:40.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:40 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3903005744' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T12:32:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:40 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-24T12:32:40.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:40.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:40.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:40.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:40.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:40.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:40.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:40.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:40.709 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:40.709 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:40.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:40.752 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:40.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:40.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:40.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:40.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:40.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:40.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:41.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:41.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:41.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:41.066 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:41 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-24T12:32:41.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:41 smithi029 ceph-mon[29919]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T12:32:41.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:41 smithi029 ceph-mon[29919]: pgmap v255: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:41.067 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:41 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-24T12:32:41.067 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:41 smithi029 ceph-mon[41500]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T12:32:41.067 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:41 smithi029 ceph-mon[41500]: pgmap v255: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:41.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:41.138 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:41.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:41.239 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:41.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:41.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:41.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:41 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-24T12:32:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:41 smithi190 ceph-mon[33748]: osdmap e158: 8 total, 8 up, 8 in 2024-03-24T12:32:41.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:41 smithi190 ceph-mon[33748]: pgmap v255: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:41.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:41.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:41.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:41.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:41.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:41.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-24T12:32:41.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:41.647 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:41.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:41.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:41.777 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:41.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:41.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:41.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:41.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:41.912 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:41.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:42.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:42.035 INFO:teuthology.orchestra.run.smithi029.stderr:set 3.11 pg_upmap_items mapping to [3->7,6->5,2->1] 2024-03-24T12:32:42.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:42.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:42.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:42.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:42 smithi029 ceph-mon[41500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:32:42.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:42 smithi029 ceph-mon[41500]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T12:32:42.139 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:42 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3265098587' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.11", "id": [3, 7, 6, 5, 2, 1]}]: dispatch 2024-03-24T12:32:42.139 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:42 smithi029 ceph-mon[29919]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:32:42.139 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:42 smithi029 ceph-mon[29919]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T12:32:42.140 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:42 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3265098587' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.11", "id": [3, 7, 6, 5, 2, 1]}]: dispatch 2024-03-24T12:32:42.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:42.176 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:42.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:42.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:42.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:42.307 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:42.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:42.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:42 smithi190 ceph-mon[33748]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:32:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:42 smithi190 ceph-mon[33748]: osdmap e159: 8 total, 8 up, 8 in 2024-03-24T12:32:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:42 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3265098587' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "3.11", "id": [3, 7, 6, 5, 2, 1]}]: dispatch 2024-03-24T12:32:42.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:42.432 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:42.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:42.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:42.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:42.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:42.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:42.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:42.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:42.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:42.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:42.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:42.818 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:42.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:42.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:42.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:42.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:42.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:43.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:43.075 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:43.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:43.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:43.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:43.201 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:43.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:43.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:43.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:43.332 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:43.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:43.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:43 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3265098587' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.11", "id": [3, 7, 6, 5, 2, 1]}]': finished 2024-03-24T12:32:43.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:43 smithi190 ceph-mon[33748]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T12:32:43.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:43 smithi190 ceph-mon[33748]: pgmap v258: 148 pgs: 1 peering, 3 creating+peering, 7 unknown, 137 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:32:43.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:43 smithi190 ceph-mon[33748]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T12:32:43.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:43 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3265098587' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.11", "id": [3, 7, 6, 5, 2, 1]}]': finished 2024-03-24T12:32:43.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:43 smithi029 ceph-mon[29919]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T12:32:43.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:43 smithi029 ceph-mon[29919]: pgmap v258: 148 pgs: 1 peering, 3 creating+peering, 7 unknown, 137 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:32:43.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:43 smithi029 ceph-mon[29919]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T12:32:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:43 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3265098587' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "3.11", "id": [3, 7, 6, 5, 2, 1]}]': finished 2024-03-24T12:32:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:43 smithi029 ceph-mon[41500]: osdmap e160: 8 total, 8 up, 8 in 2024-03-24T12:32:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:43 smithi029 ceph-mon[41500]: pgmap v258: 148 pgs: 1 peering, 3 creating+peering, 7 unknown, 137 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:32:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:43 smithi029 ceph-mon[41500]: osdmap e161: 8 total, 8 up, 8 in 2024-03-24T12:32:43.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:43.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:43.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:43.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:43.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:43.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:43.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:43.716 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:43.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:43.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:43.818 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:43.841 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:43.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:43.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:43.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:43.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:43.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:44.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:44.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:44.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:44.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:44.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:44.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:44.227 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:44.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:44.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:44.356 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:44.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:44.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:44.482 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:44.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:44.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:44.584 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:44.612 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:44.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:44.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:44.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:44.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:44.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:44.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:44.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:44.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:44.869 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:44.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:44.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:44.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:44.999 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:44.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:45.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:45.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:45.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:45.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:45.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:45.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:45.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:45.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:45.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:45.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:45.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:45.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:45.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:45.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:45.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:45.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:45.638 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:45.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:45.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:45.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:45 smithi029 ceph-mon[29919]: pgmap v260: 148 pgs: 1 peering, 3 creating+peering, 144 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:32:45.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:45 smithi029 ceph-mon[41500]: pgmap v260: 148 pgs: 1 peering, 3 creating+peering, 144 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:32:45.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:45.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:45.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:45.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:45.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:45 smithi190 ceph-mon[33748]: pgmap v260: 148 pgs: 1 peering, 3 creating+peering, 144 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:32:45.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:45.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:45.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:45.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:46.017 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:32:46.023 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:46.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:46.037 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:32:46.056 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:32:46.076 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:32:46.096 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:32:46.117 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:32:46.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:46.138 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:32:46.147 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:46.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:46.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:46.158 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:32:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:32:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:32:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:32:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:32:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:32:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:32:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:32:46.160 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:32:46.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:46.272 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:46.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:46.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:46.300 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:46.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:46.402 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:46.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:46.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:46.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:46.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:46.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:46.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:46.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:46.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:46.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:46.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:46.837 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:46 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2759854644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:32:46.837 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:46 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2759854644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:32:46.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:46 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2759854644' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:32:46.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:46.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:46.916 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:46.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:47.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:47.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:47.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:47.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:47.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:47.170 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:47.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:47.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:47.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:47.300 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:47.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:47.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:47.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:47.431 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:47.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:47.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:47.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:47.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:47.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:47.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:47.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:47.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:47.814 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:32:47.814 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:32:47.814 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-24T12:32:47.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:32:47.816 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:47.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:47.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:47.899 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:32:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:32:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:32:47.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:47.943 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:47.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:47.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:48.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:48.074 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:48.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:48.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:32:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:32:48.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:47 smithi190 ceph-mon[33748]: pgmap v261: 148 pgs: 1 peering, 147 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:48.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:48.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:48.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:47 smithi029 ceph-mon[41500]: pgmap v261: 148 pgs: 1 peering, 147 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:48.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:48.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:47 smithi029 ceph-mon[29919]: pgmap v261: 148 pgs: 1 peering, 147 active+clean; 582 KiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:48.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:32:48.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:32:48.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:48.199 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:48.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:48.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:48.329 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:48.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:48.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:48.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:48.454 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:48.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:48.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:48.582 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:48.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:48.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:48.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:48.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:48.811 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:48.835 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:48.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:48.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:48.894 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:32:48.894 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:32:48.912 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:32:48.937 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:48.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:48.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:48.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:49.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:49.091 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:49.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:49.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:48 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:48 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2791689564' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:32:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:48 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:32:49.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:48 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:49.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:48 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2791689564' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:32:49.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:48 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:32:49.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:49.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:49.216 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:49.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:49.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:49.341 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:49.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:49.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:48 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:49.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:48 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2791689564' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:32:49.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:48 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:32:49.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:49.465 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:49.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:49.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:49.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:49.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:49.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:49.719 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:49.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:49.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:32:49.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:49.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:49.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:49.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:49.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:49.978 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:49.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:49.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:50.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:50.104 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:50.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:50.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:50.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:49 smithi029 ceph-mon[29919]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:32:50.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:49 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:32:50.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:49 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:32:50.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:49 smithi029 ceph-mon[29919]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T12:32:50.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:49 smithi029 ceph-mon[41500]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:32:50.137 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:49 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:32:50.137 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:49 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:32:50.137 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:49 smithi029 ceph-mon[41500]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T12:32:50.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:50.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:50.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:50.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:50.359 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:50.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:49 smithi190 ceph-mon[33748]: pgmap v262: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-24T12:32:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:49 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:32:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:49 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:32:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:49 smithi190 ceph-mon[33748]: osdmap e162: 8 total, 8 up, 8 in 2024-03-24T12:32:50.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:50.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:50.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:50.610 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:50.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:50.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:50.736 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:50.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:50.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T12:32:50.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:50.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:50.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:50.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:50.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:50.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:50.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:50.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:50.996 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:51.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:51.106 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:50 smithi029 ceph-mon[41500]: 3.1b scrub starts 2024-03-24T12:32:51.106 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:50 smithi029 ceph-mon[41500]: 3.1b scrub ok 2024-03-24T12:32:51.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:50 smithi029 ceph-mon[29919]: 3.1b scrub starts 2024-03-24T12:32:51.106 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:50 smithi029 ceph-mon[29919]: 3.1b scrub ok 2024-03-24T12:32:51.118 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:51.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:51.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:51.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:51.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:51.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:51.372 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:51.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:51.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:51.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:50 smithi190 ceph-mon[33748]: 3.1b scrub starts 2024-03-24T12:32:51.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:50 smithi190 ceph-mon[33748]: 3.1b scrub ok 2024-03-24T12:32:51.474 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:51.502 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:51.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:51.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:51.632 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:51.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:51.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:51.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:51.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:51.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:51.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:32:51.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:51.887 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:51.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:51.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:51.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:52.014 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:52.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:52.050 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:32:52.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:32:52.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[41500]: pgmap v264: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:32:52.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[41500]: 4.6 scrub starts 2024-03-24T12:32:52.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[41500]: 4.6 scrub ok 2024-03-24T12:32:52.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[41500]: 5.2 scrub starts 2024-03-24T12:32:52.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[41500]: 5.2 scrub ok 2024-03-24T12:32:52.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:32:52.051 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:32:52.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:32:52.051 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:32:52.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[29919]: pgmap v264: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:32:52.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[29919]: 4.6 scrub starts 2024-03-24T12:32:52.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[29919]: 4.6 scrub ok 2024-03-24T12:32:52.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[29919]: 5.2 scrub starts 2024-03-24T12:32:52.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[29919]: 5.2 scrub ok 2024-03-24T12:32:52.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:32:52.052 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:51 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:32:52.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:52.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:52.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:52.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:52.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:52.270 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:52.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:52.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:52.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:51 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:32:52.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:51 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:32:52.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:51 smithi190 ceph-mon[33748]: pgmap v264: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:32:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:51 smithi190 ceph-mon[33748]: 4.6 scrub starts 2024-03-24T12:32:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:51 smithi190 ceph-mon[33748]: 4.6 scrub ok 2024-03-24T12:32:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:51 smithi190 ceph-mon[33748]: 5.2 scrub starts 2024-03-24T12:32:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:51 smithi190 ceph-mon[33748]: 5.2 scrub ok 2024-03-24T12:32:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:51 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:32:52.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:51 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:32:52.395 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:52.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:52.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:52.497 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:52.523 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:52.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:52.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:52.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:52.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:52.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:52.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:52.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:52.778 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:52.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:52.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:52.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:52.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:52.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:53.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:53.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:53.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:53.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:53.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:53.159 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:53.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:53.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:53.283 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:53.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:53.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:52 smithi190 ceph-mon[33748]: 5.1b scrub starts 2024-03-24T12:32:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:52 smithi190 ceph-mon[33748]: 5.1b scrub ok 2024-03-24T12:32:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:52 smithi190 ceph-mon[33748]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T12:32:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:52 smithi190 ceph-mon[33748]: 6.17 scrub starts 2024-03-24T12:32:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:52 smithi190 ceph-mon[33748]: 6.17 scrub ok 2024-03-24T12:32:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:52 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3005767937' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:32:53.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:52 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:32:53.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:53.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:53.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:53.414 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:53.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[29919]: 5.1b scrub starts 2024-03-24T12:32:53.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[29919]: 5.1b scrub ok 2024-03-24T12:32:53.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[29919]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T12:32:53.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[29919]: 6.17 scrub starts 2024-03-24T12:32:53.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[29919]: 6.17 scrub ok 2024-03-24T12:32:53.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3005767937' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:32:53.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:32:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[41500]: 5.1b scrub starts 2024-03-24T12:32:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[41500]: 5.1b scrub ok 2024-03-24T12:32:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[41500]: osdmap e163: 8 total, 8 up, 8 in 2024-03-24T12:32:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[41500]: 6.17 scrub starts 2024-03-24T12:32:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[41500]: 6.17 scrub ok 2024-03-24T12:32:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3005767937' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:32:53.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:52 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:32:53.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:53.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:53.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:53.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:53.598 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-24T12:32:53.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:53.668 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:53.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:53.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:53.757 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:53.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:53.795 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:53.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:53.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:53.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:53.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:53.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:53.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:54.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:54.053 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:54.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:54.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:54.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[29919]: pgmap v266: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:32:54.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[29919]: 3.10 scrub starts 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[29919]: 3.10 scrub ok 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[29919]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[29919]: 5.18 scrub starts 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[29919]: 5.18 scrub ok 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[41500]: pgmap v266: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:32:54.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[41500]: 3.10 scrub starts 2024-03-24T12:32:54.162 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[41500]: 3.10 scrub ok 2024-03-24T12:32:54.162 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:32:54.162 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[41500]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T12:32:54.162 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[41500]: 5.18 scrub starts 2024-03-24T12:32:54.162 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:53 smithi029 ceph-mon[41500]: 5.18 scrub ok 2024-03-24T12:32:54.178 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:54.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:54.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:54.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:54.302 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:54.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:54.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:53 smithi190 ceph-mon[33748]: pgmap v266: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:32:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:53 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:32:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:53 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:32:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:53 smithi190 ceph-mon[33748]: 3.10 scrub starts 2024-03-24T12:32:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:53 smithi190 ceph-mon[33748]: 3.10 scrub ok 2024-03-24T12:32:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:53 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:32:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:53 smithi190 ceph-mon[33748]: osdmap e164: 8 total, 8 up, 8 in 2024-03-24T12:32:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:53 smithi190 ceph-mon[33748]: 5.18 scrub starts 2024-03-24T12:32:54.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:53 smithi190 ceph-mon[33748]: 5.18 scrub ok 2024-03-24T12:32:54.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:54.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:54.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:54.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:54.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:54.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:54.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:54.660 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:54.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:54.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:54.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:54.813 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:54.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:54.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:54.865 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:54.915 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:54.942 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:54.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:54.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:32:55.034 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:32:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 164, 2024-03-24T12:32:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:32:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "created": "2024-03-24T12:20:18.712998+0000", 2024-03-24T12:32:55.034 INFO:teuthology.orchestra.run.smithi029.stdout: "modified": "2024-03-24T12:32:52.965311+0000", 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "last_up_change": "2024-03-24T12:25:48.676565+0000", 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "last_in_change": "2024-03-24T12:25:30.369664+0000", 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_num": 5804032, 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_set": [ 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "nodeep-scrub", 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "pglog_hardlimit", 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "purged_snapdirs", 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_deletes", 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "sortbitwise" 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:32:55.035 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_version": 18, 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_snapshot": "", 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_max": 8, 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd": 8, 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "min_compat_client": "luminous", 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "require_osd_release": "squid", 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_crimson": false, 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: "pools": [ 2024-03-24T12:32:55.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 1, 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".mgr", 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:23:41.491597+0000", 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:32:55.037 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 1, 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 1, 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 1, 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 1, 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 1, 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:32:55.038 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "22", 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:32:55.039 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:32:55.040 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:32:55.041 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:32:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:32:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:32:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:32:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:32:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:32:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:32:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:32:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:32:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_max": 32, 2024-03-24T12:32:55.042 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_min": 1 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": {} 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 7.8899998664855957, 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 7.8899998664855957, 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.37999999523162842, 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 3, 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 3, 2024-03-24T12:32:55.043 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:32:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:32:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:32:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 2, 2024-03-24T12:32:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "datapool", 2024-03-24T12:32:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:01.465080+0000", 2024-03-24T12:32:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 8193, 2024-03-24T12:32:55.044 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 3, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 3, 2024-03-24T12:32:55.045 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 3, 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 3, 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 3, 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.046 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "71", 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 2, 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 71, 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:32:55.047 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:32:55.048 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:32:55.049 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd": {} 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:32:55.050 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.6500000953674316, 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.6500000953674316, 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 3, 2024-03-24T12:32:55.051 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".rgw.root", 2024-03-24T12:32:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:02.100021+0000", 2024-03-24T12:32:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:32:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:32:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:32:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:32:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:32:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:32:55.052 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:32:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:32:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:32:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:32:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:32:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:32:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:32:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:32:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:32:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:32:55.053 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:32:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:32:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:32:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:32:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:32:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:32:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:32:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:32:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "68", 2024-03-24T12:32:55.054 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:32:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:32:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:32:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:32:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:32:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:32:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:32:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:32:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:32:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:32:55.055 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:32:55.056 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:32:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:32:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:32:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:32:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:32:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:32:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:32:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:32:55.057 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:32:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:32:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:32:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:32:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:32:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:32:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:32:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:32:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:32:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:32:55.058 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:32:55.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.059 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:32:55.059 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:32:55.059 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.5, 2024-03-24T12:32:55.059 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.5, 2024-03-24T12:32:55.059 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:32:55.059 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.5, 2024-03-24T12:32:55.059 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.5, 2024-03-24T12:32:55.059 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:32:55.059 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:32:55.060 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:32:55.060 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.060 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.060 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 4, 2024-03-24T12:32:55.060 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:32:55.060 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:03.473862+0000", 2024-03-24T12:32:55.060 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:32:55.060 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:32:55.060 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:32:55.061 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:32:55.061 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:32:55.061 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:32:55.061 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:32:55.061 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:32:55.061 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:32:55.061 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:32:55.062 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:32:55.062 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:32:55.062 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:32:55.062 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:32:55.062 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:32:55.062 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:32:55.062 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:32:55.062 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:32:55.063 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:32:55.063 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:32:55.063 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:32:55.063 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:32:55.063 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:32:55.063 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:32:55.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.063 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "70", 2024-03-24T12:32:55.063 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:32:55.064 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:32:55.064 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:32:55.064 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:32:55.064 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:32:55.064 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:32:55.064 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:32:55.064 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:32:55.064 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:32:55.064 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:32:55.065 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:32:55.065 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:32:55.065 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:32:55.065 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:32:55.065 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:32:55.065 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:32:55.065 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:32:55.065 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:32:55.065 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:32:55.066 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:32:55.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.25, 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.25, 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.25, 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2.25, 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 5, 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:05.573828+0000", 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:32:55.069 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:32:55.070 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "72", 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:32:55.071 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:32:55.072 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:32:55.073 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:32:55.074 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.25, 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.25, 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.25, 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.25, 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:32:55.075 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:32:55.076 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.076 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 6, 2024-03-24T12:32:55.076 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:32:55.076 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:07.645083+0000", 2024-03-24T12:32:55.076 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:32:55.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:55.078 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:32:55.078 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:32:55.078 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:32:55.078 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:32:55.078 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:32:55.078 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:32:55.078 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:32:55.079 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "74", 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:32:55.080 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:32:55.081 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:32:55.082 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_bias": 4 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.083 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.75, 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.75, 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.75, 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.75, 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:32:55.084 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 8, 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "unique_pool_1", 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:32:39.584334+0000", 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:32:55.085 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 16, 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 16, 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 16, 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 16, 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 16, 2024-03-24T12:32:55.086 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:32:55.087 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:32:55.087 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:32:55.087 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:32:55.087 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:32:55.087 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:32:55.087 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:32:55.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.087 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "164", 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:32:55.088 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:32:55.089 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:32:55.089 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:32:55.089 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:32:55.089 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:32:55.089 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:32:55.089 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:32:55.089 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:32:55.090 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:32:55.091 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:32:55.091 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:32:55.091 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:32:55.091 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:32:55.091 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:32:55.091 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:32:55.091 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:32:55.091 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:32:55.091 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:32:55.091 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:32:55.092 INFO:teuthology.orchestra.run.smithi029.stdout: "rados": {} 2024-03-24T12:32:55.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.092 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:32:55.092 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:32:55.092 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2, 2024-03-24T12:32:55.092 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2, 2024-03-24T12:32:55.092 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:32:55.092 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2, 2024-03-24T12:32:55.092 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2, 2024-03-24T12:32:55.093 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:32:55.093 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:32:55.093 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:32:55.093 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.093 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.093 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:32:55.093 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:32:55.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.093 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:32:55.093 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b", 2024-03-24T12:32:55.094 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:32:55.095 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:32:55.095 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:32:55.095 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:32:55.095 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:32:55.095 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:32:55.095 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 8, 2024-03-24T12:32:55.095 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6801", 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6802", 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:32:55.097 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6804", 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.098 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6803", 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6801/2514818464", 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6802/2514818464", 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6804/2514818464", 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6803/2514818464", 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:32:55.099 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc", 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:32:55.100 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 13, 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6805", 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:32:55.101 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6806", 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.102 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6808", 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6807", 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6805/2923396572", 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6806/2923396572", 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6808/2923396572", 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6807/2923396572", 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:32:55.104 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "c3f83505-25ee-4217-b60a-081c36092e92", 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 18, 2024-03-24T12:32:55.105 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 142, 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6809", 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:32:55.106 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6810", 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.107 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6812", 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6811", 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.108 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6809/126032915", 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6810/126032915", 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6812/126032915", 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6811/126032915", 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:32:55.109 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96", 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 27, 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6813", 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6814", 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:32:55.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6816", 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:32:55.112 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6815", 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6813/1080171824", 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6814/1080171824", 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6816/1080171824", 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6815/1080171824", 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:32:55.113 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "5e420304-2798-4d11-9688-d0d89482008c", 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:32:55.114 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 33, 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6800", 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6801", 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6803", 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6802", 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:32:55.117 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6800/3690258869", 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6801/3690258869", 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6803/3690258869", 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6802/3690258869", 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "1213431a-652c-4cbf-b32a-36335430c666", 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 42, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:32:55.119 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6804", 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6805", 2024-03-24T12:32:55.120 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6807", 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.121 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6806", 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6804/1327220488", 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6805/1327220488", 2024-03-24T12:32:55.122 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6807/1327220488", 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6806/1327220488", 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c", 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:32:55.123 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 51, 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.124 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6808", 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6809", 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:32:55.125 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6811", 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.126 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6810", 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6808/1295064976", 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6809/1295064976", 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6811/1295064976", 2024-03-24T12:32:55.127 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6810/1295064976", 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda", 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:32:55.128 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 60, 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 160, 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.129 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6812", 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6813", 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:32:55.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6815", 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.131 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6814", 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6812/110963949", 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6813/110963949", 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6815/110963949", 2024-03-24T12:32:55.132 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6814/110963949", 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_xinfo": [ 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:32:55.133 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:32:37.077608+0000", 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:32:55.134 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:31:58.192639+0000", 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:32:55.135 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:32:35.739678+0000", 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:32:32.374139+0000", 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:32:55.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:31:58.853798+0000", 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.137 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:32:51.012994+0000", 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:32:55.138 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:31:47.120914+0000", 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:32:55.139 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:31:55.205546+0000", 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap": [], 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_items": [ 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "3.11", 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 6, 2024-03-24T12:32:55.140 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 5 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.b", 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.f", 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.142 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.12", 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 5 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_primaries": [], 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_temp": [], 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_temp": [], 2024-03-24T12:32:55.143 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklist": { 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2633022206": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2612078930": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2365028933": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4113609697": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4130171883": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3697529912": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2929430696": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/327626505": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3715448582": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/861146182": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2109454792": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1631806715": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:32:55.144 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/852212806": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1823252077": "2024-03-25T12:21:15.151901+0000" 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: "range_blocklist": {}, 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profiles": { 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: "default": { 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-failure-domain": "osd", 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: "k": "2", 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: "m": "1", 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: "plugin": "jerasure", 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: "technique": "reed_sol_van" 2024-03-24T12:32:55.145 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps_queue": [], 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "new_removed_snaps": [], 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "new_purged_snaps": [], 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_node_flags": {}, 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "device_class_flags": {}, 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode": { 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_enabled": false, 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_bucket_count": 0, 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:32:55.146 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_bucket": 0 2024-03-24T12:32:55.147 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:32:55.147 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:32:55.171 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:55.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:55.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:55.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:55.296 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:55.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:55.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:55.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:54 smithi190 ceph-mon[33748]: 3.4 scrub starts 2024-03-24T12:32:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:54 smithi190 ceph-mon[33748]: 3.4 scrub ok 2024-03-24T12:32:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:54 smithi190 ceph-mon[33748]: 6.10 scrub starts 2024-03-24T12:32:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:54 smithi190 ceph-mon[33748]: 6.10 scrub ok 2024-03-24T12:32:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:54 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:32:55.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:54 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:32:55.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[41500]: 3.4 scrub starts 2024-03-24T12:32:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[41500]: 3.4 scrub ok 2024-03-24T12:32:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[41500]: 6.10 scrub starts 2024-03-24T12:32:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[41500]: 6.10 scrub ok 2024-03-24T12:32:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:32:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:32:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[29919]: 3.4 scrub starts 2024-03-24T12:32:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[29919]: 3.4 scrub ok 2024-03-24T12:32:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[29919]: 6.10 scrub starts 2024-03-24T12:32:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[29919]: 6.10 scrub ok 2024-03-24T12:32:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:32:55.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:54 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:32:55.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:55.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:55.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:55.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:55.554 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:55.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:55.618 DEBUG:teuthology.orchestra.run.smithi029:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-24T12:32:55.620 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T12:32:55.620 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T12:32:55.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:55.669 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:hints = 1 2024-03-24T12:32:55.669 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T12:32:55.669 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Object prefix: benchmark_data_smithi029_108893 2024-03-24T12:32:55.669 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:32:55.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 0 0 0 0 0 0 - 0 2024-03-24T12:32:55.689 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:55.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:55.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:55.835 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:55.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:55.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:55.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:55.976 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:55.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:55.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:56.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:56.122 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:56.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:56.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:56.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:56.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:56.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:56.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:55 smithi190 ceph-mon[33748]: pgmap v268: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:55 smithi190 ceph-mon[33748]: 5.3 scrub starts 2024-03-24T12:32:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:55 smithi190 ceph-mon[33748]: 5.3 scrub ok 2024-03-24T12:32:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:55 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:32:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:55 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:32:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:55 smithi190 ceph-mon[33748]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:32:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:55 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:32:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:55 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:32:56.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:55 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3472354871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:32:56.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:56.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:56.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[29919]: pgmap v268: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[29919]: 5.3 scrub starts 2024-03-24T12:32:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[29919]: 5.3 scrub ok 2024-03-24T12:32:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:32:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[29919]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3472354871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[41500]: pgmap v268: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[41500]: 5.3 scrub starts 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[41500]: 5.3 scrub ok 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[41500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:32:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:55 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3472354871' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:32:56.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:56.534 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:56.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:56.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:56.669 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 1 16 3476 3460 216.229 216.25 0.00422727 0.00461241 2024-03-24T12:32:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:56.686 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:56.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:56.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:56.824 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:56.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:56.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:56.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:56.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:56.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:56.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:57.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:57.105 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:57.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:57.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:32:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:32:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 4.4 scrub ok 2024-03-24T12:32:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:32:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:32:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:32:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:32:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 6.1a scrub starts 2024-03-24T12:32:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 6.1a scrub ok 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 6.1b scrub starts 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 6.1b scrub ok 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 4.4 scrub ok 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 6.1a scrub starts 2024-03-24T12:32:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 6.1a scrub ok 2024-03-24T12:32:57.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 6.1b scrub starts 2024-03-24T12:32:57.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 6.1b scrub ok 2024-03-24T12:32:57.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:56 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:32:57.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:57.242 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:57.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:57.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:32:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 4.4 scrub ok 2024-03-24T12:32:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:32:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:32:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:32:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:32:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 6.1a scrub starts 2024-03-24T12:32:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 6.1a scrub ok 2024-03-24T12:32:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 6.1b scrub starts 2024-03-24T12:32:57.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 6.1b scrub ok 2024-03-24T12:32:57.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:56 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:32:57.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:57.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:57.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:57.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:57.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:32:57.523 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:57.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:32:57.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:57.658 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:57.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:57.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:57.669 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 2 16 6872 6856 214.23 212.25 0.00688662 0.00465874 2024-03-24T12:32:57.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:57.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:57.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:57.895 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:57.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:32:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:32:57.933 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:57.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:58.034 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:58.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:58.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:58.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:58.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:32:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:32:58.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:57 smithi190 ceph-mon[33748]: pgmap v269: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 673 B/s rd, 0 op/s 2024-03-24T12:32:58.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:57 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:32:58.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:57 smithi190 ceph-mon[33748]: 6.6 scrub starts 2024-03-24T12:32:58.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:57 smithi190 ceph-mon[33748]: 6.6 scrub ok 2024-03-24T12:32:58.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:57 smithi190 ceph-mon[33748]: 6.1c scrub starts 2024-03-24T12:32:58.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:57 smithi190 ceph-mon[33748]: 6.1c scrub ok 2024-03-24T12:32:58.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:58.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:58.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:58.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:58.308 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:58.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:58.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:58.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[29919]: pgmap v269: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 673 B/s rd, 0 op/s 2024-03-24T12:32:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:32:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[29919]: 6.6 scrub starts 2024-03-24T12:32:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[29919]: 6.6 scrub ok 2024-03-24T12:32:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[29919]: 6.1c scrub starts 2024-03-24T12:32:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[29919]: 6.1c scrub ok 2024-03-24T12:32:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[41500]: pgmap v269: 148 pgs: 148 active+clean; 582 KiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 673 B/s rd, 0 op/s 2024-03-24T12:32:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:32:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[41500]: 6.6 scrub starts 2024-03-24T12:32:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[41500]: 6.6 scrub ok 2024-03-24T12:32:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[41500]: 6.1c scrub starts 2024-03-24T12:32:58.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:57 smithi029 ceph-mon[41500]: 6.1c scrub ok 2024-03-24T12:32:58.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:32:58.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:58.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:32:58.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:58.596 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:58.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:58.669 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 3 16 9291 9275 193.212 151.188 0.00403685 0.00516575 2024-03-24T12:32:58.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:58.739 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:58.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:58.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:58.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:32:58.865 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:58.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:32:58.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:32:59.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:59.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:32:59.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:59.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:59.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:59.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:59.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:59.264 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:59.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:59.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:59.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:32:59.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: 5.15 scrub starts 2024-03-24T12:32:59.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: 5.15 scrub ok 2024-03-24T12:32:59.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:32:59.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:32:59.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: 5.d scrub starts 2024-03-24T12:32:59.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: 5.d scrub ok 2024-03-24T12:32:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: 4.17 scrub starts 2024-03-24T12:32:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: 4.17 scrub ok 2024-03-24T12:32:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:32:59.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:58 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:32:59.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:59.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:59.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:32:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: 5.15 scrub starts 2024-03-24T12:32:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: 5.15 scrub ok 2024-03-24T12:32:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:32:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:32:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: 5.d scrub starts 2024-03-24T12:32:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: 5.d scrub ok 2024-03-24T12:32:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: 4.17 scrub starts 2024-03-24T12:32:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: 4.17 scrub ok 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: 5.15 scrub starts 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: 5.15 scrub ok 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: 5.d scrub starts 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: 5.d scrub ok 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: 4.17 scrub starts 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: 4.17 scrub ok 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:32:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:58 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:32:59.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:59.538 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:59.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:59.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:32:59.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 4 16 11539 11523 180.031 140.5 0.00386926 0.00554609 2024-03-24T12:32:59.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:32:59.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:59.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:32:59.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:32:59.820 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:59.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:59.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:32:59.855 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:32:59.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:32:59.966 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:32:59.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:32:59.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:00.068 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:00.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[29919]: pgmap v270: 148 pgs: 148 active+clean; 254 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 507 op/s 2024-03-24T12:33:00.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:33:00.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:33:00.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[29919]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T12:33:00.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[29919]: 3.e scrub starts 2024-03-24T12:33:00.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[29919]: 3.e scrub ok 2024-03-24T12:33:00.089 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[29919]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T12:33:00.089 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[41500]: pgmap v270: 148 pgs: 148 active+clean; 254 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 507 op/s 2024-03-24T12:33:00.089 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:33:00.089 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:33:00.089 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[41500]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T12:33:00.089 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[41500]: 3.e scrub starts 2024-03-24T12:33:00.089 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[41500]: 3.e scrub ok 2024-03-24T12:33:00.089 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:32:59 smithi029 ceph-mon[41500]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T12:33:00.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:00.107 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:00.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:00.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:00.261 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:00.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:00.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:59 smithi190 ceph-mon[33748]: pgmap v270: 148 pgs: 148 active+clean; 254 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 32 MiB/s wr, 507 op/s 2024-03-24T12:33:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:59 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:33:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:59 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:33:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:59 smithi190 ceph-mon[33748]: osdmap e165: 8 total, 8 up, 8 in 2024-03-24T12:33:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:59 smithi190 ceph-mon[33748]: 3.e scrub starts 2024-03-24T12:33:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:59 smithi190 ceph-mon[33748]: 3.e scrub ok 2024-03-24T12:33:00.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:32:59 smithi190 ceph-mon[33748]: osdmap e166: 8 total, 8 up, 8 in 2024-03-24T12:33:00.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:00.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:00.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:00.544 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:00.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:00.545 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:00.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:00.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 5 15 14339 14324 179.034 175.062 0.00973987 0.00557907 2024-03-24T12:33:00.684 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:00.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:00.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:00.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:00.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:00.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:00.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:00.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:00.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:01.022 INFO:tasks.thrashosds.thrasher:j is {'epoch': 164, 'fsid': 'bc017092-e9d8-11ee-95cd-87774f69a715', 'created': '2024-03-24T12:20:18.712998+0000', 'modified': '2024-03-24T12:32:52.965311+0000', 'last_up_change': '2024-03-24T12:25:48.676565+0000', 'last_in_change': '2024-03-24T12:25:30.369664+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T12:23:41.491597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-24T12:26:01.465080+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T12:26:02.100021+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T12:26:03.473862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T12:26:05.573828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T12:26:07.645083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-24T12:32:39.584334+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0aa1c49b-7887-4f74-b03c-08cc0adec90b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6801', 'nonce': 2514818464}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6802', 'nonce': 2514818464}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6804', 'nonce': 2514818464}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6803', 'nonce': 2514818464}]}, 'public_addr': '172.21.15.29:6801/2514818464', 'cluster_addr': '172.21.15.29:6802/2514818464', 'heartbeat_back_addr': '172.21.15.29:6804/2514818464', 'heartbeat_front_addr': '172.21.15.29:6803/2514818464', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99d8ef8f-eb59-4882-ab51-b398e6b53bcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6805', 'nonce': 2923396572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6806', 'nonce': 2923396572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6808', 'nonce': 2923396572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6807', 'nonce': 2923396572}]}, 'public_addr': '172.21.15.29:6805/2923396572', 'cluster_addr': '172.21.15.29:6806/2923396572', 'heartbeat_back_addr': '172.21.15.29:6808/2923396572', 'heartbeat_front_addr': '172.21.15.29:6807/2923396572', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c3f83505-25ee-4217-b60a-081c36092e92', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 142, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6809', 'nonce': 126032915}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6810', 'nonce': 126032915}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6812', 'nonce': 126032915}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6811', 'nonce': 126032915}]}, 'public_addr': '172.21.15.29:6809/126032915', 'cluster_addr': '172.21.15.29:6810/126032915', 'heartbeat_back_addr': '172.21.15.29:6812/126032915', 'heartbeat_front_addr': '172.21.15.29:6811/126032915', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f5e479a8-459a-4315-a808-711a7b84bb96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6813', 'nonce': 1080171824}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6814', 'nonce': 1080171824}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6816', 'nonce': 1080171824}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6815', 'nonce': 1080171824}]}, 'public_addr': '172.21.15.29:6813/1080171824', 'cluster_addr': '172.21.15.29:6814/1080171824', 'heartbeat_back_addr': '172.21.15.29:6816/1080171824', 'heartbeat_front_addr': '172.21.15.29:6815/1080171824', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '5e420304-2798-4d11-9688-d0d89482008c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 3690258869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6801', 'nonce': 3690258869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6803', 'nonce': 3690258869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 3690258869}]}, 'public_addr': '172.21.15.190:6800/3690258869', 'cluster_addr': '172.21.15.190:6801/3690258869', 'heartbeat_back_addr': '172.21.15.190:6803/3690258869', 'heartbeat_front_addr': '172.21.15.190:6802/3690258869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1213431a-652c-4cbf-b32a-36335430c666', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 1327220488}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6805', 'nonce': 1327220488}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6807', 'nonce': 1327220488}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 1327220488}]}, 'public_addr': '172.21.15.190:6804/1327220488', 'cluster_addr': '172.21.15.190:6805/1327220488', 'heartbeat_back_addr': '172.21.15.190:6807/1327220488', 'heartbeat_front_addr': '172.21.15.190:6806/1327220488', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0712e511-0257-44f3-a82f-ee09ab957c0c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 1295064976}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6809', 'nonce': 1295064976}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6811', 'nonce': 1295064976}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 1295064976}]}, 'public_addr': '172.21.15.190:6808/1295064976', 'cluster_addr': '172.21.15.190:6809/1295064976', 'heartbeat_back_addr': '172.21.15.190:6811/1295064976', 'heartbeat_front_addr': '172.21.15.190:6810/1295064976', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7e36bde4-14d0-4e7e-9a82-2bb97859dbda', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 110963949}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6813', 'nonce': 110963949}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6815', 'nonce': 110963949}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 110963949}]}, 'public_addr': '172.21.15.190:6812/110963949', 'cluster_addr': '172.21.15.190:6813/110963949', 'heartbeat_back_addr': '172.21.15.190:6815/110963949', 'heartbeat_front_addr': '172.21.15.190:6814/110963949', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:32:37.077608+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:31:58.192639+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:32:35.739678+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:32:32.374139+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:31:58.853798+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:32:51.012994+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:31:47.120914+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:31:55.205546+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.11', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.29:0/2633022206': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2612078930': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2365028933': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4113609697': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4130171883': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/3697529912': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2929430696': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/327626505': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/3715448582': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/861146182': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2109454792': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/1631806715': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:6800/852212806': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/1823252077': '2024-03-25T12:21:15.151901+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:33:01.022 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:33:01.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:01.108 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:01.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:01.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:01.252 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:01.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:01.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:01.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:01 smithi029 ceph-mon[41500]: 5.16 scrub starts 2024-03-24T12:33:01.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:01 smithi029 ceph-mon[41500]: 5.16 scrub ok 2024-03-24T12:33:01.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:01 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:33:01.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:01 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:33:01.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:01 smithi029 ceph-mon[29919]: 5.16 scrub starts 2024-03-24T12:33:01.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:01 smithi029 ceph-mon[29919]: 5.16 scrub ok 2024-03-24T12:33:01.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:01 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:33:01.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:01 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:33:01.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:01 smithi190 ceph-mon[33748]: 5.16 scrub starts 2024-03-24T12:33:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:01 smithi190 ceph-mon[33748]: 5.16 scrub ok 2024-03-24T12:33:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:01 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:33:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:01 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:33:01.390 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:01.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:01.391 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:01.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:01.537 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:01.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:01.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:01.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:01.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:01.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 6 16 16880 16864 175.651 158.75 0.00646191 0.0056854 2024-03-24T12:33:01.769 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:01.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:01.810 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:01.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:01.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:01.954 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:01.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:02.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:02.096 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:02.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:02.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:02.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: pgmap v273: 148 pgs: 148 active+clean; 356 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 679 B/s rd, 47 MiB/s wr, 754 op/s 2024-03-24T12:33:02.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:33:02.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: 3.17 scrub starts 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: 3.17 scrub ok 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: 5.1c scrub starts 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: 5.1c scrub ok 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: 6.12 scrub starts 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: 6.12 scrub ok 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/85719908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:02.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: pgmap v273: 148 pgs: 148 active+clean; 356 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 679 B/s rd, 47 MiB/s wr, 754 op/s 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: 3.17 scrub starts 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: 3.17 scrub ok 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: 5.1c scrub starts 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: 5.1c scrub ok 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: 6.12 scrub starts 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: 6.12 scrub ok 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/85719908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:33:02.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:02.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:02 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:02.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:02.229 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:02.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:02.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:02.374 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:02.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:02.378 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:33:02.378 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:33:02.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:33:02.378 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:33:02.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.378 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:101587 7.e 7:79bfa552:::benchmark_data_smithi029_84282_object824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T12:33:02.378 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:29:46.165039+0000", 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 196.21149154599999, 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.39611948200000002, 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 101587 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.165039+0000", 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.165039+0000", 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.165041+0000", 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5889999999999999e-06 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.165047+0000", 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5030000000000002e-06 2024-03-24T12:33:02.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.165060+0000", 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2906e-05 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.165069+0000", 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1919999999999994e-06 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.165136+0000", 2024-03-24T12:33:02.381 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6698000000000004e-05 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.165274+0000", 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000137918 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.165392+0000", 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011807900000000001 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.382 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.169943+0000", 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0045508509999999999 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.169959+0000", 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6268000000000001e-05 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.561055+0000", 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.39109550300000001 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.561112+0000", 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7228000000000002e-05 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:46.561159+0000", 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6746999999999998e-05 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:107462 7.18 7:19d2e64c:::benchmark_data_smithi029_84282_object6699:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:29:56.643738+0000", 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 185.732793055, 2024-03-24T12:33:02.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.41363156600000001, 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 107462 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.643738+0000", 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.643738+0000", 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.643739+0000", 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.33e-06 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.643745+0000", 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6250000000000004e-06 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.643755+0000", 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0081999999999999e-05 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.643764+0000", 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8799999999999997e-06 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.658709+0000", 2024-03-24T12:33:02.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.014945613999999999 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.658826+0000", 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011657799999999999 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.659181+0000", 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00035494600000000003 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.388 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.671812+0000", 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012631542000000001 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.671842+0000", 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9113000000000001e-05 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.671900+0000", 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8514000000000003e-05 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.055164+0000", 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38326378900000002 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.055242+0000", 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7749000000000004e-05 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.057369+0000", 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0021278040000000001 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.390 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:107480 7.18 7:190140a6:::benchmark_data_smithi029_84282_object6717:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:29:56.663901+0000", 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 185.712630039, 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40190178399999998, 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 107480 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.391 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.663901+0000", 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.663901+0000", 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.663903+0000", 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.069e-06 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.663934+0000", 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1254000000000003e-05 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.663947+0000", 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2563e-05 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.663963+0000", 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6492000000000001e-05 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.671943+0000", 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079793220000000005 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.672159+0000", 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021619900000000001 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.672534+0000", 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000375096 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.673959+0000", 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0014253709999999999 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.674239+0000", 2024-03-24T12:33:02.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027945699999999999 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:56.676364+0000", 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0021254989999999999 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.060542+0000", 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38417834899999997 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.396 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.060616+0000", 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3211000000000001e-05 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.065803+0000", 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0051869020000000002 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:107765 7.e 7:761d1315:::benchmark_data_smithi029_84282_object7002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:29:57.402650+0000", 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 184.97388047300001, 2024-03-24T12:33:02.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.50832110900000005, 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 107765 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.402650+0000", 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.402650+0000", 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.402652+0000", 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1960000000000001e-06 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.402656+0000", 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5820000000000003e-06 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.402666+0000", 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7969999999999995e-06 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.402672+0000", 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2090000000000002e-06 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.402726+0000", 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3980999999999997e-05 2024-03-24T12:33:02.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.402826+0000", 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010011 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.402950+0000", 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012331500000000001 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.406040+0000", 2024-03-24T12:33:02.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003090495 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.407983+0000", 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001942821 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.910867+0000", 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.50288363400000002 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.402 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.910924+0000", 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7936000000000002e-05 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.910971+0000", 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7033000000000002e-05 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.403 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:107772 7.18 7:1df3e6be:::benchmark_data_smithi029_84282_object7009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:29:57.411624+0000", 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 184.96490702700001, 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.49667633, 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 107772 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.405 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.411624+0000", 2024-03-24T12:33:02.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.405 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.405 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.411624+0000", 2024-03-24T12:33:02.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: pgmap v273: 148 pgs: 148 active+clean; 356 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 679 B/s rd, 47 MiB/s wr, 754 op/s 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: 3.17 scrub starts 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: 3.17 scrub ok 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: 5.1c scrub starts 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: 5.1c scrub ok 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: osdmap e167: 8 total, 8 up, 8 in 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: 6.12 scrub starts 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: 6.12 scrub ok 2024-03-24T12:33:02.406 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/85719908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:33:02.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:02.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:02 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:02.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.411625+0000", 2024-03-24T12:33:02.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.11e-06 2024-03-24T12:33:02.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.411629+0000", 2024-03-24T12:33:02.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7299999999999999e-06 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.411637+0000", 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2220000000000002e-06 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.411643+0000", 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1340000000000001e-06 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.412138+0000", 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00049489399999999995 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.412287+0000", 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014923099999999999 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.412635+0000", 2024-03-24T12:33:02.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00034750599999999998 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.414159+0000", 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0015245409999999999 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.414277+0000", 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000117889 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.415192+0000", 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00091478000000000002 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.905954+0000", 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.49076229799999999 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.906033+0000", 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8726000000000005e-05 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.908300+0000", 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0022672690000000001 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:113018 7.18 7:1b344b6a:::benchmark_data_smithi029_84282_object12255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:06.658391+0000", 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 175.71814026600001, 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40163311899999998, 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.412 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 113018 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.658391+0000", 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.658391+0000", 2024-03-24T12:33:02.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.658392+0000", 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5409999999999999e-06 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.658398+0000", 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6949999999999997e-06 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.658410+0000", 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2310000000000001e-05 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.658419+0000", 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1959999999999997e-06 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.670861+0000", 2024-03-24T12:33:02.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012441723 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.671006+0000", 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014521900000000001 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.671346+0000", 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000339372 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.674744+0000", 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0033980479999999999 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.675657+0000", 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00091331099999999998 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.677544+0000", 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0018871300000000001 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:07.055801+0000", 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.37825702300000003 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:07.055932+0000", 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013133099999999999 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:07.060024+0000", 2024-03-24T12:33:02.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0040912199999999996 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:113021 7.e 7:740fc4d1:::benchmark_data_smithi029_84282_object12258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e125)", 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:06.660455+0000", 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 175.71607566599999, 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.401303939, 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.419 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 113021 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.660455+0000", 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.660455+0000", 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.660457+0000", 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7349999999999999e-06 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.660462+0000", 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3800000000000002e-06 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.660474+0000", 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1304999999999999e-05 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.660481+0000", 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7849999999999992e-06 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.662366+0000", 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0018844350000000001 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.662833+0000", 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00046712199999999999 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.662984+0000", 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000150586 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.675723+0000", 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012739179 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:06.679184+0000", 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003461469 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:07.061295+0000", 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38211075 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:07.061701+0000", 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00040610199999999998 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:07.061759+0000", 2024-03-24T12:33:02.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8091000000000003e-05 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:117331 7.18 7:19b625d6:::benchmark_data_smithi029_84282_object16568:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:13.986776+0000", 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 168.389754681, 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.48671273300000001, 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.425 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 117331 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986776+0000", 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986776+0000", 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986778+0000", 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.593e-06 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986783+0000", 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5849999999999999e-06 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986795+0000", 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2003e-05 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986804+0000", 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6969999999999999e-06 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.990578+0000", 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0037737339999999999 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.990730+0000", 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015259599999999999 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.991112+0000", 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00038206399999999999 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.994075+0000", 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0029620779999999999 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.994113+0000", 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8813000000000003e-05 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:14.283222+0000", 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.28910890700000003 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:14.471209+0000", 2024-03-24T12:33:02.430 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.18798647199999999 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:14.471521+0000", 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00031215300000000002 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:14.473489+0000", 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0019680380000000001 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.431 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:117334 7.18 7:187efba6:::benchmark_data_smithi029_84282_object16571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:13.986818+0000", 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 168.38971252100001, 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.49040329300000002, 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 117334 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986818+0000", 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986818+0000", 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986819+0000", 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7800000000000002e-07 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986822+0000", 2024-03-24T12:33:02.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7219999999999999e-06 2024-03-24T12:33:02.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986828+0000", 2024-03-24T12:33:02.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5810000000000002e-06 2024-03-24T12:33:02.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.986834+0000", 2024-03-24T12:33:02.435 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3520000000000003e-06 2024-03-24T12:33:02.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.435 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.435 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.992482+0000", 2024-03-24T12:33:02.435 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.005647081 2024-03-24T12:33:02.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.435 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.992598+0000", 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000116575 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:13.992835+0000", 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023693600000000001 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:14.283231+0000", 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.29039605899999998 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:14.283307+0000", 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5863000000000002e-05 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:14.283360+0000", 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3365000000000002e-05 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:14.473588+0000", 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.190227283 2024-03-24T12:33:02.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:14.474117+0000", 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00052983900000000005 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:14.477222+0000", 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003104159 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:123641 7.18 7:1a69cc71:::benchmark_data_smithi029_84282_object22878:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:24.257349+0000", 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 158.11918166999999, 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.46551677699999999, 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 123641 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.257349+0000", 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.257349+0000", 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.257350+0000", 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.311e-06 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.257356+0000", 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3369999999999999e-06 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.257366+0000", 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0229e-05 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.257374+0000", 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1089999999999998e-06 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.441 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.259509+0000", 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0021352110000000001 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.259653+0000", 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014328799999999999 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.260231+0000", 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00057865100000000001 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.262576+0000", 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0023447839999999999 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.262941+0000", 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036500200000000001 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.266564+0000", 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0036230720000000002 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.718352+0000", 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45178782099999998 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.718514+0000", 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016191999999999999 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.722866+0000", 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0043520420000000004 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:123646 7.18 7:1a3485ac:::benchmark_data_smithi029_84282_object22883:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:24.263971+0000", 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 158.11255989099999, 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.46296282900000002, 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 123646 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.263971+0000", 2024-03-24T12:33:02.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.263971+0000", 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.263973+0000", 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.886e-06 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.446 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.263978+0000", 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3990000000000003e-06 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.263991+0000", 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2557e-05 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.264000+0000", 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7730000000000005e-06 2024-03-24T12:33:02.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.264062+0000", 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2509999999999996e-05 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.264159+0000", 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7225999999999994e-05 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.264408+0000", 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024819499999999999 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.266574+0000", 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0021664969999999999 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.266612+0000", 2024-03-24T12:33:02.449 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7939999999999999e-05 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.266640+0000", 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.8119999999999998e-05 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.723993+0000", 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45735246200000002 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.450 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.724057+0000", 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4510000000000004e-05 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:24.726934+0000", 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002876754 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.451 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:124699 7.13 7:ca893f43:::benchmark_data_smithi029_84282_object23936:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:26.244354+0000", 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 156.13217721800001, 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.48815093900000001, 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 124699 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.244354+0000", 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.244354+0000", 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.244355+0000", 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7799999999999999e-06 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.244360+0000", 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9559999999999996e-06 2024-03-24T12:33:02.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.244371+0000", 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0977e-05 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.244379+0000", 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6420000000000007e-06 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.454 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.244445+0000", 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5565e-05 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.244575+0000", 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013037699999999999 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,3,4,7", 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.244697+0000", 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012238099999999999 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.246728+0000", 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020309600000000001 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.246889+0000", 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016056499999999999 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.247069+0000", 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018050100000000001 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.249467+0000", 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0023971769999999999 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.732250+0000", 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.48278391500000001 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.732366+0000", 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011587900000000001 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.732505+0000", 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000138264 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:124702 7.e 7:7cd66e2e:::benchmark_data_smithi029_84282_object23939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:26.249110+0000", 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 156.12742055499999, 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.48682834800000002, 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.458 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 124702 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.249110+0000", 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.249110+0000", 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.249112+0000", 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.217e-06 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.249116+0000", 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7310000000000001e-06 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.249128+0000", 2024-03-24T12:33:02.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1724e-05 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.249137+0000", 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9549999999999998e-06 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.249239+0000", 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000101815 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.249373+0000", 2024-03-24T12:33:02.461 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013446700000000001 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.249496+0000", 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012266999999999999 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252260+0000", 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027636850000000001 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.462 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252275+0000", 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5835e-05 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.735832+0000", 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.483556233 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.735886+0000", 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4076999999999997e-05 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.735939+0000", 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2939000000000003e-05 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.14799.0:124705 7.15 e129/120 7:ad2c8cda:::benchmark_data_smithi029_84282_object23942:head v 129'6711, mlcod=129'6709)", 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:26.252378+0000", 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 156.12415274700001, 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.48148892599999998, 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252378+0000", 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252378+0000", 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252379+0000", 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8500000000000002e-07 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.465 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252382+0000", 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8310000000000003e-06 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252388+0000", 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7039999999999999e-06 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252392+0000", 2024-03-24T12:33:02.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0439999999999997e-06 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252437+0000", 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4450999999999998e-05 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252452+0000", 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5078e-05 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.467 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.733822+0000", 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.48137010200000002 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.733867+0000", 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5231000000000001e-05 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:124707 7.13 7:cbbff4aa:::benchmark_data_smithi029_84282_object23944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:26.252999+0000", 2024-03-24T12:33:02.468 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 156.12353161199999, 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.49545725699999998, 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 124707 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252999+0000", 2024-03-24T12:33:02.469 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.252999+0000", 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.253000+0000", 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1000000000000001e-06 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.470 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.253005+0000", 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2350000000000001e-06 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.731480+0000", 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.47847500799999998 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.731509+0000", 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9065e-05 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.471 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.732546+0000", 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010369120000000001 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.732710+0000", 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016455500000000001 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,3,4,7", 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.732858+0000", 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014832100000000001 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.738069+0000", 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0052104159999999998 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.738084+0000", 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5009e-05 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.738125+0000", 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0972e-05 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.747918+0000", 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0097930029999999998 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.747999+0000", 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0743999999999998e-05 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.748294+0000", 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029525200000000003 2024-03-24T12:33:02.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.475 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.475 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.475 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:26.748457+0000", 2024-03-24T12:33:02.475 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016266499999999999 2024-03-24T12:33:02.475 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.475 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.475 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.475 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:02.476 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:125070 7.18 7:1a6808b3:::benchmark_data_smithi029_84282_object24307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-03-24T12:33:02.476 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:27.115503+0000", 2024-03-24T12:33:02.476 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 155.26102768300001, 2024-03-24T12:33:02.476 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.398984118, 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 125070 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.115503+0000", 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.115503+0000", 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.115505+0000", 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5799999999999999e-06 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.115511+0000", 2024-03-24T12:33:02.478 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2600000000000002e-06 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.115523+0000", 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2106000000000001e-05 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.115532+0000", 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7260000000000008e-06 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.479 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.119419+0000", 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0038875680000000001 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.119577+0000", 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000157839 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.122668+0000", 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00309027 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.480 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.498950+0000", 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.37628276300000002 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.499360+0000", 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000409453 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.499742+0000", 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00038257200000000002 2024-03-24T12:33:02.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.512824+0000", 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.013081485 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.512964+0000", 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014006700000000001 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:27.514487+0000", 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001523429 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:150439 7.18 7:1ab62ec3:::benchmark_data_smithi029_84282_object49676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:07.038854+0000", 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 115.337676757, 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.46721066100000003, 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.483 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 150439 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.038854+0000", 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.038854+0000", 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.484 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.038855+0000", 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.389e-06 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.038861+0000", 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5990000000000001e-06 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.038873+0000", 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1435e-05 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.485 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.038882+0000", 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3389999999999999e-06 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.046967+0000", 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0080850890000000002 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.047106+0000", 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000138648 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.486 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.047505+0000", 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00039975299999999998 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.050396+0000", 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028903790000000002 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.052459+0000", 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020628529999999999 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.052483+0000", 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4644999999999999e-05 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.505801+0000", 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45331753600000002 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.505880+0000", 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8747999999999999e-05 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.506065+0000", 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000185248 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:150442 7.18 7:1d426019:::benchmark_data_smithi029_84282_object49679:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:07.044438+0000", 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 115.33209241500001, 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.46190901000000001, 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.489 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 150442 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044438+0000", 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044438+0000", 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044440+0000", 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.302e-06 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044445+0000", 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0679999999999996e-06 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044457+0000", 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1861e-05 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.491 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044501+0000", 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4251e-05 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.050428+0000", 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0059266880000000003 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.050562+0000", 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000134413 2024-03-24T12:33:02.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.050923+0000", 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036071900000000001 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.292531+0000", 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.241608404 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.292575+0000", 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3581000000000002e-05 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.292621+0000", 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6653999999999998e-05 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.506115+0000", 2024-03-24T12:33:02.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.213493881 2024-03-24T12:33:02.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.506172+0000", 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6721000000000001e-05 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.506347+0000", 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017546700000000001 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.529 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:150452 7.18 7:1fc35602:::benchmark_data_smithi029_84282_object49689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:07.049631+0000", 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 115.326899674, 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45716385999999998, 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 150452 2024-03-24T12:33:02.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.049631+0000", 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.049631+0000", 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.049640+0000", 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5250000000000005e-06 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.049644+0000", 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.69e-06 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.049653+0000", 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9539999999999993e-06 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.532 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.049660+0000", 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4590000000000003e-06 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.289340+0000", 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.239679853 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.289526+0000", 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000186582 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,6,7", 2024-03-24T12:33:02.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.290030+0000", 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00050422499999999996 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.299136+0000", 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0091051970000000006 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.299302+0000", 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016612500000000001 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.299352+0000", 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0154000000000001e-05 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.506409+0000", 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.207056777 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.506502+0000", 2024-03-24T12:33:02.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2935000000000004e-05 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.506795+0000", 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000293384 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:163915 7.e 7:7462b583:::benchmark_data_smithi029_84282_object63152:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:27.336481+0000", 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 95.040049710999995, 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.494139946, 2024-03-24T12:33:02.536 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 163915 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.336481+0000", 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.336481+0000", 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.336482+0000", 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0500000000000002e-07 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.336486+0000", 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3030000000000001e-06 2024-03-24T12:33:02.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.336496+0000", 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0168e-05 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.336504+0000", 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5839999999999997e-06 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.342726+0000", 2024-03-24T12:33:02.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0062222759999999997 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.343093+0000", 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036665100000000001 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.343212+0000", 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000119261 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.709407+0000", 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.36619454800000001 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.716733+0000", 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0073261209999999997 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.830408+0000", 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.113675584 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:33:02.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.830564+0000", 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000155155 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:27.830621+0000", 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.749e-05 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:02.542 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:33:02.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:33:02.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:33:02.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:33:02.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:33:02.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:33:02.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:33:02.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:33:02.543 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:33:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:33:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:33:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:33:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:33:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:33:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:33:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:33:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:33:02.585 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:02.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:02.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 7 16 19270 19254 171.896 149.375 0.00840583 0.00581073 2024-03-24T12:33:02.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:02.729 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:02.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:02.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:02.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:02.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:02.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:02.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:03.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:03.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:03.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:03.143 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:03.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:03.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:03.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:03.284 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:03.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: 5.19 scrub starts 2024-03-24T12:33:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: 5.19 scrub ok 2024-03-24T12:33:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:33:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:33:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:33:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:33:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T12:33:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: 5.1 scrub starts 2024-03-24T12:33:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: 5.1 scrub ok 2024-03-24T12:33:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: pgmap v276: 148 pgs: 148 active+clean; 820 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-03-24T12:33:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:33:03.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:03 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:33:03.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:03.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: 5.19 scrub starts 2024-03-24T12:33:03.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: 5.19 scrub ok 2024-03-24T12:33:03.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:33:03.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:33:03.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:33:03.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:33:03.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T12:33:03.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: 5.1 scrub starts 2024-03-24T12:33:03.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: 5.1 scrub ok 2024-03-24T12:33:03.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: pgmap v276: 148 pgs: 148 active+clean; 820 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-03-24T12:33:03.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:33:03.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:33:03.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: 5.19 scrub starts 2024-03-24T12:33:03.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: 5.19 scrub ok 2024-03-24T12:33:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:33:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:33:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:33:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:33:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: osdmap e168: 8 total, 8 up, 8 in 2024-03-24T12:33:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: 5.1 scrub starts 2024-03-24T12:33:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: 5.1 scrub ok 2024-03-24T12:33:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: pgmap v276: 148 pgs: 148 active+clean; 820 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 141 MiB/s wr, 2.26k op/s 2024-03-24T12:33:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:33:03.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:03 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:33:03.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:03.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:03.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:03.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:03.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:03.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:03.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:03.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:03.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 8 16 21779 21763 170.009 156.812 0.00897884 0.00587589 2024-03-24T12:33:03.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:03.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:03.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:03.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:03.834 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:03.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:03.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:03.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:03.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:03.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:04.077 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:04.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:04.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:04.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:04.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:04.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:04.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:04.369 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:04.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:04 smithi190 ceph-mon[33748]: 4.7 scrub starts 2024-03-24T12:33:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:04 smithi190 ceph-mon[33748]: 4.7 scrub ok 2024-03-24T12:33:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:04 smithi190 ceph-mon[33748]: 6.15 scrub starts 2024-03-24T12:33:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:04 smithi190 ceph-mon[33748]: 6.15 scrub ok 2024-03-24T12:33:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:04 smithi029 ceph-mon[41500]: 4.7 scrub starts 2024-03-24T12:33:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:04 smithi029 ceph-mon[41500]: 4.7 scrub ok 2024-03-24T12:33:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:04 smithi029 ceph-mon[41500]: 6.15 scrub starts 2024-03-24T12:33:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:04 smithi029 ceph-mon[41500]: 6.15 scrub ok 2024-03-24T12:33:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:04 smithi029 ceph-mon[29919]: 4.7 scrub starts 2024-03-24T12:33:04.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:04 smithi029 ceph-mon[29919]: 4.7 scrub ok 2024-03-24T12:33:04.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:04 smithi029 ceph-mon[29919]: 6.15 scrub starts 2024-03-24T12:33:04.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:04 smithi029 ceph-mon[29919]: 6.15 scrub ok 2024-03-24T12:33:04.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:04.507 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:04.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:04.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:04.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:04.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:04.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:04.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 9 16 23765 23749 164.909 124.125 0.0019012 0.00604698 2024-03-24T12:33:04.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:04.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:04.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:04.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:04.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:04.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:05.010 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:05.042 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:05.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:05.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:05.182 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:05.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:05.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:05.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:05.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:05.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:05 smithi190 ceph-mon[33748]: 4.12 scrub starts 2024-03-24T12:33:05.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:05 smithi190 ceph-mon[33748]: 4.12 scrub ok 2024-03-24T12:33:05.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:05 smithi190 ceph-mon[33748]: 6.1d scrub starts 2024-03-24T12:33:05.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:05 smithi190 ceph-mon[33748]: 6.1d scrub ok 2024-03-24T12:33:05.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:05 smithi190 ceph-mon[33748]: pgmap v277: 148 pgs: 148 active+clean; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 928 B/s rd, 145 MiB/s wr, 2.32k op/s 2024-03-24T12:33:05.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:05 smithi190 ceph-mon[33748]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T12:33:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[29919]: 4.12 scrub starts 2024-03-24T12:33:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[29919]: 4.12 scrub ok 2024-03-24T12:33:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[29919]: 6.1d scrub starts 2024-03-24T12:33:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[29919]: 6.1d scrub ok 2024-03-24T12:33:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[29919]: pgmap v277: 148 pgs: 148 active+clean; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 928 B/s rd, 145 MiB/s wr, 2.32k op/s 2024-03-24T12:33:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[29919]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T12:33:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[41500]: 4.12 scrub starts 2024-03-24T12:33:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[41500]: 4.12 scrub ok 2024-03-24T12:33:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[41500]: 6.1d scrub starts 2024-03-24T12:33:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[41500]: 6.1d scrub ok 2024-03-24T12:33:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[41500]: pgmap v277: 148 pgs: 148 active+clean; 1.1 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 928 B/s rd, 145 MiB/s wr, 2.32k op/s 2024-03-24T12:33:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:05 smithi029 ceph-mon[41500]: osdmap e169: 8 total, 8 up, 8 in 2024-03-24T12:33:05.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:05.448 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:05.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:05.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:05.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:05.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:05.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 10 16 25177 25161 157.242 88.25 0.0130464 0.00635262 2024-03-24T12:33:05.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:05.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:05.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:05.724 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:05.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:05.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:05.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:05.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:05.974 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:05.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:05.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:06.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:06.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:06.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:06.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:06.236 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:06.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:06.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:06.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:06.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:06.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:06.497 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:06.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:06.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:06.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:06.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:06.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:06.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 11 16 26551 26535 150.754 85.875 0.00357025 0.00662872 2024-03-24T12:33:06.735 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:06.775 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:06.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:06.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:06.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:07.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:07.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:07.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:07.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:07.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:07.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:07.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:07.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:07.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:07.323 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:07.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:07.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:07.469 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:07.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:07.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:07.600 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:07.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:07.675 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 12 16 28705 28689 149.408 134.625 0.00161007 0.00664015 2024-03-24T12:33:07.702 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:07.732 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:07.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:07.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:07.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:07.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:07.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:07.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:07.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:33:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:33:07.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:07 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:33:07.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:07 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:33:07.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:07 smithi190 ceph-mon[33748]: 5.14 scrub starts 2024-03-24T12:33:07.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:07 smithi190 ceph-mon[33748]: 5.14 scrub ok 2024-03-24T12:33:07.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:07 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:33:07.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:07 smithi190 ceph-mon[33748]: 3.13 scrub ok 2024-03-24T12:33:07.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:07 smithi190 ceph-mon[33748]: pgmap v279: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-24T12:33:07.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:33:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:33:07.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:33:07.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:33:07.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[41500]: 5.14 scrub starts 2024-03-24T12:33:07.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[41500]: 5.14 scrub ok 2024-03-24T12:33:07.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:33:07.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[41500]: 3.13 scrub ok 2024-03-24T12:33:07.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[41500]: pgmap v279: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-24T12:33:07.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:33:07.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:33:07.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[29919]: 5.14 scrub starts 2024-03-24T12:33:07.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[29919]: 5.14 scrub ok 2024-03-24T12:33:07.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:33:07.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[29919]: 3.13 scrub ok 2024-03-24T12:33:07.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:07 smithi029 ceph-mon[29919]: pgmap v279: 148 pgs: 148 active+clean; 1.3 GiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-24T12:33:07.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:08.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:08.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:08.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:08.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:08.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:08.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:08.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:08.280 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:08.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:08.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:08.423 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:08.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:08.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:08.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:08.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:08.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:08.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:08.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 5.b scrub starts 2024-03-24T12:33:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 5.b scrub ok 2024-03-24T12:33:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 4.1d scrub starts 2024-03-24T12:33:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 4.1d scrub ok 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 4.2 scrub starts 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 4.2 scrub ok 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 3.c scrub starts 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 3.c scrub ok 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 3.8 scrub starts 2024-03-24T12:33:08.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[41500]: 3.8 scrub ok 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 5.b scrub starts 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 5.b scrub ok 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 4.1d scrub starts 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 4.1d scrub ok 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 4.2 scrub starts 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 4.2 scrub ok 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 3.c scrub starts 2024-03-24T12:33:08.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 3.c scrub ok 2024-03-24T12:33:08.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 3.8 scrub starts 2024-03-24T12:33:08.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:08 smithi029 ceph-mon[29919]: 3.8 scrub ok 2024-03-24T12:33:08.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 13 16 30437 30421 146.241 108.25 0.00766334 0.00683219 2024-03-24T12:33:08.695 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:08.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:08.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:08.835 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:08.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:08.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 5.b scrub starts 2024-03-24T12:33:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 5.b scrub ok 2024-03-24T12:33:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 4.1d scrub starts 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 4.1d scrub ok 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 4.2 scrub starts 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 4.2 scrub ok 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 3.c scrub starts 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 3.c scrub ok 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 3.8 scrub starts 2024-03-24T12:33:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:08 smithi190 ceph-mon[33748]: 3.8 scrub ok 2024-03-24T12:33:08.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:08.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:08.974 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:08.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:09.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:09.115 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:09.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:09.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:09.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:09.250 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:09.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:09.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:09.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:09.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:09.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:09.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:09.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:09.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:09.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:09.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:09.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:09.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:09.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:09.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 14 16 32051 32035 143 100.875 0.00219241 0.0069259 2024-03-24T12:33:09.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:09.791 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:09.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:09.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:33:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:33:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: 3.9 scrub starts 2024-03-24T12:33:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: 3.9 scrub ok 2024-03-24T12:33:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: 5.4 scrub starts 2024-03-24T12:33:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: 5.4 scrub ok 2024-03-24T12:33:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: pgmap v280: 148 pgs: 148 active+clean; 1.6 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-24T12:33:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: 4.15 scrub starts 2024-03-24T12:33:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: 4.15 scrub ok 2024-03-24T12:33:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: 3.1e scrub starts 2024-03-24T12:33:09.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:09 smithi190 ceph-mon[33748]: 3.1e scrub ok 2024-03-24T12:33:09.893 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:33:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:33:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: 3.9 scrub starts 2024-03-24T12:33:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: 3.9 scrub ok 2024-03-24T12:33:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: 5.4 scrub starts 2024-03-24T12:33:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: 5.4 scrub ok 2024-03-24T12:33:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: pgmap v280: 148 pgs: 148 active+clean; 1.6 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-24T12:33:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: 4.15 scrub starts 2024-03-24T12:33:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: 4.15 scrub ok 2024-03-24T12:33:09.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: 3.1e scrub starts 2024-03-24T12:33:09.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[41500]: 3.1e scrub ok 2024-03-24T12:33:09.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:33:09.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:33:09.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: 3.9 scrub starts 2024-03-24T12:33:09.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: 3.9 scrub ok 2024-03-24T12:33:09.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: 5.4 scrub starts 2024-03-24T12:33:09.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: 5.4 scrub ok 2024-03-24T12:33:09.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: pgmap v280: 148 pgs: 148 active+clean; 1.6 GiB data, 6.2 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 151 MiB/s wr, 2.42k op/s 2024-03-24T12:33:09.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: 4.15 scrub starts 2024-03-24T12:33:09.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: 4.15 scrub ok 2024-03-24T12:33:09.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: 3.1e scrub starts 2024-03-24T12:33:09.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:09 smithi029 ceph-mon[29919]: 3.1e scrub ok 2024-03-24T12:33:09.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:09.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:09.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:10.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:10.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:10.060 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:10.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:10.161 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:10.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:10.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:10.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:10.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:10.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:10.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:10.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:10.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:10.449 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:10.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:10.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:10.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:10.574 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:10.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:10.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:10.670 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 15 16 33087 33071 137.783 64.75 0.3658 0.00715523 2024-03-24T12:33:10.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:10.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:10.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:10.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:10.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:10.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:10.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:10.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:10.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:10.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:10.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:11.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:11.110 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:11.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:11.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:11.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[29919]: 6.4 scrub starts 2024-03-24T12:33:11.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[29919]: 6.4 scrub ok 2024-03-24T12:33:11.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[29919]: 5.6 scrub starts 2024-03-24T12:33:11.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[29919]: 5.6 scrub ok 2024-03-24T12:33:11.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[29919]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T12:33:11.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[29919]: 5.7 scrub starts 2024-03-24T12:33:11.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[41500]: 6.4 scrub starts 2024-03-24T12:33:11.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[41500]: 6.4 scrub ok 2024-03-24T12:33:11.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[41500]: 5.6 scrub starts 2024-03-24T12:33:11.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[41500]: 5.6 scrub ok 2024-03-24T12:33:11.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[41500]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T12:33:11.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:10 smithi029 ceph-mon[41500]: 5.7 scrub starts 2024-03-24T12:33:11.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:11.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:11.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:11.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:11.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:11.365 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:11.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:11.366 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:11.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:10 smithi190 ceph-mon[33748]: 6.4 scrub starts 2024-03-24T12:33:11.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:10 smithi190 ceph-mon[33748]: 6.4 scrub ok 2024-03-24T12:33:11.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:10 smithi190 ceph-mon[33748]: 5.6 scrub starts 2024-03-24T12:33:11.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:10 smithi190 ceph-mon[33748]: 5.6 scrub ok 2024-03-24T12:33:11.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:10 smithi190 ceph-mon[33748]: osdmap e170: 8 total, 8 up, 8 in 2024-03-24T12:33:11.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:10 smithi190 ceph-mon[33748]: 5.7 scrub starts 2024-03-24T12:33:11.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:11.509 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:11.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:11.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:11.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:11.634 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:11.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:11.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 16 16 34523 34507 134.781 89.75 0.0019692 0.00740501 2024-03-24T12:33:11.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:11.759 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:11.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:11.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:11.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:11.900 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:11.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:11.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:12.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:12.044 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:12.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:12.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:12.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:12.169 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:12.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:12.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:12.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:12.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:12.295 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:12.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:11 smithi190 ceph-mon[33748]: 5.7 scrub ok 2024-03-24T12:33:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:11 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:33:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:11 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:33:12.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:11 smithi190 ceph-mon[33748]: pgmap v282: 148 pgs: 148 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-03-24T12:33:12.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:11 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:33:12.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:11 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:33:12.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:11 smithi190 ceph-mon[33748]: 3.5 scrub starts 2024-03-24T12:33:12.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:11 smithi190 ceph-mon[33748]: 3.5 scrub ok 2024-03-24T12:33:12.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:11 smithi190 ceph-mon[33748]: 3.11 scrub starts 2024-03-24T12:33:12.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:11 smithi190 ceph-mon[33748]: 3.11 scrub ok 2024-03-24T12:33:12.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:12.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[29919]: 5.7 scrub ok 2024-03-24T12:33:12.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:33:12.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:33:12.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[29919]: pgmap v282: 148 pgs: 148 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[29919]: 3.5 scrub starts 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[29919]: 3.5 scrub ok 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[29919]: 3.11 scrub starts 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[29919]: 3.11 scrub ok 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[41500]: 5.7 scrub ok 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[41500]: pgmap v282: 148 pgs: 148 active+clean; 1.7 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 118 MiB/s wr, 1.88k op/s 2024-03-24T12:33:12.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:33:12.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:33:12.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[41500]: 3.5 scrub starts 2024-03-24T12:33:12.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[41500]: 3.5 scrub ok 2024-03-24T12:33:12.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[41500]: 3.11 scrub starts 2024-03-24T12:33:12.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:11 smithi029 ceph-mon[41500]: 3.11 scrub ok 2024-03-24T12:33:12.436 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:12.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:12.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:12.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:12.568 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:12.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:12.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:12.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 17 16 35663 35647 131.043 71.25 0.0760286 0.00760153 2024-03-24T12:33:12.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:12.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:12.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:12.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:12.847 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:12.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:12.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:12.988 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:12.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:12.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:13.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:13.130 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:13.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:13.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:13.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:13.272 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:13.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:13.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:13.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:12 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:33:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:12 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:33:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:12 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:33:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:12 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:33:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:13.410 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:13.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:12 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:33:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:12 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:33:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:12 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:33:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:12 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:33:13.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:12 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:33:13.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:12 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:33:13.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:12 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:33:13.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:12 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:33:13.511 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:13.535 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:13.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:13.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:13.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:13.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:13.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:13.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 18 16 37242 37226 129.245 98.6875 0.00216633 0.00765844 2024-03-24T12:33:13.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:13.787 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:13.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:13.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:13.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:13.924 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:13.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:13.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:14.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:14.060 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:14.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:14.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:14.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:14.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:14.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:14.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:14.312 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:14.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:14.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:33:14.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:33:14.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:33:14.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:33:14.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[29919]: pgmap v283: 148 pgs: 148 active+clean; 2.0 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-24T12:33:14.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:33:14.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:33:14.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[29919]: 6.9 scrub starts 2024-03-24T12:33:14.355 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[29919]: 6.9 scrub ok 2024-03-24T12:33:14.355 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:33:14.355 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:33:14.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:33:14.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:33:14.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[41500]: pgmap v283: 148 pgs: 148 active+clean; 2.0 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-24T12:33:14.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:33:14.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:33:14.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[41500]: 6.9 scrub starts 2024-03-24T12:33:14.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:13 smithi029 ceph-mon[41500]: 6.9 scrub ok 2024-03-24T12:33:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:13 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:33:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:13 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:33:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:13 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:33:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:13 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:33:14.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:13 smithi190 ceph-mon[33748]: pgmap v283: 148 pgs: 148 active+clean; 2.0 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.72k op/s 2024-03-24T12:33:14.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:13 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:33:14.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:13 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:33:14.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:13 smithi190 ceph-mon[33748]: 6.9 scrub starts 2024-03-24T12:33:14.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:13 smithi190 ceph-mon[33748]: 6.9 scrub ok 2024-03-24T12:33:14.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:14.449 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:14.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:14.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:14.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:14.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:14.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 19 16 38450 38434 126.416 75.5 0.00227829 0.00789761 2024-03-24T12:33:14.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:14.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:14.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:14.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:14.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:14.842 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:14.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:14.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:14.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:14.987 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:14.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:14.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:15.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:15.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:15.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:15.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:15.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:15.249 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:15.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:15.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:15.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:15 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:33:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:15 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:33:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:15 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:33:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:15 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:33:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:15 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:33:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:15 smithi190 ceph-mon[33748]: 6.18 scrub starts 2024-03-24T12:33:15.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:15 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:33:15.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:15 smithi190 ceph-mon[33748]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T12:33:15.388 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:15.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:15.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:15.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:33:15.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:33:15.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:33:15.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:33:15.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:33:15.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[29919]: 6.18 scrub starts 2024-03-24T12:33:15.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:33:15.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[29919]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T12:33:15.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:33:15.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:33:15.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:33:15.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:33:15.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:33:15.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[41500]: 6.18 scrub starts 2024-03-24T12:33:15.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:33:15.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:15 smithi029 ceph-mon[41500]: osdmap e171: 8 total, 8 up, 8 in 2024-03-24T12:33:15.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:15.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:15.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:15.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:15.670 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:15.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:15.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:33:15.675720+0000 min lat: 0.00120017 max lat: 0.370401 avg lat: 0.00801552 2024-03-24T12:33:15.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:33:15.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 20 15 39913 39898 124.67 91.5 0.00600652 0.00801552 2024-03-24T12:33:15.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:15.796 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:15.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:15.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:15.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:15.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:16.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:16.062 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:16.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:16.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:16.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:16.198 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:16.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:16.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:16.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:16.329 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:16.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:16.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:16 smithi190 ceph-mon[33748]: 6.18 scrub ok 2024-03-24T12:33:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:16 smithi190 ceph-mon[33748]: pgmap v284: 148 pgs: 148 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-24T12:33:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:16 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:33:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:16 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:33:16.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:16 smithi190 ceph-mon[33748]: 5.1e scrub starts 2024-03-24T12:33:16.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:16 smithi190 ceph-mon[33748]: 5.1e scrub ok 2024-03-24T12:33:16.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:16 smithi190 ceph-mon[33748]: 3.7 scrub starts 2024-03-24T12:33:16.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:16 smithi190 ceph-mon[33748]: 3.7 scrub ok 2024-03-24T12:33:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[29919]: 6.18 scrub ok 2024-03-24T12:33:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[29919]: pgmap v284: 148 pgs: 148 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-24T12:33:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:33:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:33:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[29919]: 5.1e scrub starts 2024-03-24T12:33:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[29919]: 5.1e scrub ok 2024-03-24T12:33:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[29919]: 3.7 scrub starts 2024-03-24T12:33:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[29919]: 3.7 scrub ok 2024-03-24T12:33:16.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[41500]: 6.18 scrub ok 2024-03-24T12:33:16.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[41500]: pgmap v284: 148 pgs: 148 active+clean; 2.2 GiB data, 7.8 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 111 MiB/s wr, 1.78k op/s 2024-03-24T12:33:16.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:33:16.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:33:16.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[41500]: 5.1e scrub starts 2024-03-24T12:33:16.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[41500]: 5.1e scrub ok 2024-03-24T12:33:16.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[41500]: 3.7 scrub starts 2024-03-24T12:33:16.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:16 smithi029 ceph-mon[41500]: 3.7 scrub ok 2024-03-24T12:33:16.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:16.470 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:16.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:16.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:16.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:16.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:16.597 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:16.597 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:16.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 21 16 40871 40855 121.581 59.8125 0.00272037 0.00820269 2024-03-24T12:33:16.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:16.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:16.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:16.864 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:16.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:16.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:16.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:16.996 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:16.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:16.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:17.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:17.136 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:17.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:17.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:17.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:17.261 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:17.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:17.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:17 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:33:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:17 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:33:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:17 smithi190 ceph-mon[33748]: pgmap v286: 148 pgs: 148 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:33:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:17.385 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:17.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:17.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:17 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:33:17.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:17 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:33:17.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:17 smithi029 ceph-mon[29919]: pgmap v286: 148 pgs: 148 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:33:17.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:17.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:17 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:33:17.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:17 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:33:17.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:17 smithi029 ceph-mon[41500]: pgmap v286: 148 pgs: 148 active+clean; 2.2 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:33:17.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:17.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:17.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:17.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:17.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:17.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:17.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:17.662 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:17.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:17.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 22 16 42118 42102 119.597 77.9375 0.00947952 0.00835546 2024-03-24T12:33:17.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:17.802 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:17.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:17.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:17.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:33:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:33:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:33:17.927 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:17.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:17.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:18.008 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:33:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:33:18.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:18.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:18.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:18.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:18.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:18.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:18.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:18.340 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:18.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:18 smithi190 ceph-mon[33748]: 5.a scrub starts 2024-03-24T12:33:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:18 smithi190 ceph-mon[33748]: 5.a scrub ok 2024-03-24T12:33:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:18 smithi190 ceph-mon[33748]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T12:33:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:18 smithi029 ceph-mon[41500]: 5.a scrub starts 2024-03-24T12:33:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:18 smithi029 ceph-mon[41500]: 5.a scrub ok 2024-03-24T12:33:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:18 smithi029 ceph-mon[41500]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T12:33:18.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:18.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:18 smithi029 ceph-mon[29919]: 5.a scrub starts 2024-03-24T12:33:18.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:18 smithi029 ceph-mon[29919]: 5.a scrub ok 2024-03-24T12:33:18.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:18 smithi029 ceph-mon[29919]: osdmap e172: 8 total, 8 up, 8 in 2024-03-24T12:33:18.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:18.441 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:18.483 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:18.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:18.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:18.584 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:18.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:18.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:18.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 23 16 43520 43504 118.206 87.625 0.00215158 0.0084491 2024-03-24T12:33:18.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:18.749 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:18.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:18.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:18.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:18.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:18.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:18.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:19.000 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:19.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:19.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:19.124 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:19.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:19.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:19.262 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:19.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:19.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:19.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:19.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:19 smithi190 ceph-mon[33748]: 3.14 scrub starts 2024-03-24T12:33:19.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:19 smithi190 ceph-mon[33748]: 3.14 scrub ok 2024-03-24T12:33:19.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:19 smithi190 ceph-mon[33748]: 4.b scrub starts 2024-03-24T12:33:19.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:19 smithi190 ceph-mon[33748]: 4.b scrub ok 2024-03-24T12:33:19.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:19 smithi190 ceph-mon[33748]: 6.7 scrub starts 2024-03-24T12:33:19.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:19 smithi190 ceph-mon[33748]: pgmap v288: 148 pgs: 148 active+clean; 2.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-24T12:33:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:19.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:19.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[41500]: 3.14 scrub starts 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[41500]: 3.14 scrub ok 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[41500]: 4.b scrub starts 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[41500]: 4.b scrub ok 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[41500]: 6.7 scrub starts 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[41500]: pgmap v288: 148 pgs: 148 active+clean; 2.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[29919]: 3.14 scrub starts 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[29919]: 3.14 scrub ok 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[29919]: 4.b scrub starts 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[29919]: 4.b scrub ok 2024-03-24T12:33:19.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[29919]: 6.7 scrub starts 2024-03-24T12:33:19.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:19 smithi029 ceph-mon[29919]: pgmap v288: 148 pgs: 148 active+clean; 2.5 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-24T12:33:19.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:19.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:19.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:19.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:19.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:19.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:19.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:19.671 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 24 16 44119 44103 114.841 37.4375 0.00635061 0.00866041 2024-03-24T12:33:19.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:19.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:19.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:19.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:19.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:19.911 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:19.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:19.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:20.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:20.055 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:20.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:20.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:20.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:20.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:20.192 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:20.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:20.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:20.316 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:20.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:20.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 6.7 scrub ok 2024-03-24T12:33:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 3.1a scrub starts 2024-03-24T12:33:20.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 3.1a scrub ok 2024-03-24T12:33:20.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 5.17 scrub starts 2024-03-24T12:33:20.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 5.17 scrub ok 2024-03-24T12:33:20.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:33:20.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:33:20.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:33:20.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:33:20.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 4.3 scrub starts 2024-03-24T12:33:20.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: 4.3 scrub ok 2024-03-24T12:33:20.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:20 smithi190 ceph-mon[33748]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T12:33:20.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 6.7 scrub ok 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 3.1a scrub starts 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 3.1a scrub ok 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 5.17 scrub starts 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 5.17 scrub ok 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 4.3 scrub starts 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: 4.3 scrub ok 2024-03-24T12:33:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[29919]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 6.7 scrub ok 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 3.1a scrub starts 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 3.1a scrub ok 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 5.17 scrub starts 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 5.17 scrub ok 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 4.3 scrub starts 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: 4.3 scrub ok 2024-03-24T12:33:20.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:20 smithi029 ceph-mon[41500]: osdmap e173: 8 total, 8 up, 8 in 2024-03-24T12:33:20.442 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:20.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:20.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:20.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:20.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:20.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:20.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:20.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 25 16 45114 45098 112.734 62.1875 0.0224566 0.00886306 2024-03-24T12:33:20.700 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:20.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:20.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:20.846 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:20.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:20.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:20.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:20.970 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:20.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:21.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:21.095 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:21.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:21.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:21.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:21 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:33:21.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:21 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:33:21.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:21 smithi029 ceph-mon[41500]: 6.c scrub starts 2024-03-24T12:33:21.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:21 smithi029 ceph-mon[41500]: 6.c scrub ok 2024-03-24T12:33:21.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:21 smithi029 ceph-mon[41500]: pgmap v290: 148 pgs: 148 active+clean; 2.6 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T12:33:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:21 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:33:21.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:21 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:33:21.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:21 smithi029 ceph-mon[29919]: 6.c scrub starts 2024-03-24T12:33:21.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:21 smithi029 ceph-mon[29919]: 6.c scrub ok 2024-03-24T12:33:21.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:21 smithi029 ceph-mon[29919]: pgmap v290: 148 pgs: 148 active+clean; 2.6 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T12:33:21.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:21.234 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:21.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:21.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:21.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:21.373 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:21.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:21 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:33:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:21 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:33:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:21 smithi190 ceph-mon[33748]: 6.c scrub starts 2024-03-24T12:33:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:21 smithi190 ceph-mon[33748]: 6.c scrub ok 2024-03-24T12:33:21.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:21 smithi190 ceph-mon[33748]: pgmap v290: 148 pgs: 148 active+clean; 2.6 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.08k op/s 2024-03-24T12:33:21.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:21.502 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:21.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:21.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:21.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:21.644 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:21.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:21.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:21.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 26 16 46383 46367 111.448 79.3125 0.00935785 0.00896763 2024-03-24T12:33:21.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:21.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:21.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:21.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:21.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:21.915 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:21.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:22.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:22.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:22.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:22.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:22.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:22.193 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:22.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:22.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:22.336 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:22.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:22.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[29919]: 4.1 scrub starts 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[29919]: 4.1 scrub ok 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[29919]: 6.b scrub starts 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[29919]: 6.b scrub ok 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[29919]: 5.12 scrub starts 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[29919]: 5.12 scrub ok 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[41500]: 4.1 scrub starts 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[41500]: 4.1 scrub ok 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[41500]: 6.b scrub starts 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[41500]: 6.b scrub ok 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[41500]: 5.12 scrub starts 2024-03-24T12:33:22.353 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:22 smithi029 ceph-mon[41500]: 5.12 scrub ok 2024-03-24T12:33:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:22 smithi190 ceph-mon[33748]: 4.1 scrub starts 2024-03-24T12:33:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:22 smithi190 ceph-mon[33748]: 4.1 scrub ok 2024-03-24T12:33:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:22 smithi190 ceph-mon[33748]: 6.b scrub starts 2024-03-24T12:33:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:22 smithi190 ceph-mon[33748]: 6.b scrub ok 2024-03-24T12:33:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:22 smithi190 ceph-mon[33748]: 5.12 scrub starts 2024-03-24T12:33:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:22 smithi190 ceph-mon[33748]: 5.12 scrub ok 2024-03-24T12:33:22.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:22.486 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:22.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:22.587 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:22.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:22.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:22.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:22.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 27 16 48018 48002 111.105 102.188 0.0061805 0.00899573 2024-03-24T12:33:22.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:22.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:22.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:22.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:22.900 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:22.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:23.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:23.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T12:33:23.051 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:23.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:23.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:23.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:23.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:23.184 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:23.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:23.253 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[41500]: 6.19 scrub starts 2024-03-24T12:33:23.253 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[41500]: 6.19 scrub ok 2024-03-24T12:33:23.253 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[41500]: 3.b scrub starts 2024-03-24T12:33:23.253 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[41500]: 3.b scrub ok 2024-03-24T12:33:23.253 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:33:23.253 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:33:23.253 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:33:23.253 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:33:23.253 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[41500]: pgmap v291: 148 pgs: 148 active+clean; 2.8 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:33:23.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[29919]: 6.19 scrub starts 2024-03-24T12:33:23.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[29919]: 6.19 scrub ok 2024-03-24T12:33:23.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[29919]: 3.b scrub starts 2024-03-24T12:33:23.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[29919]: 3.b scrub ok 2024-03-24T12:33:23.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:33:23.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:33:23.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:33:23.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:33:23.254 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:23 smithi029 ceph-mon[29919]: pgmap v291: 148 pgs: 148 active+clean; 2.8 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:33:23.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:23.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:23.326 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:23.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:23 smithi190 ceph-mon[33748]: 6.19 scrub starts 2024-03-24T12:33:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:23 smithi190 ceph-mon[33748]: 6.19 scrub ok 2024-03-24T12:33:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:23 smithi190 ceph-mon[33748]: 3.b scrub starts 2024-03-24T12:33:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:23 smithi190 ceph-mon[33748]: 3.b scrub ok 2024-03-24T12:33:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:23 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:33:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:23 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:33:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:23 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:33:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:23 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:33:23.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:23 smithi190 ceph-mon[33748]: pgmap v291: 148 pgs: 148 active+clean; 2.8 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.28k op/s 2024-03-24T12:33:23.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:23.477 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:23.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:23.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:23.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:23.622 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:23.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:23.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 28 16 49644 49628 110.766 101.625 0.0114698 0.00902234 2024-03-24T12:33:23.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:23.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:23.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:23.797 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:33:23.805 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:33:23.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:23.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:23.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:24.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:24.091 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:24.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:24.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:24.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:24.213 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:24 smithi029 ceph-mon[41500]: 3.1c scrub starts 2024-03-24T12:33:24.213 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:24 smithi029 ceph-mon[41500]: 3.1c scrub ok 2024-03-24T12:33:24.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:24 smithi029 ceph-mon[29919]: 3.1c scrub starts 2024-03-24T12:33:24.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:24 smithi029 ceph-mon[29919]: 3.1c scrub ok 2024-03-24T12:33:24.241 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:24.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:24.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:24.374 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:24.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:24.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:24.508 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:24.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:24.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:24.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:24.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:24 smithi190 ceph-mon[33748]: 3.1c scrub starts 2024-03-24T12:33:24.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:24 smithi190 ceph-mon[33748]: 3.1c scrub ok 2024-03-24T12:33:24.650 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:24.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:24.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:24.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 29 16 51194 51178 110.287 96.875 0.00629932 0.00906284 2024-03-24T12:33:24.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:24.804 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:24.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:24.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:24.905 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:24.945 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:24.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:25.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:25.092 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:25.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:25.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:25.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:33:25.164 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:33:25.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:25.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:25.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:25.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:25 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3230017180' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:33:25.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:25 smithi029 ceph-mon[41500]: 3.2 scrub starts 2024-03-24T12:33:25.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:25 smithi029 ceph-mon[41500]: 3.2 scrub ok 2024-03-24T12:33:25.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:25 smithi029 ceph-mon[41500]: pgmap v292: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-24T12:33:25.308 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:25 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3230017180' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:33:25.308 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:25 smithi029 ceph-mon[29919]: 3.2 scrub starts 2024-03-24T12:33:25.308 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:25 smithi029 ceph-mon[29919]: 3.2 scrub ok 2024-03-24T12:33:25.308 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:25 smithi029 ceph-mon[29919]: pgmap v292: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-24T12:33:25.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:25.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:25.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:25.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:25.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:25.503 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:25.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:25.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:25.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:25 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3230017180' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:33:25.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:25 smithi190 ceph-mon[33748]: 3.2 scrub starts 2024-03-24T12:33:25.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:25 smithi190 ceph-mon[33748]: 3.2 scrub ok 2024-03-24T12:33:25.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:25 smithi190 ceph-mon[33748]: pgmap v292: 148 pgs: 148 active+clean; 2.9 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.37k op/s 2024-03-24T12:33:25.646 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:25.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:25.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 30 16 52759 52743 109.871 97.8125 0.0105734 0.00909658 2024-03-24T12:33:25.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:25.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:25.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:25.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:25.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:25.919 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:25.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:25.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:26.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:26.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:26.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:26.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:26.182 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:26.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:26.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:26.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:26.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:26 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:33:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:26 smithi029 ceph-mon[41500]: Cluster is now healthy 2024-03-24T12:33:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:26 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3230017180' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:33:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:26 smithi029 ceph-mon[41500]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T12:33:26.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:26 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:33:26.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:26 smithi029 ceph-mon[29919]: Cluster is now healthy 2024-03-24T12:33:26.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:26 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3230017180' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:33:26.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:26 smithi029 ceph-mon[29919]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T12:33:26.428 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:26.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:26.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:26.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:26.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:26.593 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:26.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:26.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:26.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:26 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:33:26.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:26 smithi190 ceph-mon[33748]: Cluster is now healthy 2024-03-24T12:33:26.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:26 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3230017180' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:33:26.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:26 smithi190 ceph-mon[33748]: osdmap e174: 8 total, 8 up, 8 in 2024-03-24T12:33:26.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 31 16 54328 54312 109.49 98.0625 0.00426963 0.00912863 2024-03-24T12:33:26.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:26.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:26.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:26.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:26.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:26.869 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:26.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:26.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:26.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:27.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:27.134 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:27.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:27.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:27.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:27.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:27.409 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:27.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[29919]: 4.8 scrub starts 2024-03-24T12:33:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[29919]: 4.8 scrub ok 2024-03-24T12:33:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:33:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:33:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[29919]: 3.12 scrub starts 2024-03-24T12:33:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[29919]: 3.12 scrub ok 2024-03-24T12:33:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[29919]: pgmap v294: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-24T12:33:27.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[41500]: 4.8 scrub starts 2024-03-24T12:33:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[41500]: 4.8 scrub ok 2024-03-24T12:33:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:33:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:33:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[41500]: 3.12 scrub starts 2024-03-24T12:33:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[41500]: 3.12 scrub ok 2024-03-24T12:33:27.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-mon[41500]: pgmap v294: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-24T12:33:27.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:27.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:27 smithi190 ceph-mon[33748]: 4.8 scrub starts 2024-03-24T12:33:27.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:27 smithi190 ceph-mon[33748]: 4.8 scrub ok 2024-03-24T12:33:27.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:27 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:33:27.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:27 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:33:27.534 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:27 smithi190 ceph-mon[33748]: 3.12 scrub starts 2024-03-24T12:33:27.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:27 smithi190 ceph-mon[33748]: 3.12 scrub ok 2024-03-24T12:33:27.535 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:27 smithi190 ceph-mon[33748]: pgmap v294: 148 pgs: 148 active+clean; 3.0 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-24T12:33:27.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:27.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:27.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:27.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:27.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 32 15 55827 55812 108.998 93.75 0.00915208 0.00916997 2024-03-24T12:33:27.691 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:27.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:27.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:27.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:27.838 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:27.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:27.871 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:33:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:33:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:33:27.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:33:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:33:27.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:27.978 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:27.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:28.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:28.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:28.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:28.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:28.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:28.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:28.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:28.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:28.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:28.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:28.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:28 smithi029 ceph-mon[41500]: 3.16 scrub starts 2024-03-24T12:33:28.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:28 smithi029 ceph-mon[41500]: 3.16 scrub ok 2024-03-24T12:33:28.422 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:28.422 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:28 smithi029 ceph-mon[29919]: 3.16 scrub starts 2024-03-24T12:33:28.422 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:28 smithi029 ceph-mon[29919]: 3.16 scrub ok 2024-03-24T12:33:28.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:28.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:28.528 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:28.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:28.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:28.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:28.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:28 smithi190 ceph-mon[33748]: 3.16 scrub starts 2024-03-24T12:33:28.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:28 smithi190 ceph-mon[33748]: 3.16 scrub ok 2024-03-24T12:33:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:28.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:28.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:28.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 33 16 57381 57365 108.636 97.0625 0.01443 0.00920044 2024-03-24T12:33:28.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:28.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:28.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:28.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:28.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:29.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:29.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:29.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:29.072 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:33:29.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:29.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:29.223 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:29.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:29.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:29 smithi029 ceph-mon[41500]: 4.10 scrub starts 2024-03-24T12:33:29.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:29 smithi029 ceph-mon[41500]: 4.10 scrub ok 2024-03-24T12:33:29.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:29 smithi029 ceph-mon[41500]: pgmap v295: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T12:33:29.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:29 smithi029 ceph-mon[29919]: 4.10 scrub starts 2024-03-24T12:33:29.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:29 smithi029 ceph-mon[29919]: 4.10 scrub ok 2024-03-24T12:33:29.322 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:29 smithi029 ceph-mon[29919]: pgmap v295: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T12:33:29.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:29.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:29.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:29.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:29.513 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:29.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:29.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:29.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:29 smithi190 ceph-mon[33748]: 4.10 scrub starts 2024-03-24T12:33:29.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:29 smithi190 ceph-mon[33748]: 4.10 scrub ok 2024-03-24T12:33:29.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:29 smithi190 ceph-mon[33748]: pgmap v295: 148 pgs: 148 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T12:33:29.644 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:29.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:29.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:29.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 34 16 58822 58806 108.089 90.0625 0.00843909 0.00924674 2024-03-24T12:33:29.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:33:29.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:29.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:29.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:29.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:29.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:29.925 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:33:29.925 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:33:29.925 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.925 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.24815.0:59068 8.7 e174/158 8:e4ce6e7d:::benchmark_data_smithi029_108893_object59067:head v 174'3723, mlcod=174'3719)", 2024-03-24T12:33:29.925 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:29.918343+0000", 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0090872980000000006, 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0091177889999999994, 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.918343+0000", 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:29.926 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.918343+0000", 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.918343+0000", 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.89e-07 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.918369+0000", 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6117000000000001e-05 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.927 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.918378+0000", 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3629999999999994e-06 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.918384+0000", 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7360000000000004e-06 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.924245+0000", 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.005860105 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.928 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.924308+0000", 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3841000000000001e-05 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.24815.0:59076 8.c e174/158 8:377e365d:::benchmark_data_smithi029_108893_object59075:head v 174'3560, mlcod=174'3558)", 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:29.925498+0000", 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0019323529999999999, 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020379539999999998, 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:33:29.929 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.925498+0000", 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.925498+0000", 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:29.930 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.925499+0000", 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5799999999999998e-07 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.925528+0000", 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9598999999999999e-05 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.925545+0000", 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6439e-05 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:29.931 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.925554+0000", 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.901e-06 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.925616+0000", 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2300999999999999e-05 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.925643+0000", 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7350000000000001e-05 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:29.932 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.24815.0:59058 8.f e174/158 ondisk, result = 0)", 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:29.926848+0000", 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.00058202700000000004, 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00076171599999999996, 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.926848+0000", 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.933 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.926848+0000", 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.926849+0000", 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.16e-07 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.926855+0000", 2024-03-24T12:33:29.934 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6729999999999997e-06 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.926865+0000", 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0743e-05 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.926873+0000", 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7610000000000007e-06 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.927170+0000", 2024-03-24T12:33:29.935 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029638499999999998 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:29.927197+0000", 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7949999999999998e-05 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 3 2024-03-24T12:33:29.936 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:33:29.943 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:29.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:29.968 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:33:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:33:30.045 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:30.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:30.099 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:30.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:30.181 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[41500]: 3.18 scrub starts 2024-03-24T12:33:30.181 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[41500]: 3.18 scrub ok 2024-03-24T12:33:30.182 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:33:30.182 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:33:30.182 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[41500]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:33:30.182 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[41500]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T12:33:30.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[29919]: 3.18 scrub starts 2024-03-24T12:33:30.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[29919]: 3.18 scrub ok 2024-03-24T12:33:30.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:33:30.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:33:30.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[29919]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:33:30.182 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:30 smithi029 ceph-mon[29919]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T12:33:30.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:30.244 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:30.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:30.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:30.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:30.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:30.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:30.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:30.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:30.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:30.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:30.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:30 smithi190 ceph-mon[33748]: 3.18 scrub starts 2024-03-24T12:33:30.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:30 smithi190 ceph-mon[33748]: 3.18 scrub ok 2024-03-24T12:33:30.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:30 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:33:30.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:30 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:33:30.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:30 smithi190 ceph-mon[33748]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:33:30.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:30 smithi190 ceph-mon[33748]: osdmap e175: 8 total, 8 up, 8 in 2024-03-24T12:33:30.672 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 35 16 60137 60121 107.349 82.1875 0.0184914 0.00931011 2024-03-24T12:33:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:30.679 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:30.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:30.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:30.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:30.831 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:30.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:30.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:30.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:30.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:30.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:31.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:31.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:31.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:31.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:31.259 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:31.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:31.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T12:33:31.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:31.392 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:31.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:31.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:31 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:33:31.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:31 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:33:31.490 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:31 smithi029 ceph-mon[29919]: pgmap v297: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-24T12:33:31.490 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:31 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:33:31.490 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:31 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:33:31.490 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:31 smithi029 ceph-mon[41500]: pgmap v297: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-24T12:33:31.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:31.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:31.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:31.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:31.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:31.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:31 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:33:31.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:31 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:33:31.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:31 smithi190 ceph-mon[33748]: pgmap v297: 148 pgs: 148 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-03-24T12:33:31.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 36 16 61576 61560 106.865 89.9375 0.00447204 0.00935214 2024-03-24T12:33:31.681 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:31.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:31.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:31.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:31.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:31.830 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:31.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:31.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:31.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:31.969 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:31.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:32.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:32.111 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:32.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:32.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:32.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:32.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:32.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:32.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:32.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:32.440 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:32 smithi029 ceph-mon[29919]: 3.1f scrub starts 2024-03-24T12:33:32.440 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:32 smithi029 ceph-mon[29919]: 3.1f scrub ok 2024-03-24T12:33:32.440 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:32 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:32.440 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:32 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:32.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:32 smithi029 ceph-mon[41500]: 3.1f scrub starts 2024-03-24T12:33:32.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:32 smithi029 ceph-mon[41500]: 3.1f scrub ok 2024-03-24T12:33:32.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:32 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:32.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:32 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:32.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:32.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:32.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:32.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:32.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:32.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:32 smithi190 ceph-mon[33748]: 3.1f scrub starts 2024-03-24T12:33:32.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:32 smithi190 ceph-mon[33748]: 3.1f scrub ok 2024-03-24T12:33:32.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:32 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:32.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:32 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:32.664 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:32.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:32.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:32.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 37 16 63077 63061 106.512 93.8125 0.00785675 0.00938336 2024-03-24T12:33:32.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:32.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:32.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:32.873 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-03-24T12:33:28.515857+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3533117790,"num_objects":54100,"num_object_clones":0,"num_object_copies":162300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54100,"num_whiteouts":0,"num_read":1371,"num_read_kb":1217,"num_write":54517,"num_write_kb":3452189,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771434,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54568,"ondisk_log_size":54568,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11971412,"kb_used_data":10367056,"kb_used_omap":12,"kb_used_meta":1604275,"kb_avail":737989804,"statfs":{"total":767960285184,"available":755701559296,"internally_reserved":0,"allocated":10615865344,"data_stored":10612053973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":1642778161},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[11,4,4,8,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":785580032,"num_objects":11987,"num_object_clones":0,"num_object_copies":35961,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11987,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11987,"num_write_kb":767168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.551738"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":239,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175812+0000","last_change":"2024-03-24T12:32:55.993821+0000","last_active":"2024-03-24T12:33:25.175812+0000","last_peered":"2024-03-24T12:33:25.175812+0000","last_clean":"2024-03-24T12:33:25.175812+0000","last_became_active":"2024-03-24T12:26:09.350521+0000","last_became_peered":"2024-03-24T12:26:09.350521+0000","last_unstale":"2024-03-24T12:33:25.175812+0000","last_undegraded":"2024-03-24T12:33:25.175812+0000","last_fullsized":"2024-03-24T12:33:25.175812+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:55.993649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.993649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:18.074718+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.182751+0000","last_change":"2024-03-24T12:33:01.510002+0000","last_active":"2024-03-24T12:33:25.182751+0000","last_peered":"2024-03-24T12:33:25.182751+0000","last_clean":"2024-03-24T12:33:25.182751+0000","last_became_active":"2024-03-24T12:26:07.335517+0000","last_became_peered":"2024-03-24T12:26:07.335517+0000","last_unstale":"2024-03-24T12:33:25.182751+0000","last_undegraded":"2024-03-24T12:33:25.182751+0000","last_fullsized":"2024-03-24T12:33:25.182751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:01.509739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:01.509739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:06.241233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":236,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.165866+0000","last_change":"2024-03-24T12:29:00.447969+0000","last_active":"2024-03-24T12:33:25.165866+0000","last_peered":"2024-03-24T12:33:25.165866+0000","last_clean":"2024-03-24T12:33:25.165866+0000","last_became_active":"2024-03-24T12:26:03.170822+0000","last_became_peered":"2024-03-24T12:26:03.170822+0000","last_unstale":"2024-03-24T12:33:25.165866+0000","last_undegraded":"2024-03-24T12:33:25.165866+0000","last_fullsized":"2024-03-24T12:33:25.165866+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:00.447769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:00.447769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":264,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175642+0000","last_change":"2024-03-24T12:33:12.325849+0000","last_active":"2024-03-24T12:33:25.175642+0000","last_peered":"2024-03-24T12:33:25.175642+0000","last_clean":"2024-03-24T12:33:25.175642+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:33:25.175642+0000","last_undegraded":"2024-03-24T12:33:25.175642+0000","last_fullsized":"2024-03-24T12:33:25.175642+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:12.325694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.325694+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:40.240287+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":229,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.968685+0000","last_change":"2024-03-24T12:32:56.130518+0000","last_active":"2024-03-24T12:33:19.968685+0000","last_peered":"2024-03-24T12:33:19.968685+0000","last_clean":"2024-03-24T12:33:19.968685+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:33:19.968685+0000","last_undegraded":"2024-03-24T12:33:19.968685+0000","last_fullsized":"2024-03-24T12:33:19.968685+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:56.130373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:56.130373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:10.585684+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175663+0000","last_change":"2024-03-24T12:32:53.023292+0000","last_active":"2024-03-24T12:33:25.175663+0000","last_peered":"2024-03-24T12:33:25.175663+0000","last_clean":"2024-03-24T12:33:25.175663+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:33:25.175663+0000","last_undegraded":"2024-03-24T12:33:25.175663+0000","last_fullsized":"2024-03-24T12:33:25.175663+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:53.023160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:53.023160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:22.383712+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.968195+0000","last_change":"2024-03-24T12:33:09.263042+0000","last_active":"2024-03-24T12:33:19.968195+0000","last_peered":"2024-03-24T12:33:19.968195+0000","last_clean":"2024-03-24T12:33:19.968195+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:33:19.968195+0000","last_undegraded":"2024-03-24T12:33:19.968195+0000","last_fullsized":"2024-03-24T12:33:19.968195+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:09.262901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:09.262901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:09.802133+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"90'22","reported_seq":269,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.967376+0000","last_change":"2024-03-24T12:29:01.369083+0000","last_active":"2024-03-24T12:33:19.967376+0000","last_peered":"2024-03-24T12:33:19.967376+0000","last_clean":"2024-03-24T12:33:19.967376+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:33:19.967376+0000","last_undegraded":"2024-03-24T12:33:19.967376+0000","last_fullsized":"2024-03-24T12:33:19.967376+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:29:01.368806+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:29:01.368806+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":239,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.165577+0000","last_change":"2024-03-24T12:33:14.363693+0000","last_active":"2024-03-24T12:33:25.165577+0000","last_peered":"2024-03-24T12:33:25.165577+0000","last_clean":"2024-03-24T12:33:25.165577+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:33:25.165577+0000","last_undegraded":"2024-03-24T12:33:25.165577+0000","last_fullsized":"2024-03-24T12:33:25.165577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:14.363553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:33:14.363553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:20.233556+0000","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":235,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.161767+0000","last_change":"2024-03-24T12:32:51.275896+0000","last_active":"2024-03-24T12:33:25.161767+0000","last_peered":"2024-03-24T12:33:25.161767+0000","last_clean":"2024-03-24T12:33:25.161767+0000","last_became_active":"2024-03-24T12:26:07.331562+0000","last_became_peered":"2024-03-24T12:26:07.331562+0000","last_unstale":"2024-03-24T12:33:25.161767+0000","last_undegraded":"2024-03-24T12:33:25.161767+0000","last_fullsized":"2024-03-24T12:33:25.161767+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:51.275761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:51.275761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:07.436140+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":243,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.161731+0000","last_change":"2024-03-24T12:29:09.487195+0000","last_active":"2024-03-24T12:33:25.161731+0000","last_peered":"2024-03-24T12:33:25.161731+0000","last_clean":"2024-03-24T12:33:25.161731+0000","last_became_active":"2024-03-24T12:26:03.161544+0000","last_became_peered":"2024-03-24T12:26:03.161544+0000","last_unstale":"2024-03-24T12:33:25.161731+0000","last_undegraded":"2024-03-24T12:33:25.161731+0000","last_fullsized":"2024-03-24T12:33:25.161731+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:09.487112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:09.487112+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":255,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175958+0000","last_change":"2024-03-24T12:31:08.707454+0000","last_active":"2024-03-24T12:33:25.175958+0000","last_peered":"2024-03-24T12:33:25.175958+0000","last_clean":"2024-03-24T12:33:25.175958+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:33:25.175958+0000","last_undegraded":"2024-03-24T12:33:25.175958+0000","last_fullsized":"2024-03-24T12:33:25.175958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:31:08.707365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:31:08.707365+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.161494+0000","last_change":"2024-03-24T12:33:21.155477+0000","last_active":"2024-03-24T12:33:25.161494+0000","last_peered":"2024-03-24T12:33:25.161494+0000","last_clean":"2024-03-24T12:33:25.161494+0000","last_became_active":"2024-03-24T12:26:09.350721+0000","last_became_peered":"2024-03-24T12:26:09.350721+0000","last_unstale":"2024-03-24T12:33:25.161494+0000","last_undegraded":"2024-03-24T12:33:25.161494+0000","last_fullsized":"2024-03-24T12:33:25.161494+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.155345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.155345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:50.784827+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":235,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.159004+0000","last_change":"2024-03-24T12:29:03.114986+0000","last_active":"2024-03-24T12:33:25.159004+0000","last_peered":"2024-03-24T12:33:25.159004+0000","last_clean":"2024-03-24T12:33:25.159004+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:33:25.159004+0000","last_undegraded":"2024-03-24T12:33:25.159004+0000","last_fullsized":"2024-03-24T12:33:25.159004+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:03.114851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:29:03.114851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":251,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.161617+0000","last_change":"2024-03-24T12:33:23.095826+0000","last_active":"2024-03-24T12:33:25.161617+0000","last_peered":"2024-03-24T12:33:25.161617+0000","last_clean":"2024-03-24T12:33:25.161617+0000","last_became_active":"2024-03-24T12:26:03.159997+0000","last_became_peered":"2024-03-24T12:26:03.159997+0000","last_unstale":"2024-03-24T12:33:25.161617+0000","last_undegraded":"2024-03-24T12:33:25.161617+0000","last_fullsized":"2024-03-24T12:33:25.161617+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:23.095706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:33:23.095706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:52.991000+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":257,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176086+0000","last_change":"2024-03-24T12:33:19.235392+0000","last_active":"2024-03-24T12:33:25.176086+0000","last_peered":"2024-03-24T12:33:25.176086+0000","last_clean":"2024-03-24T12:33:25.176086+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:33:25.176086+0000","last_undegraded":"2024-03-24T12:33:25.176086+0000","last_fullsized":"2024-03-24T12:33:25.176086+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T12:33:19.235269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.235269+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:30.727313+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176803+0000","last_change":"2024-03-24T12:31:23.766176+0000","last_active":"2024-03-24T12:33:25.176803+0000","last_peered":"2024-03-24T12:33:25.176803+0000","last_clean":"2024-03-24T12:33:25.176803+0000","last_became_active":"2024-03-24T12:26:09.643671+0000","last_became_peered":"2024-03-24T12:26:09.643671+0000","last_unstale":"2024-03-24T12:33:25.176803+0000","last_undegraded":"2024-03-24T12:33:25.176803+0000","last_fullsized":"2024-03-24T12:33:25.176803+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:23.766078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:31:23.766078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":251,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.181057+0000","last_change":"2024-03-24T12:33:14.795387+0000","last_active":"2024-03-24T12:33:25.181057+0000","last_peered":"2024-03-24T12:33:25.181057+0000","last_clean":"2024-03-24T12:33:25.181057+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:33:25.181057+0000","last_undegraded":"2024-03-24T12:33:25.181057+0000","last_fullsized":"2024-03-24T12:33:25.181057+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:14.795243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:14.795243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:17.078470+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":243,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.166083+0000","last_change":"2024-03-24T12:32:49.778838+0000","last_active":"2024-03-24T12:33:25.166083+0000","last_peered":"2024-03-24T12:33:25.166083+0000","last_clean":"2024-03-24T12:33:25.166083+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:33:25.166083+0000","last_undegraded":"2024-03-24T12:33:25.166083+0000","last_fullsized":"2024-03-24T12:33:25.166083+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:49.778759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:32:49.778759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:03.791627+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":272,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.163538+0000","last_change":"2024-03-24T12:32:55.627790+0000","last_active":"2024-03-24T12:33:25.163538+0000","last_peered":"2024-03-24T12:33:25.163538+0000","last_clean":"2024-03-24T12:33:25.163538+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:33:25.163538+0000","last_undegraded":"2024-03-24T12:33:25.163538+0000","last_fullsized":"2024-03-24T12:33:25.163538+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:32:55.627693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.627693+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:20.361574+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.969360+0000","last_change":"2024-03-24T12:29:13.161639+0000","last_active":"2024-03-24T12:33:19.969360+0000","last_peered":"2024-03-24T12:33:19.969360+0000","last_clean":"2024-03-24T12:33:19.969360+0000","last_became_active":"2024-03-24T12:26:09.643738+0000","last_became_peered":"2024-03-24T12:26:09.643738+0000","last_unstale":"2024-03-24T12:33:19.969360+0000","last_undegraded":"2024-03-24T12:33:19.969360+0000","last_fullsized":"2024-03-24T12:33:19.969360+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:13.161510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:13.161510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":243,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176634+0000","last_change":"2024-03-24T12:33:01.021282+0000","last_active":"2024-03-24T12:33:25.176634+0000","last_peered":"2024-03-24T12:33:25.176634+0000","last_clean":"2024-03-24T12:33:25.176634+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:33:25.176634+0000","last_undegraded":"2024-03-24T12:33:25.176634+0000","last_fullsized":"2024-03-24T12:33:25.176634+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:01.021148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:33:01.021148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:13.101811+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":251,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175975+0000","last_change":"2024-03-24T12:33:18.258903+0000","last_active":"2024-03-24T12:33:25.175975+0000","last_peered":"2024-03-24T12:33:25.175975+0000","last_clean":"2024-03-24T12:33:25.175975+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:33:25.175975+0000","last_undegraded":"2024-03-24T12:33:25.175975+0000","last_fullsized":"2024-03-24T12:33:25.175975+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:18.258782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.258782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:38.365035+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":260,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.967718+0000","last_change":"2024-03-24T12:33:07.191584+0000","last_active":"2024-03-24T12:33:19.967718+0000","last_peered":"2024-03-24T12:33:19.967718+0000","last_clean":"2024-03-24T12:33:19.967718+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:33:19.967718+0000","last_undegraded":"2024-03-24T12:33:19.967718+0000","last_fullsized":"2024-03-24T12:33:19.967718+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:33:07.191431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.191431+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:24.936962+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.159755+0000","last_change":"2024-03-24T12:32:57.372314+0000","last_active":"2024-03-24T12:33:25.159755+0000","last_peered":"2024-03-24T12:33:25.159755+0000","last_clean":"2024-03-24T12:33:25.159755+0000","last_became_active":"2024-03-24T12:26:09.343786+0000","last_became_peered":"2024-03-24T12:26:09.343786+0000","last_unstale":"2024-03-24T12:33:25.159755+0000","last_undegraded":"2024-03-24T12:33:25.159755+0000","last_fullsized":"2024-03-24T12:33:25.159755+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:32:57.372183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:57.372183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:01.311760+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":344,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175159+0000","last_change":"2024-03-24T12:33:07.975256+0000","last_active":"2024-03-24T12:33:25.175159+0000","last_peered":"2024-03-24T12:33:25.175159+0000","last_clean":"2024-03-24T12:33:25.175159+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:33:25.175159+0000","last_undegraded":"2024-03-24T12:33:25.175159+0000","last_fullsized":"2024-03-24T12:33:25.175159+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:33:07.975118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.975118+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:19.326350+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":261,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.181803+0000","last_change":"2024-03-24T12:33:10.675622+0000","last_active":"2024-03-24T12:33:25.181803+0000","last_peered":"2024-03-24T12:33:25.181803+0000","last_clean":"2024-03-24T12:33:25.181803+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:33:25.181803+0000","last_undegraded":"2024-03-24T12:33:25.181803+0000","last_fullsized":"2024-03-24T12:33:25.181803+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:10.675383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:10.675383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:20.108309+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":263,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.173206+0000","last_change":"2024-03-24T12:33:21.442809+0000","last_active":"2024-03-24T12:33:25.173206+0000","last_peered":"2024-03-24T12:33:25.173206+0000","last_clean":"2024-03-24T12:33:25.173206+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:33:25.173206+0000","last_undegraded":"2024-03-24T12:33:25.173206+0000","last_fullsized":"2024-03-24T12:33:25.173206+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:33:21.442670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.442670+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:30.230404+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":247,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.181202+0000","last_change":"2024-03-24T12:33:03.556099+0000","last_active":"2024-03-24T12:33:25.181202+0000","last_peered":"2024-03-24T12:33:25.181202+0000","last_clean":"2024-03-24T12:33:25.181202+0000","last_became_active":"2024-03-24T12:26:09.348355+0000","last_became_peered":"2024-03-24T12:26:09.348355+0000","last_unstale":"2024-03-24T12:33:25.181202+0000","last_undegraded":"2024-03-24T12:33:25.181202+0000","last_fullsized":"2024-03-24T12:33:25.181202+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:03.555975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.555975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:20.468125+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":236,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.173448+0000","last_change":"2024-03-24T12:33:15.244131+0000","last_active":"2024-03-24T12:33:25.173448+0000","last_peered":"2024-03-24T12:33:25.173448+0000","last_clean":"2024-03-24T12:33:25.173448+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:33:25.173448+0000","last_undegraded":"2024-03-24T12:33:25.173448+0000","last_fullsized":"2024-03-24T12:33:25.173448+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:15.244006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:15.244006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:42.717046+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.971033+0000","last_change":"2024-03-24T12:28:57.028578+0000","last_active":"2024-03-24T12:33:19.971033+0000","last_peered":"2024-03-24T12:33:19.971033+0000","last_clean":"2024-03-24T12:33:19.971033+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:33:19.971033+0000","last_undegraded":"2024-03-24T12:33:19.971033+0000","last_fullsized":"2024-03-24T12:33:19.971033+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:57.028457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:28:57.028457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":267,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175138+0000","last_change":"2024-03-24T12:32:55.882888+0000","last_active":"2024-03-24T12:33:25.175138+0000","last_peered":"2024-03-24T12:33:25.175138+0000","last_clean":"2024-03-24T12:33:25.175138+0000","last_became_active":"2024-03-24T12:26:05.317587+0000","last_became_peered":"2024-03-24T12:26:05.317587+0000","last_unstale":"2024-03-24T12:33:25.175138+0000","last_undegraded":"2024-03-24T12:33:25.175138+0000","last_fullsized":"2024-03-24T12:33:25.175138+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:32:55.882733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.882733+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:57.755957+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.158895+0000","last_change":"2024-03-24T12:33:01.322667+0000","last_active":"2024-03-24T12:33:25.158895+0000","last_peered":"2024-03-24T12:33:25.158895+0000","last_clean":"2024-03-24T12:33:25.158895+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:33:25.158895+0000","last_undegraded":"2024-03-24T12:33:25.158895+0000","last_fullsized":"2024-03-24T12:33:25.158895+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:01.322531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:01.322531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:27.713706+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.174149+0000","last_change":"2024-03-24T12:33:12.983236+0000","last_active":"2024-03-24T12:33:25.174149+0000","last_peered":"2024-03-24T12:33:25.174149+0000","last_clean":"2024-03-24T12:33:25.174149+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:33:25.174149+0000","last_undegraded":"2024-03-24T12:33:25.174149+0000","last_fullsized":"2024-03-24T12:33:25.174149+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:12.983157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.983157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:34.212572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.166183+0000","last_change":"2024-03-24T12:33:00.920226+0000","last_active":"2024-03-24T12:33:25.166183+0000","last_peered":"2024-03-24T12:33:25.166183+0000","last_clean":"2024-03-24T12:33:25.166183+0000","last_became_active":"2024-03-24T12:26:03.170395+0000","last_became_peered":"2024-03-24T12:26:03.170395+0000","last_unstale":"2024-03-24T12:33:25.166183+0000","last_undegraded":"2024-03-24T12:33:25.166183+0000","last_fullsized":"2024-03-24T12:33:25.166183+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:00.920105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:00.920105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:27.249347+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":244,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.971123+0000","last_change":"2024-03-24T12:28:54.970495+0000","last_active":"2024-03-24T12:33:19.971123+0000","last_peered":"2024-03-24T12:33:19.971123+0000","last_clean":"2024-03-24T12:33:19.971123+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:33:19.971123+0000","last_undegraded":"2024-03-24T12:33:19.971123+0000","last_fullsized":"2024-03-24T12:33:19.971123+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:28:54.970319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:28:54.970319+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.971473+0000","last_change":"2024-03-24T12:31:29.996318+0000","last_active":"2024-03-24T12:33:19.971473+0000","last_peered":"2024-03-24T12:33:19.971473+0000","last_clean":"2024-03-24T12:33:19.971473+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:33:19.971473+0000","last_undegraded":"2024-03-24T12:33:19.971473+0000","last_fullsized":"2024-03-24T12:33:19.971473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:31:29.996197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:26.229993+0000","last_change":"2024-03-24T12:33:26.230030+0000","last_active":"2024-03-24T12:33:26.229993+0000","last_peered":"2024-03-24T12:33:26.229993+0000","last_clean":"2024-03-24T12:33:26.229993+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:33:26.229993+0000","last_undegraded":"2024-03-24T12:33:26.229993+0000","last_fullsized":"2024-03-24T12:33:26.229993+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:26.229911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:26.229911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.062544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":250,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.161926+0000","last_change":"2024-03-24T12:29:11.579113+0000","last_active":"2024-03-24T12:33:25.161926+0000","last_peered":"2024-03-24T12:33:25.161926+0000","last_clean":"2024-03-24T12:33:25.161926+0000","last_became_active":"2024-03-24T12:26:03.165083+0000","last_became_peered":"2024-03-24T12:26:03.165083+0000","last_unstale":"2024-03-24T12:33:25.161926+0000","last_undegraded":"2024-03-24T12:33:25.161926+0000","last_fullsized":"2024-03-24T12:33:25.161926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:29:11.579030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:11.579030+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":257,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.967277+0000","last_change":"2024-03-24T12:31:29.027987+0000","last_active":"2024-03-24T12:33:19.967277+0000","last_peered":"2024-03-24T12:33:19.967277+0000","last_clean":"2024-03-24T12:33:19.967277+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:33:19.967277+0000","last_undegraded":"2024-03-24T12:33:19.967277+0000","last_fullsized":"2024-03-24T12:33:19.967277+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:31:29.027885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:31:29.027885+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.165008+0000","last_change":"2024-03-24T12:32:53.875297+0000","last_active":"2024-03-24T12:33:25.165008+0000","last_peered":"2024-03-24T12:33:25.165008+0000","last_clean":"2024-03-24T12:33:25.165008+0000","last_became_active":"2024-03-24T12:26:09.350656+0000","last_became_peered":"2024-03-24T12:26:09.350656+0000","last_unstale":"2024-03-24T12:33:25.165008+0000","last_undegraded":"2024-03-24T12:33:25.165008+0000","last_fullsized":"2024-03-24T12:33:25.165008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:53.875206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:32:53.875206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:17.534984+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":234,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.968593+0000","last_change":"2024-03-24T12:28:56.010757+0000","last_active":"2024-03-24T12:33:19.968593+0000","last_peered":"2024-03-24T12:33:19.968593+0000","last_clean":"2024-03-24T12:33:19.968593+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:33:19.968593+0000","last_undegraded":"2024-03-24T12:33:19.968593+0000","last_fullsized":"2024-03-24T12:33:19.968593+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:56.010642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:20.458419+0000","last_clean_scrub_stamp":"2024-03-24T12:28:56.010642+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.160108+0000","last_change":"2024-03-24T12:33:12.283498+0000","last_active":"2024-03-24T12:33:25.160108+0000","last_peered":"2024-03-24T12:33:25.160108+0000","last_clean":"2024-03-24T12:33:25.160108+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:33:25.160108+0000","last_undegraded":"2024-03-24T12:33:25.160108+0000","last_fullsized":"2024-03-24T12:33:25.160108+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:12.283417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.283417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:30.310733+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":234,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.162137+0000","last_change":"2024-03-24T12:33:03.187495+0000","last_active":"2024-03-24T12:33:25.162137+0000","last_peered":"2024-03-24T12:33:25.162137+0000","last_clean":"2024-03-24T12:33:25.162137+0000","last_became_active":"2024-03-24T12:26:16.874170+0000","last_became_peered":"2024-03-24T12:26:16.874170+0000","last_unstale":"2024-03-24T12:33:25.162137+0000","last_undegraded":"2024-03-24T12:33:25.162137+0000","last_fullsized":"2024-03-24T12:33:25.162137+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:03.187410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.187410+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:14.188131+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.966959+0000","last_change":"2024-03-24T12:29:05.266028+0000","last_active":"2024-03-24T12:33:19.966959+0000","last_peered":"2024-03-24T12:33:19.966959+0000","last_clean":"2024-03-24T12:33:19.966959+0000","last_became_active":"2024-03-24T12:26:09.343296+0000","last_became_peered":"2024-03-24T12:26:09.343296+0000","last_unstale":"2024-03-24T12:33:19.966959+0000","last_undegraded":"2024-03-24T12:33:19.966959+0000","last_fullsized":"2024-03-24T12:33:19.966959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:05.265893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:29:05.265893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.182133+0000","last_change":"2024-03-24T12:33:21.145032+0000","last_active":"2024-03-24T12:33:25.182133+0000","last_peered":"2024-03-24T12:33:25.182133+0000","last_clean":"2024-03-24T12:33:25.182133+0000","last_became_active":"2024-03-24T12:26:07.330811+0000","last_became_peered":"2024-03-24T12:26:07.330811+0000","last_unstale":"2024-03-24T12:33:25.182133+0000","last_undegraded":"2024-03-24T12:33:25.182133+0000","last_fullsized":"2024-03-24T12:33:25.182133+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.144883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.144883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:24.165078+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":260,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176439+0000","last_change":"2024-03-24T12:33:17.214730+0000","last_active":"2024-03-24T12:33:25.176439+0000","last_peered":"2024-03-24T12:33:25.176439+0000","last_clean":"2024-03-24T12:33:25.176439+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:33:25.176439+0000","last_undegraded":"2024-03-24T12:33:25.176439+0000","last_fullsized":"2024-03-24T12:33:25.176439+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:17.214655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:17.214655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:43.429599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"88'16","reported_seq":280,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176325+0000","last_change":"2024-03-24T12:32:50.139927+0000","last_active":"2024-03-24T12:33:25.176325+0000","last_peered":"2024-03-24T12:33:25.176325+0000","last_clean":"2024-03-24T12:33:25.176325+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:33:25.176325+0000","last_undegraded":"2024-03-24T12:33:25.176325+0000","last_fullsized":"2024-03-24T12:33:25.176325+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:32:50.139785+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:32:50.139785+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:14.241062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":233,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.172966+0000","last_change":"2024-03-24T12:33:08.045743+0000","last_active":"2024-03-24T12:33:25.172966+0000","last_peered":"2024-03-24T12:33:25.172966+0000","last_clean":"2024-03-24T12:33:25.172966+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:33:25.172966+0000","last_undegraded":"2024-03-24T12:33:25.172966+0000","last_fullsized":"2024-03-24T12:33:25.172966+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:33:08.045492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.045492+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:20.112237+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":333,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.161863+0000","last_change":"2024-03-24T12:32:57.178519+0000","last_active":"2024-03-24T12:33:25.161863+0000","last_peered":"2024-03-24T12:33:25.161863+0000","last_clean":"2024-03-24T12:33:25.161863+0000","last_became_active":"2024-03-24T12:26:07.329487+0000","last_became_peered":"2024-03-24T12:26:07.329487+0000","last_unstale":"2024-03-24T12:33:25.161863+0000","last_undegraded":"2024-03-24T12:33:25.161863+0000","last_fullsized":"2024-03-24T12:33:25.161863+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:32:57.177894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:57.177894+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:19.759589+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.159264+0000","last_change":"2024-03-24T12:33:06.290383+0000","last_active":"2024-03-24T12:33:25.159264+0000","last_peered":"2024-03-24T12:33:25.159264+0000","last_clean":"2024-03-24T12:33:25.159264+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:33:25.159264+0000","last_undegraded":"2024-03-24T12:33:25.159264+0000","last_fullsized":"2024-03-24T12:33:25.159264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:06.290232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:06.290232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:22.464064+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":253,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.971273+0000","last_change":"2024-03-24T12:29:15.283980+0000","last_active":"2024-03-24T12:33:19.971273+0000","last_peered":"2024-03-24T12:33:19.971273+0000","last_clean":"2024-03-24T12:33:19.971273+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:33:19.971273+0000","last_undegraded":"2024-03-24T12:33:19.971273+0000","last_fullsized":"2024-03-24T12:33:19.971273+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:29:15.283868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:29:15.283868+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176975+0000","last_change":"2024-03-24T12:32:52.071488+0000","last_active":"2024-03-24T12:33:25.176975+0000","last_peered":"2024-03-24T12:33:25.176975+0000","last_clean":"2024-03-24T12:33:25.176975+0000","last_became_active":"2024-03-24T12:26:09.346062+0000","last_became_peered":"2024-03-24T12:26:09.346062+0000","last_unstale":"2024-03-24T12:33:25.176975+0000","last_undegraded":"2024-03-24T12:33:25.176975+0000","last_fullsized":"2024-03-24T12:33:25.176975+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:52.071390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:52.071390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:08.927078+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"93'9","reported_seq":332,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.971226+0000","last_change":"2024-03-24T12:33:06.165605+0000","last_active":"2024-03-24T12:33:19.971226+0000","last_peered":"2024-03-24T12:33:19.971226+0000","last_clean":"2024-03-24T12:33:19.971226+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:33:19.971226+0000","last_undegraded":"2024-03-24T12:33:19.971226+0000","last_fullsized":"2024-03-24T12:33:19.971226+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'9","last_scrub_stamp":"2024-03-24T12:33:06.165466+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:33:06.165466+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:25.687119+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.166631+0000","last_change":"2024-03-24T12:29:05.327318+0000","last_active":"2024-03-24T12:33:25.166631+0000","last_peered":"2024-03-24T12:33:25.166631+0000","last_clean":"2024-03-24T12:33:25.166631+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:33:25.166631+0000","last_undegraded":"2024-03-24T12:33:25.166631+0000","last_fullsized":"2024-03-24T12:33:25.166631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:05.327071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:05.327071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'13","reported_seq":268,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.162027+0000","last_change":"2024-03-24T12:33:09.042723+0000","last_active":"2024-03-24T12:33:25.162027+0000","last_peered":"2024-03-24T12:33:25.162027+0000","last_clean":"2024-03-24T12:33:25.162027+0000","last_became_active":"2024-03-24T12:26:05.358213+0000","last_became_peered":"2024-03-24T12:26:05.358213+0000","last_unstale":"2024-03-24T12:33:25.162027+0000","last_undegraded":"2024-03-24T12:33:25.162027+0000","last_fullsized":"2024-03-24T12:33:25.162027+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:33:09.042629+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:33:09.042629+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:15.135068+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.163843+0000","last_change":"2024-03-24T12:33:00.629949+0000","last_active":"2024-03-24T12:33:25.163843+0000","last_peered":"2024-03-24T12:33:25.163843+0000","last_clean":"2024-03-24T12:33:25.163843+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:33:25.163843+0000","last_undegraded":"2024-03-24T12:33:25.163843+0000","last_fullsized":"2024-03-24T12:33:25.163843+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:00.629793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:33:00.629793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:15.360069+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":234,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.969311+0000","last_change":"2024-03-24T12:33:18.355672+0000","last_active":"2024-03-24T12:33:19.969311+0000","last_peered":"2024-03-24T12:33:19.969311+0000","last_clean":"2024-03-24T12:33:19.969311+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:33:19.969311+0000","last_undegraded":"2024-03-24T12:33:19.969311+0000","last_fullsized":"2024-03-24T12:33:19.969311+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:18.355449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.355449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:44.546629+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.158958+0000","last_change":"2024-03-24T12:33:11.251158+0000","last_active":"2024-03-24T12:33:25.158958+0000","last_peered":"2024-03-24T12:33:25.158958+0000","last_clean":"2024-03-24T12:33:25.158958+0000","last_became_active":"2024-03-24T12:32:43.256833+0000","last_became_peered":"2024-03-24T12:32:43.256833+0000","last_unstale":"2024-03-24T12:33:25.158958+0000","last_undegraded":"2024-03-24T12:33:25.158958+0000","last_fullsized":"2024-03-24T12:33:25.158958+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:11.251080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:11.251080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:19.103711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":264,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.173293+0000","last_change":"2024-03-24T12:33:18.443409+0000","last_active":"2024-03-24T12:33:25.173293+0000","last_peered":"2024-03-24T12:33:25.173293+0000","last_clean":"2024-03-24T12:33:25.173293+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:33:25.173293+0000","last_undegraded":"2024-03-24T12:33:25.173293+0000","last_fullsized":"2024-03-24T12:33:25.173293+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:18.443276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.443276+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:40.854569+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.159940+0000","last_change":"2024-03-24T12:33:03.328835+0000","last_active":"2024-03-24T12:33:25.159940+0000","last_peered":"2024-03-24T12:33:25.159940+0000","last_clean":"2024-03-24T12:33:25.159940+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:33:25.159940+0000","last_undegraded":"2024-03-24T12:33:25.159940+0000","last_fullsized":"2024-03-24T12:33:25.159940+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:03.328728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.328728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:30.116042+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":236,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.161929+0000","last_change":"2024-03-24T12:32:59.274958+0000","last_active":"2024-03-24T12:33:25.161929+0000","last_peered":"2024-03-24T12:33:25.161929+0000","last_clean":"2024-03-24T12:33:25.161929+0000","last_became_active":"2024-03-24T12:26:07.329117+0000","last_became_peered":"2024-03-24T12:26:07.329117+0000","last_unstale":"2024-03-24T12:33:25.161929+0000","last_undegraded":"2024-03-24T12:33:25.161929+0000","last_fullsized":"2024-03-24T12:33:25.161929+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:59.274820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:59.274820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:20.251927+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.173277+0000","last_change":"2024-03-24T12:32:52.926826+0000","last_active":"2024-03-24T12:33:25.173277+0000","last_peered":"2024-03-24T12:33:25.173277+0000","last_clean":"2024-03-24T12:33:25.173277+0000","last_became_active":"2024-03-24T12:26:03.161414+0000","last_became_peered":"2024-03-24T12:26:03.161414+0000","last_unstale":"2024-03-24T12:33:25.173277+0000","last_undegraded":"2024-03-24T12:33:25.173277+0000","last_fullsized":"2024-03-24T12:33:25.173277+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:52.926693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:52.926693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:59.823576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":247,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.969497+0000","last_change":"2024-03-24T12:32:58.025762+0000","last_active":"2024-03-24T12:33:19.969497+0000","last_peered":"2024-03-24T12:33:19.969497+0000","last_clean":"2024-03-24T12:33:19.969497+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:33:19.969497+0000","last_undegraded":"2024-03-24T12:33:19.969497+0000","last_fullsized":"2024-03-24T12:33:19.969497+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T12:32:58.023697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:58.023697+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:16.940453+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"174'3558","reported_seq":3600,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:27.437458+0000","last_change":"2024-03-24T12:32:41.565569+0000","last_active":"2024-03-24T12:33:27.437458+0000","last_peered":"2024-03-24T12:33:27.437458+0000","last_clean":"2024-03-24T12:33:27.437458+0000","last_became_active":"2024-03-24T12:32:41.565218+0000","last_became_peered":"2024-03-24T12:32:41.565218+0000","last_unstale":"2024-03-24T12:33:27.437458+0000","last_undegraded":"2024-03-24T12:33:27.437458+0000","last_fullsized":"2024-03-24T12:33:27.437458+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3558,"log_dups_size":0,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:52.535210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233177088,"num_objects":3558,"num_object_clones":0,"num_object_copies":10674,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3558,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3558,"num_write_kb":227712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.162342+0000","last_change":"2024-03-24T12:29:15.585891+0000","last_active":"2024-03-24T12:33:25.162342+0000","last_peered":"2024-03-24T12:33:25.162342+0000","last_clean":"2024-03-24T12:33:25.162342+0000","last_became_active":"2024-03-24T12:26:09.643602+0000","last_became_peered":"2024-03-24T12:26:09.643602+0000","last_unstale":"2024-03-24T12:33:25.162342+0000","last_undegraded":"2024-03-24T12:33:25.162342+0000","last_fullsized":"2024-03-24T12:33:25.162342+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:15.585805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:15.585805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":256,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.160263+0000","last_change":"2024-03-24T12:29:13.102973+0000","last_active":"2024-03-24T12:33:25.160263+0000","last_peered":"2024-03-24T12:33:25.160263+0000","last_clean":"2024-03-24T12:33:25.160263+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:33:25.160263+0000","last_undegraded":"2024-03-24T12:33:25.160263+0000","last_fullsized":"2024-03-24T12:33:25.160263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:29:13.102846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:13.102846+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":273,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:26.095713+0000","last_change":"2024-03-24T12:33:26.095777+0000","last_active":"2024-03-24T12:33:26.095713+0000","last_peered":"2024-03-24T12:33:26.095713+0000","last_clean":"2024-03-24T12:33:26.095713+0000","last_became_active":"2024-03-24T12:26:05.358217+0000","last_became_peered":"2024-03-24T12:26:05.358217+0000","last_unstale":"2024-03-24T12:33:26.095713+0000","last_undegraded":"2024-03-24T12:33:26.095713+0000","last_fullsized":"2024-03-24T12:33:26.095713+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:33:26.094785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:26.094785+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:51.228338+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":344,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.159146+0000","last_change":"2024-03-24T12:33:22.181670+0000","last_active":"2024-03-24T12:33:25.159146+0000","last_peered":"2024-03-24T12:33:25.159146+0000","last_clean":"2024-03-24T12:33:25.159146+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:33:25.159146+0000","last_undegraded":"2024-03-24T12:33:25.159146+0000","last_fullsized":"2024-03-24T12:33:25.159146+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:33:22.181560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:22.181560+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:22.438349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"174'3399","reported_seq":3440,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:27.457312+0000","last_change":"2024-03-24T12:32:41.045588+0000","last_active":"2024-03-24T12:33:27.457312+0000","last_peered":"2024-03-24T12:33:27.457312+0000","last_clean":"2024-03-24T12:33:27.457312+0000","last_became_active":"2024-03-24T12:32:41.045244+0000","last_became_peered":"2024-03-24T12:32:41.045244+0000","last_unstale":"2024-03-24T12:33:27.457312+0000","last_undegraded":"2024-03-24T12:33:27.457312+0000","last_fullsized":"2024-03-24T12:33:27.457312+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3399,"log_dups_size":0,"ondisk_log_size":3399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:53.791851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222756864,"num_objects":3399,"num_object_clones":0,"num_object_copies":10197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3399,"num_write_kb":217536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:20.675848+0000","last_change":"2024-03-24T12:33:20.675885+0000","last_active":"2024-03-24T12:33:20.675848+0000","last_peered":"2024-03-24T12:33:20.675848+0000","last_clean":"2024-03-24T12:33:20.675848+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:33:20.675848+0000","last_undegraded":"2024-03-24T12:33:20.675848+0000","last_fullsized":"2024-03-24T12:33:20.675848+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:20.675649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:20.675649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:34.909534+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.160219+0000","last_change":"2024-03-24T12:32:59.313688+0000","last_active":"2024-03-24T12:33:25.160219+0000","last_peered":"2024-03-24T12:33:25.160219+0000","last_clean":"2024-03-24T12:33:25.160219+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:33:25.160219+0000","last_undegraded":"2024-03-24T12:33:25.160219+0000","last_fullsized":"2024-03-24T12:33:25.160219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:59.313589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:59.313589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:29.092011+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"140'17","reported_seq":276,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175954+0000","last_change":"2024-03-24T12:31:25.769891+0000","last_active":"2024-03-24T12:33:25.175954+0000","last_peered":"2024-03-24T12:33:25.175954+0000","last_clean":"2024-03-24T12:33:25.175954+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:33:25.175954+0000","last_undegraded":"2024-03-24T12:33:25.175954+0000","last_fullsized":"2024-03-24T12:33:25.175954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'17","last_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:31:25.769706+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.162734+0000","last_change":"2024-03-24T12:33:13.794657+0000","last_active":"2024-03-24T12:33:25.162734+0000","last_peered":"2024-03-24T12:33:25.162734+0000","last_clean":"2024-03-24T12:33:25.162734+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:33:25.162734+0000","last_undegraded":"2024-03-24T12:33:25.162734+0000","last_fullsized":"2024-03-24T12:33:25.162734+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:13.794520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:13.794520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:21.943137+0000","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"174'3372","reported_seq":3412,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.680715+0000","last_change":"2024-03-24T12:32:41.041630+0000","last_active":"2024-03-24T12:33:25.680715+0000","last_peered":"2024-03-24T12:33:25.680715+0000","last_clean":"2024-03-24T12:33:25.680715+0000","last_became_active":"2024-03-24T12:32:41.041303+0000","last_became_peered":"2024-03-24T12:32:41.041303+0000","last_unstale":"2024-03-24T12:33:25.680715+0000","last_undegraded":"2024-03-24T12:33:25.680715+0000","last_fullsized":"2024-03-24T12:33:25.680715+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3372,"log_dups_size":0,"ondisk_log_size":3372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:57.281947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220987392,"num_objects":3372,"num_object_clones":0,"num_object_copies":10116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3372,"num_write_kb":215808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.160099+0000","last_change":"2024-03-24T12:33:07.283118+0000","last_active":"2024-03-24T12:33:25.160099+0000","last_peered":"2024-03-24T12:33:25.160099+0000","last_clean":"2024-03-24T12:33:25.160099+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:33:25.160099+0000","last_undegraded":"2024-03-24T12:33:25.160099+0000","last_fullsized":"2024-03-24T12:33:25.160099+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:07.282989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.282989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:08.518970+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.163180+0000","last_change":"2024-03-24T12:33:06.631005+0000","last_active":"2024-03-24T12:33:25.163180+0000","last_peered":"2024-03-24T12:33:25.163180+0000","last_clean":"2024-03-24T12:33:25.163180+0000","last_became_active":"2024-03-24T12:26:07.331416+0000","last_became_peered":"2024-03-24T12:26:07.331416+0000","last_unstale":"2024-03-24T12:33:25.163180+0000","last_undegraded":"2024-03-24T12:33:25.163180+0000","last_fullsized":"2024-03-24T12:33:25.163180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:06.630869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:06.630869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.817810+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.160383+0000","last_change":"2024-03-24T12:29:12.028957+0000","last_active":"2024-03-24T12:33:25.160383+0000","last_peered":"2024-03-24T12:33:25.160383+0000","last_clean":"2024-03-24T12:33:25.160383+0000","last_became_active":"2024-03-24T12:26:03.165214+0000","last_became_peered":"2024-03-24T12:26:03.165214+0000","last_unstale":"2024-03-24T12:33:25.160383+0000","last_undegraded":"2024-03-24T12:33:25.160383+0000","last_fullsized":"2024-03-24T12:33:25.160383+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:12.028882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:12.028882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":279,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.173637+0000","last_change":"2024-03-24T12:33:01.910428+0000","last_active":"2024-03-24T12:33:25.173637+0000","last_peered":"2024-03-24T12:33:25.173637+0000","last_clean":"2024-03-24T12:33:25.173637+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:33:25.173637+0000","last_undegraded":"2024-03-24T12:33:25.173637+0000","last_fullsized":"2024-03-24T12:33:25.173637+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:33:01.910310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:01.910310+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:22.584536+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"174'3535","reported_seq":3577,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:27.777351+0000","last_change":"2024-03-24T12:32:41.042352+0000","last_active":"2024-03-24T12:33:27.777351+0000","last_peered":"2024-03-24T12:33:27.777351+0000","last_clean":"2024-03-24T12:33:27.777351+0000","last_became_active":"2024-03-24T12:32:41.042042+0000","last_became_peered":"2024-03-24T12:32:41.042042+0000","last_unstale":"2024-03-24T12:33:27.777351+0000","last_undegraded":"2024-03-24T12:33:27.777351+0000","last_fullsized":"2024-03-24T12:33:27.777351+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3535,"log_dups_size":0,"ondisk_log_size":3535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:43.502749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231669760,"num_objects":3535,"num_object_clones":0,"num_object_copies":10605,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3535,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3535,"num_write_kb":226240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.170627+0000","last_change":"2024-03-24T12:33:13.176741+0000","last_active":"2024-03-24T12:33:25.170627+0000","last_peered":"2024-03-24T12:33:25.170627+0000","last_clean":"2024-03-24T12:33:25.170627+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:33:25.170627+0000","last_undegraded":"2024-03-24T12:33:25.170627+0000","last_fullsized":"2024-03-24T12:33:25.170627+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:13.176371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:13.176371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.337454+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.164564+0000","last_change":"2024-03-24T12:33:16.756497+0000","last_active":"2024-03-24T12:33:25.164564+0000","last_peered":"2024-03-24T12:33:25.164564+0000","last_clean":"2024-03-24T12:33:25.164564+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:33:25.164564+0000","last_undegraded":"2024-03-24T12:33:25.164564+0000","last_fullsized":"2024-03-24T12:33:25.164564+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:16.756102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:33:16.756102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:20.821467+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":259,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.161625+0000","last_change":"2024-03-24T12:33:08.075378+0000","last_active":"2024-03-24T12:33:25.161625+0000","last_peered":"2024-03-24T12:33:25.161625+0000","last_clean":"2024-03-24T12:33:25.161625+0000","last_became_active":"2024-03-24T12:26:03.162562+0000","last_became_peered":"2024-03-24T12:26:03.162562+0000","last_unstale":"2024-03-24T12:33:25.161625+0000","last_undegraded":"2024-03-24T12:33:25.161625+0000","last_fullsized":"2024-03-24T12:33:25.161625+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:33:08.075093+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.075093+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:28.745816+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":272,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.165886+0000","last_change":"2024-03-24T12:33:17.435894+0000","last_active":"2024-03-24T12:33:25.165886+0000","last_peered":"2024-03-24T12:33:25.165886+0000","last_clean":"2024-03-24T12:33:25.165886+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:33:25.165886+0000","last_undegraded":"2024-03-24T12:33:25.165886+0000","last_fullsized":"2024-03-24T12:33:25.165886+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:17.435756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:17.435756+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.934217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"174'3481","reported_seq":3523,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:27.425443+0000","last_change":"2024-03-24T12:32:41.044116+0000","last_active":"2024-03-24T12:33:27.425443+0000","last_peered":"2024-03-24T12:33:27.425443+0000","last_clean":"2024-03-24T12:33:27.425443+0000","last_became_active":"2024-03-24T12:32:41.043709+0000","last_became_peered":"2024-03-24T12:32:41.043709+0000","last_unstale":"2024-03-24T12:33:27.425443+0000","last_undegraded":"2024-03-24T12:33:27.425443+0000","last_fullsized":"2024-03-24T12:33:27.425443+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3481,"log_dups_size":0,"ondisk_log_size":3481,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:56.385775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":228130816,"num_objects":3481,"num_object_clones":0,"num_object_copies":10443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3481,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3481,"num_write_kb":222784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.968378+0000","last_change":"2024-03-24T12:32:57.050467+0000","last_active":"2024-03-24T12:33:19.968378+0000","last_peered":"2024-03-24T12:33:19.968378+0000","last_clean":"2024-03-24T12:33:19.968378+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:33:19.968378+0000","last_undegraded":"2024-03-24T12:33:19.968378+0000","last_fullsized":"2024-03-24T12:33:19.968378+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:57.050342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:32:57.050342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:16.075230+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":253,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.162334+0000","last_change":"2024-03-24T12:33:06.156977+0000","last_active":"2024-03-24T12:33:25.162334+0000","last_peered":"2024-03-24T12:33:25.162334+0000","last_clean":"2024-03-24T12:33:25.162334+0000","last_became_active":"2024-03-24T12:26:03.165771+0000","last_became_peered":"2024-03-24T12:26:03.165771+0000","last_unstale":"2024-03-24T12:33:25.162334+0000","last_undegraded":"2024-03-24T12:33:25.162334+0000","last_fullsized":"2024-03-24T12:33:25.162334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:33:06.156811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:06.156811+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:12.292892+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176127+0000","last_change":"2024-03-24T12:32:54.061412+0000","last_active":"2024-03-24T12:33:25.176127+0000","last_peered":"2024-03-24T12:33:25.176127+0000","last_clean":"2024-03-24T12:33:25.176127+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:33:25.176127+0000","last_undegraded":"2024-03-24T12:33:25.176127+0000","last_fullsized":"2024-03-24T12:33:25.176127+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:54.061297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:54.061297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:03.131784+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"90'38","reported_seq":395,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.180532+0000","last_change":"2024-03-24T12:32:55.403983+0000","last_active":"2024-03-24T12:33:25.180532+0000","last_peered":"2024-03-24T12:33:25.180532+0000","last_clean":"2024-03-24T12:33:25.180532+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:33:25.180532+0000","last_undegraded":"2024-03-24T12:33:25.180532+0000","last_fullsized":"2024-03-24T12:33:25.180532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'38","last_scrub_stamp":"2024-03-24T12:32:55.403873+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.403873+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:18.319932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":236,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.165526+0000","last_change":"2024-03-24T12:32:52.817749+0000","last_active":"2024-03-24T12:33:25.165526+0000","last_peered":"2024-03-24T12:33:25.165526+0000","last_clean":"2024-03-24T12:33:25.165526+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:33:25.165526+0000","last_undegraded":"2024-03-24T12:33:25.165526+0000","last_fullsized":"2024-03-24T12:33:25.165526+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:52.817660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:52.817660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:58.641089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"173'2995","reported_seq":3035,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:23.616534+0000","last_change":"2024-03-24T12:32:41.045390+0000","last_active":"2024-03-24T12:33:23.616534+0000","last_peered":"2024-03-24T12:33:23.616534+0000","last_clean":"2024-03-24T12:33:23.616534+0000","last_became_active":"2024-03-24T12:32:41.045113+0000","last_became_peered":"2024-03-24T12:32:41.045113+0000","last_unstale":"2024-03-24T12:33:23.616534+0000","last_undegraded":"2024-03-24T12:33:23.616534+0000","last_fullsized":"2024-03-24T12:33:23.616534+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":2995,"log_dups_size":0,"ondisk_log_size":2995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:06.954161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196280320,"num_objects":2995,"num_object_clones":0,"num_object_copies":8985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2995,"num_write_kb":191680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.160437+0000","last_change":"2024-03-24T12:29:09.997581+0000","last_active":"2024-03-24T12:33:25.160437+0000","last_peered":"2024-03-24T12:33:25.160437+0000","last_clean":"2024-03-24T12:33:25.160437+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:33:25.160437+0000","last_undegraded":"2024-03-24T12:33:25.160437+0000","last_fullsized":"2024-03-24T12:33:25.160437+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:09.997481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:09.997481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":260,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.181255+0000","last_change":"2024-03-24T12:33:19.170350+0000","last_active":"2024-03-24T12:33:25.181255+0000","last_peered":"2024-03-24T12:33:25.181255+0000","last_clean":"2024-03-24T12:33:25.181255+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:33:25.181255+0000","last_undegraded":"2024-03-24T12:33:25.181255+0000","last_fullsized":"2024-03-24T12:33:25.181255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:19.170147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.170147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:40.775761+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":259,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.168081+0000","last_change":"2024-03-24T12:32:58.630218+0000","last_active":"2024-03-24T12:33:25.168081+0000","last_peered":"2024-03-24T12:33:25.168081+0000","last_clean":"2024-03-24T12:33:25.168081+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:33:25.168081+0000","last_undegraded":"2024-03-24T12:33:25.168081+0000","last_fullsized":"2024-03-24T12:33:25.168081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:58.630094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:58.630094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:11.169979+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":288,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.180338+0000","last_change":"2024-03-24T12:33:02.509385+0000","last_active":"2024-03-24T12:33:25.180338+0000","last_peered":"2024-03-24T12:33:25.180338+0000","last_clean":"2024-03-24T12:33:25.180338+0000","last_became_active":"2024-03-24T12:26:05.315639+0000","last_became_peered":"2024-03-24T12:26:05.315639+0000","last_unstale":"2024-03-24T12:33:25.180338+0000","last_undegraded":"2024-03-24T12:33:25.180338+0000","last_fullsized":"2024-03-24T12:33:25.180338+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:33:02.509218+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:31:08.237387+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.509218+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:29.191546+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.169198+0000","last_change":"2024-03-24T12:33:09.679795+0000","last_active":"2024-03-24T12:33:25.169198+0000","last_peered":"2024-03-24T12:33:25.169198+0000","last_clean":"2024-03-24T12:33:25.169198+0000","last_became_active":"2024-03-24T12:26:07.343010+0000","last_became_peered":"2024-03-24T12:26:07.343010+0000","last_unstale":"2024-03-24T12:33:25.169198+0000","last_undegraded":"2024-03-24T12:33:25.169198+0000","last_fullsized":"2024-03-24T12:33:25.169198+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:09.679721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:09.679721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.507477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"174'3463","reported_seq":3505,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:27.466233+0000","last_change":"2024-03-24T12:32:41.042784+0000","last_active":"2024-03-24T12:33:27.466233+0000","last_peered":"2024-03-24T12:33:27.466233+0000","last_clean":"2024-03-24T12:33:27.466233+0000","last_became_active":"2024-03-24T12:32:41.042370+0000","last_became_peered":"2024-03-24T12:32:41.042370+0000","last_unstale":"2024-03-24T12:33:27.466233+0000","last_undegraded":"2024-03-24T12:33:27.466233+0000","last_fullsized":"2024-03-24T12:33:27.466233+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3463,"log_dups_size":0,"ondisk_log_size":3463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:52.594390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226951168,"num_objects":3463,"num_object_clones":0,"num_object_copies":10389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3463,"num_write_kb":221632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.162080+0000","last_change":"2024-03-24T12:33:18.134199+0000","last_active":"2024-03-24T12:33:25.162080+0000","last_peered":"2024-03-24T12:33:25.162080+0000","last_clean":"2024-03-24T12:33:25.162080+0000","last_became_active":"2024-03-24T12:26:09.347821+0000","last_became_peered":"2024-03-24T12:26:09.347821+0000","last_unstale":"2024-03-24T12:33:25.162080+0000","last_undegraded":"2024-03-24T12:33:25.162080+0000","last_fullsized":"2024-03-24T12:33:25.162080+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:18.134111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.134111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:39.370943+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":505,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.160502+0000","last_change":"2024-03-24T12:33:14.339596+0000","last_active":"2024-03-24T12:33:25.160502+0000","last_peered":"2024-03-24T12:33:25.160502+0000","last_clean":"2024-03-24T12:33:25.160502+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:33:25.160502+0000","last_undegraded":"2024-03-24T12:33:25.160502+0000","last_fullsized":"2024-03-24T12:33:25.160502+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:33:14.339456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:33:14.339456+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:40.390099+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.967607+0000","last_change":"2024-03-24T12:28:45.877854+0000","last_active":"2024-03-24T12:33:19.967607+0000","last_peered":"2024-03-24T12:33:19.967607+0000","last_clean":"2024-03-24T12:33:19.967607+0000","last_became_active":"2024-03-24T12:26:03.166467+0000","last_became_peered":"2024-03-24T12:26:03.166467+0000","last_unstale":"2024-03-24T12:33:19.967607+0000","last_undegraded":"2024-03-24T12:33:19.967607+0000","last_fullsized":"2024-03-24T12:33:19.967607+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:28:45.877722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:45.877722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":359,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.174493+0000","last_change":"2024-03-24T12:32:59.916431+0000","last_active":"2024-03-24T12:33:25.174493+0000","last_peered":"2024-03-24T12:33:25.174493+0000","last_clean":"2024-03-24T12:33:25.174493+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:33:25.174493+0000","last_undegraded":"2024-03-24T12:33:25.174493+0000","last_fullsized":"2024-03-24T12:33:25.174493+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:32:59.916303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:59.916303+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:27.718672+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":236,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.159777+0000","last_change":"2024-03-24T12:33:08.288494+0000","last_active":"2024-03-24T12:33:25.159777+0000","last_peered":"2024-03-24T12:33:25.159777+0000","last_clean":"2024-03-24T12:33:25.159777+0000","last_became_active":"2024-03-24T12:26:07.337548+0000","last_became_peered":"2024-03-24T12:26:07.337548+0000","last_unstale":"2024-03-24T12:33:25.159777+0000","last_undegraded":"2024-03-24T12:33:25.159777+0000","last_fullsized":"2024-03-24T12:33:25.159777+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:08.288377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.288377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:08.963524+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"174'3353","reported_seq":3395,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:27.796439+0000","last_change":"2024-03-24T12:32:41.565670+0000","last_active":"2024-03-24T12:33:27.796439+0000","last_peered":"2024-03-24T12:33:27.796439+0000","last_clean":"2024-03-24T12:33:27.796439+0000","last_became_active":"2024-03-24T12:32:41.565379+0000","last_became_peered":"2024-03-24T12:32:41.565379+0000","last_unstale":"2024-03-24T12:33:27.796439+0000","last_undegraded":"2024-03-24T12:33:27.796439+0000","last_fullsized":"2024-03-24T12:33:27.796439+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3353,"log_dups_size":0,"ondisk_log_size":3353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:06.570548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219742208,"num_objects":3353,"num_object_clones":0,"num_object_copies":10059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3353,"num_write_kb":214592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":248,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.180193+0000","last_change":"2024-03-24T12:33:09.517074+0000","last_active":"2024-03-24T12:33:25.180193+0000","last_peered":"2024-03-24T12:33:25.180193+0000","last_clean":"2024-03-24T12:33:25.180193+0000","last_became_active":"2024-03-24T12:26:09.349435+0000","last_became_peered":"2024-03-24T12:26:09.349435+0000","last_unstale":"2024-03-24T12:33:25.180193+0000","last_undegraded":"2024-03-24T12:33:25.180193+0000","last_fullsized":"2024-03-24T12:33:25.180193+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:09.516956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:09.516956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:10.172189+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.170319+0000","last_change":"2024-03-24T12:29:10.434582+0000","last_active":"2024-03-24T12:33:25.170319+0000","last_peered":"2024-03-24T12:33:25.170319+0000","last_clean":"2024-03-24T12:33:25.170319+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:33:25.170319+0000","last_undegraded":"2024-03-24T12:33:25.170319+0000","last_fullsized":"2024-03-24T12:33:25.170319+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:10.434471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:10.434471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":691,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.953217+0000","last_change":"2024-03-24T12:28:53.153811+0000","last_active":"2024-03-24T12:33:25.953217+0000","last_peered":"2024-03-24T12:33:25.953217+0000","last_clean":"2024-03-24T12:33:25.953217+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:33:25.953217+0000","last_undegraded":"2024-03-24T12:33:25.953217+0000","last_fullsized":"2024-03-24T12:33:25.953217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:28:53.153687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:28:53.153687+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":437,"num_read_kb":432,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.173027+0000","last_change":"2024-03-24T12:32:50.752283+0000","last_active":"2024-03-24T12:33:25.173027+0000","last_peered":"2024-03-24T12:33:25.173027+0000","last_clean":"2024-03-24T12:33:25.173027+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:33:25.173027+0000","last_undegraded":"2024-03-24T12:33:25.173027+0000","last_fullsized":"2024-03-24T12:33:25.173027+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:32:50.752146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:50.752146+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:55.901641+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.162430+0000","last_change":"2024-03-24T12:33:10.102515+0000","last_active":"2024-03-24T12:33:25.162430+0000","last_peered":"2024-03-24T12:33:25.162430+0000","last_clean":"2024-03-24T12:33:25.162430+0000","last_became_active":"2024-03-24T12:26:07.326457+0000","last_became_peered":"2024-03-24T12:26:07.326457+0000","last_unstale":"2024-03-24T12:33:25.162430+0000","last_undegraded":"2024-03-24T12:33:25.162430+0000","last_fullsized":"2024-03-24T12:33:25.162430+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:10.102394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:10.102394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:10.605825+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"173'3114","reported_seq":3154,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:23.611087+0000","last_change":"2024-03-24T12:32:41.563958+0000","last_active":"2024-03-24T12:33:23.611087+0000","last_peered":"2024-03-24T12:33:23.611087+0000","last_clean":"2024-03-24T12:33:23.611087+0000","last_became_active":"2024-03-24T12:32:41.563612+0000","last_became_peered":"2024-03-24T12:32:41.563612+0000","last_unstale":"2024-03-24T12:33:23.611087+0000","last_undegraded":"2024-03-24T12:33:23.611087+0000","last_fullsized":"2024-03-24T12:33:23.611087+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3114,"log_dups_size":0,"ondisk_log_size":3114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:07.122354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204079104,"num_objects":3114,"num_object_clones":0,"num_object_copies":9342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3114,"num_write_kb":199296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.182753+0000","last_change":"2024-03-24T12:32:57.476369+0000","last_active":"2024-03-24T12:33:25.182753+0000","last_peered":"2024-03-24T12:33:25.182753+0000","last_clean":"2024-03-24T12:33:25.182753+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:33:25.182753+0000","last_undegraded":"2024-03-24T12:33:25.182753+0000","last_fullsized":"2024-03-24T12:33:25.182753+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:57.476212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:57.476212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:57.633314+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.180436+0000","last_change":"2024-03-24T12:32:53.384154+0000","last_active":"2024-03-24T12:33:25.180436+0000","last_peered":"2024-03-24T12:33:25.180436+0000","last_clean":"2024-03-24T12:33:25.180436+0000","last_became_active":"2024-03-24T12:26:03.167244+0000","last_became_peered":"2024-03-24T12:26:03.167244+0000","last_unstale":"2024-03-24T12:33:25.180436+0000","last_undegraded":"2024-03-24T12:33:25.180436+0000","last_fullsized":"2024-03-24T12:33:25.180436+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:53.384075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:53.384075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:08.368141+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":276,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.165978+0000","last_change":"2024-03-24T12:33:19.405563+0000","last_active":"2024-03-24T12:33:25.165978+0000","last_peered":"2024-03-24T12:33:25.165978+0000","last_clean":"2024-03-24T12:33:25.165978+0000","last_became_active":"2024-03-24T12:26:05.356745+0000","last_became_peered":"2024-03-24T12:26:05.356745+0000","last_unstale":"2024-03-24T12:33:25.165978+0000","last_undegraded":"2024-03-24T12:33:25.165978+0000","last_fullsized":"2024-03-24T12:33:25.165978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:33:19.405401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.405401+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.845660+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.173730+0000","last_change":"2024-03-24T12:32:50.951383+0000","last_active":"2024-03-24T12:33:25.173730+0000","last_peered":"2024-03-24T12:33:25.173730+0000","last_clean":"2024-03-24T12:33:25.173730+0000","last_became_active":"2024-03-24T12:26:07.337303+0000","last_became_peered":"2024-03-24T12:26:07.337303+0000","last_unstale":"2024-03-24T12:33:25.173730+0000","last_undegraded":"2024-03-24T12:33:25.173730+0000","last_fullsized":"2024-03-24T12:33:25.173730+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:50.951269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:32:50.951269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:58.720625+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"174'3390","reported_seq":3430,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.685202+0000","last_change":"2024-03-24T12:32:41.565895+0000","last_active":"2024-03-24T12:33:25.685202+0000","last_peered":"2024-03-24T12:33:25.685202+0000","last_clean":"2024-03-24T12:33:25.685202+0000","last_became_active":"2024-03-24T12:32:41.565563+0000","last_became_peered":"2024-03-24T12:32:41.565563+0000","last_unstale":"2024-03-24T12:33:25.685202+0000","last_undegraded":"2024-03-24T12:33:25.685202+0000","last_fullsized":"2024-03-24T12:33:25.685202+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:09.532720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222167040,"num_objects":3390,"num_object_clones":0,"num_object_copies":10170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3390,"num_write_kb":216960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.165156+0000","last_change":"2024-03-24T12:33:12.332284+0000","last_active":"2024-03-24T12:33:25.165156+0000","last_peered":"2024-03-24T12:33:25.165156+0000","last_clean":"2024-03-24T12:33:25.165156+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:33:25.165156+0000","last_undegraded":"2024-03-24T12:33:25.165156+0000","last_fullsized":"2024-03-24T12:33:25.165156+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:12.332162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.332162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:34.976750+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.161949+0000","last_change":"2024-03-24T12:33:11.083121+0000","last_active":"2024-03-24T12:33:25.161949+0000","last_peered":"2024-03-24T12:33:25.161949+0000","last_clean":"2024-03-24T12:33:25.161949+0000","last_became_active":"2024-03-24T12:26:03.168184+0000","last_became_peered":"2024-03-24T12:26:03.168184+0000","last_unstale":"2024-03-24T12:33:25.161949+0000","last_undegraded":"2024-03-24T12:33:25.161949+0000","last_fullsized":"2024-03-24T12:33:25.161949+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:11.083006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:11.083006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:14.119006+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"138'15","reported_seq":276,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.182222+0000","last_change":"2024-03-24T12:33:07.518493+0000","last_active":"2024-03-24T12:33:25.182222+0000","last_peered":"2024-03-24T12:33:25.182222+0000","last_clean":"2024-03-24T12:33:25.182222+0000","last_became_active":"2024-03-24T12:26:05.316739+0000","last_became_peered":"2024-03-24T12:26:05.316739+0000","last_unstale":"2024-03-24T12:33:25.182222+0000","last_undegraded":"2024-03-24T12:33:25.182222+0000","last_fullsized":"2024-03-24T12:33:25.182222+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:33:07.518368+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.518368+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.802570+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":333,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.165676+0000","last_change":"2024-03-24T12:32:54.818766+0000","last_active":"2024-03-24T12:33:25.165676+0000","last_peered":"2024-03-24T12:33:25.165676+0000","last_clean":"2024-03-24T12:33:25.165676+0000","last_became_active":"2024-03-24T12:26:07.336080+0000","last_became_peered":"2024-03-24T12:26:07.336080+0000","last_unstale":"2024-03-24T12:33:25.165676+0000","last_undegraded":"2024-03-24T12:33:25.165676+0000","last_fullsized":"2024-03-24T12:33:25.165676+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:32:54.818624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:54.818624+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:18.024216+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"174'3465","reported_seq":3507,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:27.433703+0000","last_change":"2024-03-24T12:32:41.564142+0000","last_active":"2024-03-24T12:33:27.433703+0000","last_peered":"2024-03-24T12:33:27.433703+0000","last_clean":"2024-03-24T12:33:27.433703+0000","last_became_active":"2024-03-24T12:32:41.563736+0000","last_became_peered":"2024-03-24T12:32:41.563736+0000","last_unstale":"2024-03-24T12:33:27.433703+0000","last_undegraded":"2024-03-24T12:33:27.433703+0000","last_fullsized":"2024-03-24T12:33:27.433703+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3465,"log_dups_size":0,"ondisk_log_size":3465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:07.677475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227082240,"num_objects":3465,"num_object_clones":0,"num_object_copies":10395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3465,"num_write_kb":221760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.159534+0000","last_change":"2024-03-24T12:29:05.078366+0000","last_active":"2024-03-24T12:33:25.159534+0000","last_peered":"2024-03-24T12:33:25.159534+0000","last_clean":"2024-03-24T12:33:25.159534+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:33:25.159534+0000","last_undegraded":"2024-03-24T12:33:25.159534+0000","last_fullsized":"2024-03-24T12:33:25.159534+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:05.078288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:05.078288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":245,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.173120+0000","last_change":"2024-03-24T12:29:08.601851+0000","last_active":"2024-03-24T12:33:25.173120+0000","last_peered":"2024-03-24T12:33:25.173120+0000","last_clean":"2024-03-24T12:33:25.173120+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:33:25.173120+0000","last_undegraded":"2024-03-24T12:33:25.173120+0000","last_fullsized":"2024-03-24T12:33:25.173120+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:29:08.601720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:08.601720+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":151,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"93'21","reported_seq":276,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175865+0000","last_change":"2024-03-24T12:33:20.268750+0000","last_active":"2024-03-24T12:33:25.175865+0000","last_peered":"2024-03-24T12:33:25.175865+0000","last_clean":"2024-03-24T12:33:25.175865+0000","last_became_active":"2024-03-24T12:26:05.315454+0000","last_became_peered":"2024-03-24T12:26:05.315454+0000","last_unstale":"2024-03-24T12:33:25.175865+0000","last_undegraded":"2024-03-24T12:33:25.175865+0000","last_fullsized":"2024-03-24T12:33:25.175865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'21","last_scrub_stamp":"2024-03-24T12:33:20.268587+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:33:20.268587+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:22.445291+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":331,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.971431+0000","last_change":"2024-03-24T12:28:52.872991+0000","last_active":"2024-03-24T12:33:19.971431+0000","last_peered":"2024-03-24T12:33:19.971431+0000","last_clean":"2024-03-24T12:33:19.971431+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:33:19.971431+0000","last_undegraded":"2024-03-24T12:33:19.971431+0000","last_fullsized":"2024-03-24T12:33:19.971431+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:28:52.872867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:28:52.872867+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"174'3170","reported_seq":3211,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.693803+0000","last_change":"2024-03-24T12:32:41.036940+0000","last_active":"2024-03-24T12:33:25.693803+0000","last_peered":"2024-03-24T12:33:25.693803+0000","last_clean":"2024-03-24T12:33:25.693803+0000","last_became_active":"2024-03-24T12:32:41.036596+0000","last_became_peered":"2024-03-24T12:32:41.036596+0000","last_unstale":"2024-03-24T12:33:25.693803+0000","last_undegraded":"2024-03-24T12:33:25.693803+0000","last_fullsized":"2024-03-24T12:33:25.693803+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3170,"log_dups_size":0,"ondisk_log_size":3170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:44.925063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":207749120,"num_objects":3170,"num_object_clones":0,"num_object_copies":9510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3170,"num_write_kb":202880,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175856+0000","last_change":"2024-03-24T12:31:13.677850+0000","last_active":"2024-03-24T12:33:25.175856+0000","last_peered":"2024-03-24T12:33:25.175856+0000","last_clean":"2024-03-24T12:33:25.175856+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:33:25.175856+0000","last_undegraded":"2024-03-24T12:33:25.175856+0000","last_fullsized":"2024-03-24T12:33:25.175856+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:13.677718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:31:13.677718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.969066+0000","last_change":"2024-03-24T12:33:15.388058+0000","last_active":"2024-03-24T12:33:19.969066+0000","last_peered":"2024-03-24T12:33:19.969066+0000","last_clean":"2024-03-24T12:33:19.969066+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:33:19.969066+0000","last_undegraded":"2024-03-24T12:33:19.969066+0000","last_fullsized":"2024-03-24T12:33:19.969066+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:15.387796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:15.387796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:20.872560+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":270,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.971298+0000","last_change":"2024-03-24T12:28:53.983340+0000","last_active":"2024-03-24T12:33:19.971298+0000","last_peered":"2024-03-24T12:33:19.971298+0000","last_clean":"2024-03-24T12:33:19.971298+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:33:19.971298+0000","last_undegraded":"2024-03-24T12:33:19.971298+0000","last_fullsized":"2024-03-24T12:33:19.971298+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T12:28:53.982595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:28:53.982595+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175724+0000","last_change":"2024-03-24T12:33:02.044769+0000","last_active":"2024-03-24T12:33:25.175724+0000","last_peered":"2024-03-24T12:33:25.175724+0000","last_clean":"2024-03-24T12:33:25.175724+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:33:25.175724+0000","last_undegraded":"2024-03-24T12:33:25.175724+0000","last_fullsized":"2024-03-24T12:33:25.175724+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:02.044666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.044666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.491534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"174'3503","reported_seq":3545,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:27.770541+0000","last_change":"2024-03-24T12:32:41.043057+0000","last_active":"2024-03-24T12:33:27.770541+0000","last_peered":"2024-03-24T12:33:27.770541+0000","last_clean":"2024-03-24T12:33:27.770541+0000","last_became_active":"2024-03-24T12:32:41.042732+0000","last_became_peered":"2024-03-24T12:32:41.042732+0000","last_unstale":"2024-03-24T12:33:27.770541+0000","last_undegraded":"2024-03-24T12:33:27.770541+0000","last_fullsized":"2024-03-24T12:33:27.770541+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3503,"log_dups_size":0,"ondisk_log_size":3503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:44.670186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229572608,"num_objects":3503,"num_object_clones":0,"num_object_copies":10509,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3503,"num_write_kb":224192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.161617+0000","last_change":"2024-03-24T12:29:12.576453+0000","last_active":"2024-03-24T12:33:25.161617+0000","last_peered":"2024-03-24T12:33:25.161617+0000","last_clean":"2024-03-24T12:33:25.161617+0000","last_became_active":"2024-03-24T12:26:09.351061+0000","last_became_peered":"2024-03-24T12:26:09.351061+0000","last_unstale":"2024-03-24T12:33:25.161617+0000","last_undegraded":"2024-03-24T12:33:25.161617+0000","last_fullsized":"2024-03-24T12:33:25.161617+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:12.576328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:12.576328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"140'9","reported_seq":342,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176835+0000","last_change":"2024-03-24T12:31:21.743353+0000","last_active":"2024-03-24T12:33:25.176835+0000","last_peered":"2024-03-24T12:33:25.176835+0000","last_clean":"2024-03-24T12:33:25.176835+0000","last_became_active":"2024-03-24T12:26:07.328595+0000","last_became_peered":"2024-03-24T12:26:07.328595+0000","last_unstale":"2024-03-24T12:33:25.176835+0000","last_undegraded":"2024-03-24T12:33:25.176835+0000","last_fullsized":"2024-03-24T12:33:25.176835+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'9","last_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:31:21.743220+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.971214+0000","last_change":"2024-03-24T12:33:08.247372+0000","last_active":"2024-03-24T12:33:19.971214+0000","last_peered":"2024-03-24T12:33:19.971214+0000","last_clean":"2024-03-24T12:33:19.971214+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:33:19.971214+0000","last_undegraded":"2024-03-24T12:33:19.971214+0000","last_fullsized":"2024-03-24T12:33:19.971214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:08.247253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.247253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:27.139927+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"92'22","reported_seq":255,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.166367+0000","last_change":"2024-03-24T12:33:02.599767+0000","last_active":"2024-03-24T12:33:25.166367+0000","last_peered":"2024-03-24T12:33:25.166367+0000","last_clean":"2024-03-24T12:33:25.166367+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:33:25.166367+0000","last_undegraded":"2024-03-24T12:33:25.166367+0000","last_fullsized":"2024-03-24T12:33:25.166367+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-24T12:33:02.599612+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.599612+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:07.009811+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"174'3453","reported_seq":3495,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:27.429791+0000","last_change":"2024-03-24T12:32:41.044757+0000","last_active":"2024-03-24T12:33:27.429791+0000","last_peered":"2024-03-24T12:33:27.429791+0000","last_clean":"2024-03-24T12:33:27.429791+0000","last_became_active":"2024-03-24T12:32:41.044207+0000","last_became_peered":"2024-03-24T12:32:41.044207+0000","last_unstale":"2024-03-24T12:33:27.429791+0000","last_undegraded":"2024-03-24T12:33:27.429791+0000","last_fullsized":"2024-03-24T12:33:27.429791+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3453,"log_dups_size":0,"ondisk_log_size":3453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:06.684718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226295808,"num_objects":3453,"num_object_clones":0,"num_object_copies":10359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3453,"num_write_kb":220992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":230,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:19.968638+0000","last_change":"2024-03-24T12:33:19.377017+0000","last_active":"2024-03-24T12:33:19.968638+0000","last_peered":"2024-03-24T12:33:19.968638+0000","last_clean":"2024-03-24T12:33:19.968638+0000","last_became_active":"2024-03-24T12:26:09.643739+0000","last_became_peered":"2024-03-24T12:26:09.643739+0000","last_unstale":"2024-03-24T12:33:19.968638+0000","last_undegraded":"2024-03-24T12:33:19.968638+0000","last_fullsized":"2024-03-24T12:33:19.968638+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:19.376894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.376894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:25.318752+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.162353+0000","last_change":"2024-03-24T12:33:12.115065+0000","last_active":"2024-03-24T12:33:25.162353+0000","last_peered":"2024-03-24T12:33:25.162353+0000","last_clean":"2024-03-24T12:33:25.162353+0000","last_became_active":"2024-03-24T12:26:07.332766+0000","last_became_peered":"2024-03-24T12:26:07.332766+0000","last_unstale":"2024-03-24T12:33:25.162353+0000","last_undegraded":"2024-03-24T12:33:25.162353+0000","last_fullsized":"2024-03-24T12:33:25.162353+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:12.114959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.114959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:21.136837+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.175755+0000","last_change":"2024-03-24T12:33:08.165239+0000","last_active":"2024-03-24T12:33:25.175755+0000","last_peered":"2024-03-24T12:33:25.175755+0000","last_clean":"2024-03-24T12:33:25.175755+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:33:25.175755+0000","last_undegraded":"2024-03-24T12:33:25.175755+0000","last_fullsized":"2024-03-24T12:33:25.175755+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:08.165107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.165107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.648796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":267,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176736+0000","last_change":"2024-03-24T12:32:55.028177+0000","last_active":"2024-03-24T12:33:25.176736+0000","last_peered":"2024-03-24T12:33:25.176736+0000","last_clean":"2024-03-24T12:33:25.176736+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:33:25.176736+0000","last_undegraded":"2024-03-24T12:33:25.176736+0000","last_fullsized":"2024-03-24T12:33:25.176736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:32:55.028060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.028060+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:59.146480+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"174'3522","reported_seq":3564,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:27.823983+0000","last_change":"2024-03-24T12:32:41.564197+0000","last_active":"2024-03-24T12:33:27.823983+0000","last_peered":"2024-03-24T12:33:27.823983+0000","last_clean":"2024-03-24T12:33:27.823983+0000","last_became_active":"2024-03-24T12:32:41.563886+0000","last_became_peered":"2024-03-24T12:32:41.563886+0000","last_unstale":"2024-03-24T12:33:27.823983+0000","last_undegraded":"2024-03-24T12:33:27.823983+0000","last_fullsized":"2024-03-24T12:33:27.823983+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3522,"log_dups_size":0,"ondisk_log_size":3522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:41.261261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230817792,"num_objects":3522,"num_object_clones":0,"num_object_copies":10566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3522,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3522,"num_write_kb":225408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.166538+0000","last_change":"2024-03-24T12:32:56.667268+0000","last_active":"2024-03-24T12:33:25.166538+0000","last_peered":"2024-03-24T12:33:25.166538+0000","last_clean":"2024-03-24T12:33:25.166538+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:33:25.166538+0000","last_undegraded":"2024-03-24T12:33:25.166538+0000","last_fullsized":"2024-03-24T12:33:25.166538+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:56.667140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:56.667140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:03.052496+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.180698+0000","last_change":"2024-03-24T12:33:13.797360+0000","last_active":"2024-03-24T12:33:25.180698+0000","last_peered":"2024-03-24T12:33:25.180698+0000","last_clean":"2024-03-24T12:33:25.180698+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:33:25.180698+0000","last_undegraded":"2024-03-24T12:33:25.180698+0000","last_fullsized":"2024-03-24T12:33:25.180698+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:13.797243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:13.797243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:17.636222+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.173532+0000","last_change":"2024-03-24T12:32:54.886366+0000","last_active":"2024-03-24T12:33:25.173532+0000","last_peered":"2024-03-24T12:33:25.173532+0000","last_clean":"2024-03-24T12:33:25.173532+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:33:25.173532+0000","last_undegraded":"2024-03-24T12:33:25.173532+0000","last_fullsized":"2024-03-24T12:33:25.173532+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:54.886269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:54.886269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:20.523417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"94'25","reported_seq":276,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176971+0000","last_change":"2024-03-24T12:31:09.668493+0000","last_active":"2024-03-24T12:33:25.176971+0000","last_peered":"2024-03-24T12:33:25.176971+0000","last_clean":"2024-03-24T12:33:25.176971+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:33:25.176971+0000","last_undegraded":"2024-03-24T12:33:25.176971+0000","last_fullsized":"2024-03-24T12:33:25.176971+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'25","last_scrub_stamp":"2024-03-24T12:31:09.668392+0000","last_deep_scrub":"94'25","last_deep_scrub_stamp":"2024-03-24T12:28:02.436198+0000","last_clean_scrub_stamp":"2024-03-24T12:31:09.668392+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"173'3129","reported_seq":3168,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:23.631037+0000","last_change":"2024-03-24T12:32:41.040868+0000","last_active":"2024-03-24T12:33:23.631037+0000","last_peered":"2024-03-24T12:33:23.631037+0000","last_clean":"2024-03-24T12:33:23.631037+0000","last_became_active":"2024-03-24T12:32:41.040533+0000","last_became_peered":"2024-03-24T12:32:41.040533+0000","last_unstale":"2024-03-24T12:33:23.631037+0000","last_undegraded":"2024-03-24T12:33:23.631037+0000","last_fullsized":"2024-03-24T12:33:23.631037+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":3129,"log_dups_size":0,"ondisk_log_size":3129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:33:41.803591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205062144,"num_objects":3129,"num_object_clones":0,"num_object_copies":9387,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3129,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3129,"num_write_kb":200256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":248,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176956+0000","last_change":"2024-03-24T12:33:10.193697+0000","last_active":"2024-03-24T12:33:25.176956+0000","last_peered":"2024-03-24T12:33:25.176956+0000","last_clean":"2024-03-24T12:33:25.176956+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:33:25.176956+0000","last_undegraded":"2024-03-24T12:33:25.176956+0000","last_fullsized":"2024-03-24T12:33:25.176956+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:10.193581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:10.193581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:23.343839+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":342,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.162864+0000","last_change":"2024-03-24T12:32:57.614709+0000","last_active":"2024-03-24T12:33:25.162864+0000","last_peered":"2024-03-24T12:33:25.162864+0000","last_clean":"2024-03-24T12:33:25.162864+0000","last_became_active":"2024-03-24T12:26:07.334798+0000","last_became_peered":"2024-03-24T12:26:07.334798+0000","last_unstale":"2024-03-24T12:33:25.162864+0000","last_undegraded":"2024-03-24T12:33:25.162864+0000","last_fullsized":"2024-03-24T12:33:25.162864+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:32:57.614572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:57.614572+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:16.763319+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":242,"reported_epoch":173,"state":"active+clean","last_fresh":"2024-03-24T12:33:21.383796+0000","last_change":"2024-03-24T12:33:21.383831+0000","last_active":"2024-03-24T12:33:21.383796+0000","last_peered":"2024-03-24T12:33:21.383796+0000","last_clean":"2024-03-24T12:33:21.383796+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:33:21.383796+0000","last_undegraded":"2024-03-24T12:33:21.383796+0000","last_fullsized":"2024-03-24T12:33:21.383796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.383703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.383703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:28.813196+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":263,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-24T12:33:25.176576+0000","last_change":"2024-03-24T12:31:17.834752+0000","last_active":"2024-03-24T12:33:25.176576+0000","last_peered":"2024-03-24T12:33:25.176576+0000","last_clean":"2024-03-24T12:33:25.176576+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:33:25.176576+0000","last_undegraded":"2024-03-24T12:33:25.176576+0000","last_fullsized":"2024-03-24T12:33:25.176576+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:31:17.834622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:31:17.834622+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3532521472,"num_objects":53902,"num_object_clones":0,"num_object_copies":161706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53902,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53902,"num_write_kb":3449728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737352704,"data_stored":10737352704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53902,"ondisk_log_size":53902,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":437,"num_read_kb":432,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037853,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1294112,"kb_used_data":1096344,"kb_used_omap":1,"kb_used_meta":197758,"kb_avail":92451040,"statfs":{"total":95995035648,"available":94669864960,"internally_reserved":0,"allocated":1122656256,"data_stored":1122175172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":202504650},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332195,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1284488,"kb_used_data":1100608,"kb_used_omap":1,"kb_used_meta":183870,"kb_avail":92460664,"statfs":{"total":95995035648,"available":94679719936,"internally_reserved":0,"allocated":1127022592,"data_stored":1126565496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":188283335},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574974,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1707000,"kb_used_data":1531440,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":92038152,"statfs":{"total":95995035648,"available":94247067648,"internally_reserved":0,"allocated":1568194560,"data_stored":1567688746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1544292,"kb_used_data":1256280,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":92200860,"statfs":{"total":95995035648,"available":94413680640,"internally_reserved":0,"allocated":1286430720,"data_stored":1285948980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":294910405},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411448,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1598724,"kb_used_data":1346812,"kb_used_omap":1,"kb_used_meta":251902,"kb_avail":92146428,"statfs":{"total":95995035648,"available":94357942272,"internally_reserved":0,"allocated":1379135488,"data_stored":1378681404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":257948101},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117104,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1372640,"kb_used_data":1170712,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92372512,"statfs":{"total":95995035648,"available":94589452288,"internally_reserved":0,"allocated":1198809088,"data_stored":1198320923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":206764487},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,0,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920877,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1271260,"kb_used_data":1109588,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":92473892,"statfs":{"total":95995035648,"available":94693265408,"internally_reserved":0,"allocated":1136218112,"data_stored":1135739722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626536,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1898896,"kb_used_data":1755272,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":91846256,"statfs":{"total":95995035648,"available":94050566144,"internally_reserved":0,"allocated":1797398528,"data_stored":1796933530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1300496384,"data_stored":1300496384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1601044480,"data_stored":1601044480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1404633088,"data_stored":1404633088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1207304192,"data_stored":1207304192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1139212288,"data_stored":1139212288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1815674880,"data_stored":1815674880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1139933184,"data_stored":1139933184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1129054208,"data_stored":1129054208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:33:32.879 INFO:tasks.thrashosds.thrasher:Setting 8.0 to [3, 5, 2] 2024-03-24T12:33:32.879 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '8.0', '3', '5', '2'] 2024-03-24T12:33:32.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pg-upmap 8.0 3 5 2 2024-03-24T12:33:32.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:32.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:32.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:32.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:33.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:33.147 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:33.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:33.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:33.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:33.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:33.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:33.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[29919]: 5.13 deep-scrub starts 2024-03-24T12:33:33.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[29919]: 5.13 deep-scrub ok 2024-03-24T12:33:33.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:33:33.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:33:33.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[29919]: pgmap v298: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-03-24T12:33:33.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:33:33.402 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[41500]: 5.13 deep-scrub starts 2024-03-24T12:33:33.402 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[41500]: 5.13 deep-scrub ok 2024-03-24T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[41500]: pgmap v298: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-03-24T12:33:33.403 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:33:33.412 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:33:33.412 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:33:33.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:33.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:33.552 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:33:33.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:33.597 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:33:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:33.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:33 smithi190 ceph-mon[33748]: 5.13 deep-scrub starts 2024-03-24T12:33:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:33 smithi190 ceph-mon[33748]: 5.13 deep-scrub ok 2024-03-24T12:33:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:33 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:33:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:33 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:33:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:33 smithi190 ceph-mon[33748]: pgmap v298: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 93 MiB/s wr, 1.48k op/s 2024-03-24T12:33:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:33:33.641 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:33:33.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 38 16 64256 64240 105.647 73.6875 0.00965577 0.00946009 2024-03-24T12:33:33.684 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:33:33.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:33.729 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:33:33.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:33.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:33.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:33.778 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:33:33.823 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:33:33.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:33.870 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:33:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:33:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:33:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:33:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:33:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:33:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:33:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:33:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:33:33.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:33.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:33.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:33.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:34.029 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:34.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:34.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:34.173 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:34.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:34.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:34.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:34.316 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:34.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:34.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:34.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:34.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:34.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:34.608 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:34.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:34.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:34 smithi190 ceph-mon[33748]: 3.1d scrub starts 2024-03-24T12:33:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:34 smithi190 ceph-mon[33748]: 3.1d scrub ok 2024-03-24T12:33:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:34 smithi190 ceph-mon[33748]: 4.0 scrub starts 2024-03-24T12:33:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:34 smithi190 ceph-mon[33748]: 4.0 scrub ok 2024-03-24T12:33:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:34 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1463458670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:33:34.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:34.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[41500]: 3.1d scrub starts 2024-03-24T12:33:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[41500]: 3.1d scrub ok 2024-03-24T12:33:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[41500]: 4.0 scrub starts 2024-03-24T12:33:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[41500]: 4.0 scrub ok 2024-03-24T12:33:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1463458670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:33:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:34.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:34.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[29919]: 3.1d scrub starts 2024-03-24T12:33:34.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[29919]: 3.1d scrub ok 2024-03-24T12:33:34.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[29919]: 4.0 scrub starts 2024-03-24T12:33:34.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[29919]: 4.0 scrub ok 2024-03-24T12:33:34.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1463458670' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:33:34.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:34.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:34.673 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 39 16 65504 65488 104.937 78 0.0107167 0.00952428 2024-03-24T12:33:34.710 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:34.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:34.753 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:34.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:34.854 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:34.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:34.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:34.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:35.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:35.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:35.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:35.191 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:35.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:35.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:35.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:35.336 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:35.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:35.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: 6.a scrub starts 2024-03-24T12:33:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: 6.a scrub ok 2024-03-24T12:33:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: 3.d deep-scrub starts 2024-03-24T12:33:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: 3.d deep-scrub ok 2024-03-24T12:33:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: 3.1 deep-scrub starts 2024-03-24T12:33:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: 3.1 deep-scrub ok 2024-03-24T12:33:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: 4.1a scrub starts 2024-03-24T12:33:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: 4.1a scrub ok 2024-03-24T12:33:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:35 smithi190 ceph-mon[33748]: pgmap v299: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-24T12:33:35.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: 6.a scrub starts 2024-03-24T12:33:35.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: 6.a scrub ok 2024-03-24T12:33:35.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: 3.d deep-scrub starts 2024-03-24T12:33:35.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: 3.d deep-scrub ok 2024-03-24T12:33:35.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: 3.1 deep-scrub starts 2024-03-24T12:33:35.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: 3.1 deep-scrub ok 2024-03-24T12:33:35.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: 4.1a scrub starts 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: 4.1a scrub ok 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[29919]: pgmap v299: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: 6.a scrub starts 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: 6.a scrub ok 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: 3.d deep-scrub starts 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: 3.d deep-scrub ok 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: 3.1 deep-scrub starts 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: 3.1 deep-scrub ok 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:35.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: 4.1a scrub starts 2024-03-24T12:33:35.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: 4.1a scrub ok 2024-03-24T12:33:35.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:35 smithi029 ceph-mon[41500]: pgmap v299: 148 pgs: 148 active+clean; 3.8 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.1 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-24T12:33:35.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:35.480 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:35.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:35.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:35.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:35.622 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:35.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:35.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:33:35.678368+0000 min lat: 0.00120017 max lat: 0.373948 avg lat: 0.00957368 2024-03-24T12:33:35.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:33:35.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 40 16 66838 66822 104.398 83.375 0.00932711 0.00957368 2024-03-24T12:33:35.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:35.760 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:35.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:35.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:35.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:35.908 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:35.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:36.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:36.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:36.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:36.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:36.187 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:36.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:36.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:36.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:36.323 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:36.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:36.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:36.376 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:33:36.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:33:36.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:33:36.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[41500]: 5.0 scrub starts 2024-03-24T12:33:36.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[41500]: 5.0 scrub ok 2024-03-24T12:33:36.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:33:36.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:33:36.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:33:36.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:33:36.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[29919]: 5.0 scrub starts 2024-03-24T12:33:36.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[29919]: 5.0 scrub ok 2024-03-24T12:33:36.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:33:36.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:36 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:33:36.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:36.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:36.466 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:36.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:36.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:36.603 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:36.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:36.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:36.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:36 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:33:36.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:36 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:33:36.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:36 smithi190 ceph-mon[33748]: 5.0 scrub starts 2024-03-24T12:33:36.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:36 smithi190 ceph-mon[33748]: 5.0 scrub ok 2024-03-24T12:33:36.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:36 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:33:36.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:36 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:33:36.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 41 16 68200 68184 103.927 85.125 0.0161864 0.009617 2024-03-24T12:33:36.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:36.742 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:36.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:36.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:36.844 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:36.882 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:36.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:36.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:37.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:37.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:37.121 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:37.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:33:37.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:37.164 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:37.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:37.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:37.299 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:37.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:37.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:37.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-mon[29919]: 6.d scrub starts 2024-03-24T12:33:37.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-mon[29919]: 6.d scrub ok 2024-03-24T12:33:37.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-mon[29919]: 4.19 scrub starts 2024-03-24T12:33:37.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-mon[29919]: 4.19 scrub ok 2024-03-24T12:33:37.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-mon[29919]: pgmap v300: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-24T12:33:37.410 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-mon[41500]: 6.d scrub starts 2024-03-24T12:33:37.410 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-mon[41500]: 6.d scrub ok 2024-03-24T12:33:37.410 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-mon[41500]: 4.19 scrub starts 2024-03-24T12:33:37.410 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-mon[41500]: 4.19 scrub ok 2024-03-24T12:33:37.410 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-mon[41500]: pgmap v300: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-24T12:33:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:37.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:37.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:37.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:37.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:37.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:37.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:37 smithi190 ceph-mon[33748]: 6.d scrub starts 2024-03-24T12:33:37.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:37 smithi190 ceph-mon[33748]: 6.d scrub ok 2024-03-24T12:33:37.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:37 smithi190 ceph-mon[33748]: 4.19 scrub starts 2024-03-24T12:33:37.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:37 smithi190 ceph-mon[33748]: 4.19 scrub ok 2024-03-24T12:33:37.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:37 smithi190 ceph-mon[33748]: pgmap v300: 148 pgs: 148 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.48k op/s 2024-03-24T12:33:37.670 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:33:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:33:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:33:37.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 42 16 69500 69484 103.387 81.25 0.00696596 0.00966734 2024-03-24T12:33:37.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:37.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:37.725 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:37.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:37.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:37.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:37.874 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:37.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:37.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:38.028 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:38.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:38.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:38.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:33:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:33:38.174 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:38.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:38.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:38.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:38.317 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:33:38.318 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:33:38.327 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:38.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:38.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:38.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:38.465 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:38.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:38.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:38.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:38.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:38.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:38 smithi190 ceph-mon[33748]: 5.1a scrub starts 2024-03-24T12:33:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:38 smithi190 ceph-mon[33748]: 5.1a scrub ok 2024-03-24T12:33:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:38 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:38.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:38 smithi029 ceph-mon[41500]: 5.1a scrub starts 2024-03-24T12:33:38.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:38 smithi029 ceph-mon[41500]: 5.1a scrub ok 2024-03-24T12:33:38.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:38 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:38.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:38 smithi029 ceph-mon[29919]: 5.1a scrub starts 2024-03-24T12:33:38.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:38 smithi029 ceph-mon[29919]: 5.1a scrub ok 2024-03-24T12:33:38.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:38 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:38.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 43 16 70670 70654 102.683 73.125 0.0214439 0.00973373 2024-03-24T12:33:38.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:38.739 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:38.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:38.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:38.873 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:38.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:38.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:38.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:39.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:39.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:39.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:39.111 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:39.162 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:39.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:39.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:39.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:39.302 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:39.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:39.314 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:33:39.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:39.450 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:39.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:39.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T12:33:39.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:39.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:39 smithi029 ceph-mon[29919]: 6.3 scrub starts 2024-03-24T12:33:39.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:39 smithi029 ceph-mon[29919]: 6.3 scrub ok 2024-03-24T12:33:39.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:39 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2914336683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:33:39.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:39 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:33:39.586 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:39 smithi029 ceph-mon[29919]: pgmap v301: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T12:33:39.586 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:39 smithi029 ceph-mon[41500]: 6.3 scrub starts 2024-03-24T12:33:39.586 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:39 smithi029 ceph-mon[41500]: 6.3 scrub ok 2024-03-24T12:33:39.586 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:39 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2914336683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:33:39.586 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:39 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:33:39.587 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:39 smithi029 ceph-mon[41500]: pgmap v301: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T12:33:39.590 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:39.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:39.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:39 smithi190 ceph-mon[33748]: 6.3 scrub starts 2024-03-24T12:33:39.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:39 smithi190 ceph-mon[33748]: 6.3 scrub ok 2024-03-24T12:33:39.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:39 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2914336683' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:33:39.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:39 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:33:39.599 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:39 smithi190 ceph-mon[33748]: pgmap v301: 148 pgs: 148 active+clean; 4.1 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-03-24T12:33:39.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 44 16 71893 71877 102.086 76.4375 0.0081255 0.00978967 2024-03-24T12:33:39.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:39.734 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:39.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:39.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:39.869 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:39.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:39.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:39.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:40.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:40.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:40.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:40.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:40.150 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:40.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:40.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:40.292 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:40.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:40.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:40.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:40.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:40.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:40 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:33:40.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:40 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:33:40.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:40 smithi190 ceph-mon[33748]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T12:33:40.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:40 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:33:40.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:40 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:33:40.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:40 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:33:40.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:40 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:33:40.556 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:40 smithi190 ceph-mon[33748]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T12:33:40.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:40.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:40.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:40.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:40.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:33:40.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:33:40.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[29919]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T12:33:40.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:33:40.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:33:40.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:33:40.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:33:40.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[29919]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T12:33:40.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:33:40.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:33:40.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[41500]: osdmap e176: 8 total, 8 up, 8 in 2024-03-24T12:33:40.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:33:40.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:33:40.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:33:40.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:33:40.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:40 smithi029 ceph-mon[41500]: osdmap e177: 8 total, 8 up, 8 in 2024-03-24T12:33:40.674 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 45 16 73021 73005 101.384 70.5 0.0112127 0.00985758 2024-03-24T12:33:40.702 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:40.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:40.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:40.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:40.851 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:40.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:40.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:40.978 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:40.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:41.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:41.129 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:41.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:41.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:41.258 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:41.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:41.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:41.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:41.401 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:41.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:41.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:41 smithi029 ceph-mon[29919]: pgmap v304: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-24T12:33:41.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:41 smithi029 ceph-mon[41500]: pgmap v304: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-24T12:33:41.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:41.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:41.535 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:41.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:41.585 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:33:41.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:41 smithi190 ceph-mon[33748]: pgmap v304: 148 pgs: 148 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 75 MiB/s wr, 1.21k op/s 2024-03-24T12:33:41.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:41.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:41.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:41.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:41.675 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 46 16 74156 74140 100.722 70.9375 0.00725424 0.0099232 2024-03-24T12:33:41.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:41.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:41.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:41.845 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:33:41.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:41.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:41.954 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:41.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:42.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:42.078 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:42.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:42.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:42.221 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:42.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:42.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:42.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:42.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:42.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:42.500 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:42.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:42.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:42.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:42.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:42 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:33:42.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:42 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:33:42.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:42 smithi190 ceph-mon[33748]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T12:33:42.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:42 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:42.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:42 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:42.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:42.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:42.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:42.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:42 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:33:42.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:42 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:33:42.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:42 smithi029 ceph-mon[29919]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T12:33:42.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:42 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:42.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:42 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:42.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:42 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:33:42.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:42 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:33:42.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:42 smithi029 ceph-mon[41500]: osdmap e178: 8 total, 8 up, 8 in 2024-03-24T12:33:42.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:42 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:42.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:42 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:42.675 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 47 16 75222 75206 99.9963 66.625 0.0103686 0.00999571 2024-03-24T12:33:42.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:42.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:42.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:42.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:42.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:42.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:42.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:43.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:43.037 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:43.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:43.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:43.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:43.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:43.176 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:43.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:43.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:43.316 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:43.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:43.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:43.417 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:43.441 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:43.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:43.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:43.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:33:43.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:43.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:43.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:43.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:43 smithi029 ceph-mon[29919]: pgmap v306: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T12:33:43.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:43 smithi029 ceph-mon[41500]: pgmap v306: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T12:33:43.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:43 smithi190 ceph-mon[33748]: pgmap v306: 148 pgs: 148 active+clean; 4.4 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 89 MiB/s wr, 1.42k op/s 2024-03-24T12:33:43.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:43.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 48 16 76395 76379 99.4348 73.3125 0.0101896 0.010051 2024-03-24T12:33:43.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:43.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:43.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:43.842 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:43.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:43.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:43.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:43.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:43.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:43.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:44.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:44.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:44.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:44.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:44.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:44.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:44.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:44.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:44.375 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:44.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:44.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:44.500 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:44.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:44.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:44.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:44.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:44.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:44.678 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 49 16 77313 77297 98.5764 57.375 0.00771584 0.0101392 2024-03-24T12:33:44.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:44.767 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:44.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:44.870 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:44.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:44.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:45.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:45.026 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:45.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:45.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:45.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:45.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:45.162 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:45.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:45.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:45.304 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:45.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:45.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:45.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:45.436 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:45.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:45.437 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:45.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:45.561 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:45.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:45.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:45.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 50 16 78241 78225 97.7625 58 0.0167519 0.0102211 2024-03-24T12:33:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:45.700 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:45.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:45.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:45.835 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:45.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:45.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:45.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:45.980 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:45.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:45.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:46.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:46.128 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:46.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:46.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:46.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:46.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:46.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:46.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:46.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:46.378 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:46.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:46.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:46.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:45 smithi190 ceph-mon[33748]: pgmap v307: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:33:46.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:45 smithi190 ceph-mon[33748]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T12:33:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:45 smithi029 ceph-mon[29919]: pgmap v307: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:33:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:45 smithi029 ceph-mon[29919]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T12:33:46.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:45 smithi029 ceph-mon[41500]: pgmap v307: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-24T12:33:46.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:45 smithi029 ceph-mon[41500]: osdmap e179: 8 total, 8 up, 8 in 2024-03-24T12:33:46.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:46.511 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:46.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:46.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:46.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:46.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:46.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:46.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 51 16 79239 79223 97.0685 62.375 0.00296759 0.0102925 2024-03-24T12:33:46.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:46.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:46.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:46.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:46.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:46.929 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:46.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:47.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:47.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:47.066 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:47.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:47.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:47.205 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:47.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:47.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:47.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:47.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:47.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:47.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:47.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:47.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:47.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:47.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:47.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:47.606 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:47.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:47.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:47.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 52 16 80306 80290 96.4842 66.6875 0.0139253 0.0103593 2024-03-24T12:33:47.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:47.746 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:47.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:47.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:47.848 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:47.891 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:47.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:47.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:47.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:33:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:33:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:33:47.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:48.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:48.032 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:48.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:48.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:48.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:33:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:33:48.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:47 smithi190 ceph-mon[33748]: pgmap v309: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 781 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T12:33:48.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:48.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:48.173 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:48.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:48.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:48.275 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:48.314 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:48.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:48.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:48.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:47 smithi029 ceph-mon[41500]: pgmap v309: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 781 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T12:33:48.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:48.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:47 smithi029 ceph-mon[29919]: pgmap v309: 148 pgs: 148 active+clean; 4.6 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 781 B/s rd, 65 MiB/s wr, 1.04k op/s 2024-03-24T12:33:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:33:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:33:48.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:48.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:48.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:48.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:48.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:48.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:48.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 53 16 81381 81365 95.9313 67.1875 0.00252293 0.0103864 2024-03-24T12:33:48.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:48.704 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:48.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:48.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:48.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:48.850 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:48.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:48.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:48.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:49.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:49.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:49.124 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:49.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:49.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:49.248 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:49.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:49.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:49.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:49.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:49.386 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:49.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:49.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:49.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:49.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:49.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:49.512 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:49.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:49.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:49.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:49.636 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:49.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:49.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 54 16 81870 81854 94.7208 30.5625 0.00248391 0.0104932 2024-03-24T12:33:49.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:49.763 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:49.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:49.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:49.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:49.890 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:49.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:49.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:49.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:50.028 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:50.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:50.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:50.167 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:50.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:50.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:50.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:50.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:50.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:50 smithi190 ceph-mon[33748]: pgmap v310: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:33:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:50 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3549810939' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.0", "id": [3, 5, 2]}]: dispatch 2024-03-24T12:33:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:50 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.0", "id": [3, 5, 2]}]: dispatch 2024-03-24T12:33:50.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:50.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:50 smithi029 ceph-mon[41500]: pgmap v310: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:33:50.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:50 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3549810939' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.0", "id": [3, 5, 2]}]: dispatch 2024-03-24T12:33:50.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:50 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.0", "id": [3, 5, 2]}]: dispatch 2024-03-24T12:33:50.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:50 smithi029 ceph-mon[29919]: pgmap v310: 148 pgs: 148 active+clean; 4.8 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:33:50.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:50 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3549810939' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.0", "id": [3, 5, 2]}]: dispatch 2024-03-24T12:33:50.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:50 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.0", "id": [3, 5, 2]}]: dispatch 2024-03-24T12:33:50.422 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:50.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:50.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:50.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:50.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:50.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:50.673 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:50.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:50.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 55 16 82144 82128 93.3101 17.125 0.00256808 0.0106175 2024-03-24T12:33:50.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:50.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:50.799 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:50.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:50.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:50.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:51.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:51.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:51.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:51.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:51.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:51.177 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:51.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:51.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:51.306 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:51.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:51.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:51.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:51 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.0", "id": [3, 5, 2]}]': finished 2024-03-24T12:33:51.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:51 smithi190 ceph-mon[33748]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T12:33:51.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:51 smithi190 ceph-mon[33748]: pgmap v312: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 951 op/s 2024-03-24T12:33:51.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:51.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:51 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.0", "id": [3, 5, 2]}]': finished 2024-03-24T12:33:51.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:51 smithi029 ceph-mon[29919]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T12:33:51.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:51 smithi029 ceph-mon[29919]: pgmap v312: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 951 op/s 2024-03-24T12:33:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:51 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.0", "id": [3, 5, 2]}]': finished 2024-03-24T12:33:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:51 smithi029 ceph-mon[41500]: osdmap e180: 8 total, 8 up, 8 in 2024-03-24T12:33:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:51 smithi029 ceph-mon[41500]: pgmap v312: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 59 MiB/s wr, 951 op/s 2024-03-24T12:33:51.433 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:51.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:51.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:51.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:51.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:51.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:51.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:51.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 56 16 82144 82128 91.644 0 - 0.0106175 2024-03-24T12:33:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:51.689 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:51.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:51.790 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:51.813 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:51.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:51.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:51.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:51.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:51.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:52.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:52.067 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:52.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:52.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:52.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:52.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:52.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:52.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:52.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:52.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:52 smithi190 ceph-mon[33748]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T12:33:52.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:52 smithi029 ceph-mon[41500]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T12:33:52.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:52 smithi029 ceph-mon[29919]: osdmap e181: 8 total, 8 up, 8 in 2024-03-24T12:33:52.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:52.446 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:52.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:52.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:52.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:52.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:52.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:52.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:52.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 57 16 82177 82161 90.0726 1.03125 0.00394849 0.0108225 2024-03-24T12:33:52.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:52.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:52.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:52.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:52.840 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:52.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:52.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:52.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:52.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:53.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:53.110 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:53.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:53.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:53.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:53.234 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:53.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:53.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:53.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:53.381 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:53.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:53 smithi190 ceph-mon[33748]: 8.0 starting backfill to osd.2 from (0'0,0'0] MAX to 179'5175 2024-03-24T12:33:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:53 smithi190 ceph-mon[33748]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T12:33:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:53 smithi190 ceph-mon[33748]: pgmap v315: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 936 op/s 2024-03-24T12:33:53.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:53 smithi029 ceph-mon[41500]: 8.0 starting backfill to osd.2 from (0'0,0'0] MAX to 179'5175 2024-03-24T12:33:53.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:53 smithi029 ceph-mon[41500]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T12:33:53.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:53 smithi029 ceph-mon[41500]: pgmap v315: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 936 op/s 2024-03-24T12:33:53.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:53 smithi029 ceph-mon[29919]: 8.0 starting backfill to osd.2 from (0'0,0'0] MAX to 179'5175 2024-03-24T12:33:53.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:53 smithi029 ceph-mon[29919]: osdmap e182: 8 total, 8 up, 8 in 2024-03-24T12:33:53.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:53 smithi029 ceph-mon[29919]: pgmap v315: 148 pgs: 148 active+clean; 5.0 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 58 MiB/s wr, 936 op/s 2024-03-24T12:33:53.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:53.522 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:53.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:53.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:53.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:53.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:53.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:53.679 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 58 15 83014 82999 89.4226 52.375 0.00795511 0.0111774 2024-03-24T12:33:53.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:53.780 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:53.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:53.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:53.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:53.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:53.905 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:53.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:54.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:54.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:54.036 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:54.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:54.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:54.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:54.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:54.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:54.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:54.301 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:54.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:54.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:54.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:54.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:54.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:54.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:54.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:54.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:54.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:54.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:54.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 59 16 83915 83899 88.8603 56.25 0.013149 0.011248 2024-03-24T12:33:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:54.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:54.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:54.853 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:54.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:54.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:54.988 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:54.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:55.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:55.131 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:55.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:55.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:55.270 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:55.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:55.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:55.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:55.400 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:55.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:55.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:55.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:55.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:55.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:55.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:55.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:55.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:55.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:55.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:55.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:33:55.684390+0000 min lat: 0.00120017 max lat: 2.54295 avg lat: 0.0113418 2024-03-24T12:33:55.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:33:55.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 60 16 84636 84620 88.1303 45.0625 0.00593994 0.0113418 2024-03-24T12:33:55.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:55.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:55.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:55.905 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:55.944 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:55.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:56.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:56.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:56.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:56.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:56.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:56.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:56.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:set 8.0 pg_upmap mapping to [3,5,2] 2024-03-24T12:33:56.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:55 smithi029 ceph-mon[29919]: pgmap v316: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 33 MiB/s wr, 536 op/s; 5150/247716 objects misplaced (2.079%); 970 KiB/s, 15 objects/s recovering 2024-03-24T12:33:56.306 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:55 smithi029 ceph-mon[41500]: pgmap v316: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 33 MiB/s wr, 536 op/s; 5150/247716 objects misplaced (2.079%); 970 KiB/s, 15 objects/s recovering 2024-03-24T12:33:56.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:56.348 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:56.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:55 smithi190 ceph-mon[33748]: pgmap v316: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 33 MiB/s wr, 536 op/s; 5150/247716 objects misplaced (2.079%); 970 KiB/s, 15 objects/s recovering 2024-03-24T12:33:56.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:56.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:56.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:56.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:56.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:56.619 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:56.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:56.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:56.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 61 16 85825 85809 87.9037 74.3125 0.00510601 0.0113717 2024-03-24T12:33:56.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:56.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:56.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:56.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:56.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:56.897 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:56.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:56.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:56.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:57.041 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:57.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:57.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:57.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:57.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:57.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:57.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:33:57.335 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:57.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:57.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:33:57.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:57.475 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:57.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:57.576 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:57.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:57.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:57.669 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:33:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:33:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:33:57.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 62 16 86959 86943 87.629 70.875 0.00945235 0.011406 2024-03-24T12:33:57.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:57.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:57.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:57.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:57.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:57.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:57.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:58.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:58.014 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:58.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:58.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:58.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:57 smithi190 ceph-mon[33748]: pgmap v317: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 795 B/s rd, 27 MiB/s wr, 430 op/s; 5150/249135 objects misplaced (2.067%); 905 KiB/s, 14 objects/s recovering 2024-03-24T12:33:58.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:33:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:33:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:58.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:58.237 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:57 smithi029 ceph-mon[41500]: pgmap v317: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 795 B/s rd, 27 MiB/s wr, 430 op/s; 5150/249135 objects misplaced (2.067%); 905 KiB/s, 14 objects/s recovering 2024-03-24T12:33:58.237 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:57 smithi029 ceph-mon[29919]: pgmap v317: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 795 B/s rd, 27 MiB/s wr, 430 op/s; 5150/249135 objects misplaced (2.067%); 905 KiB/s, 14 objects/s recovering 2024-03-24T12:33:58.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:33:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:58.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:58.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:33:58.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:33:58.438 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:58.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:33:58.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:58.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:58.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:58.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 63 16 87748 87732 87.0207 49.3125 0.00426338 0.0114778 2024-03-24T12:33:58.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:58.709 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:58.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:58.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:58.811 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:58.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:58.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:58.959 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:58.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:58.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:59.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:59.100 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:59.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:59.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:59.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:33:59.241 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:59.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:59.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:33:59.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:33:59.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:59.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:33:59.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:33:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:59.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:33:58 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:33:58 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:33:59.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:33:59.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:59.538 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:33:59.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:33:59.639 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:33:59.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:59.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:59.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:33:59.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 64 16 88573 88557 86.4667 51.5625 0.0210166 0.0115605 2024-03-24T12:33:59.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:59.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:59.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:59.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:33:59.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:33:59.953 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:33:59.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:33:59.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:00.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:00.091 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:00.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:00.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:00.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:00.232 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:00.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:00.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:00 smithi029 ceph-mon[29919]: pgmap v318: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 707 op/s; 5150/257784 objects misplaced (1.998%); 728 KiB/s, 11 objects/s recovering 2024-03-24T12:34:00.249 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:00 smithi029 ceph-mon[41500]: pgmap v318: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 707 op/s; 5150/257784 objects misplaced (1.998%); 728 KiB/s, 11 objects/s recovering 2024-03-24T12:34:00.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:00.378 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:00.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:00.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:00 smithi190 ceph-mon[33748]: pgmap v318: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 44 MiB/s wr, 707 op/s; 5150/257784 objects misplaced (1.998%); 728 KiB/s, 11 objects/s recovering 2024-03-24T12:34:00.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:00.517 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:00.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:00.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:00.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:00.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:00.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:00.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 65 15 89722 89707 86.2421 71.875 0.0101977 0.0115905 2024-03-24T12:34:00.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:00.781 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:00.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:00.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:00.926 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:00.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:00.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:01.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:01.068 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:01.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:01.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:01.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:01.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:01.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:01.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:01.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:01.335 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:01.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:01.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:01 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:34:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:01 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:34:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:01 smithi190 ceph-mon[33748]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T12:34:01.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:01 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:34:01.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:01 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:34:01.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:01 smithi029 ceph-mon[29919]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T12:34:01.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:01 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:34:01.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:01 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:34:01.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:01 smithi029 ceph-mon[41500]: osdmap e183: 8 total, 8 up, 8 in 2024-03-24T12:34:01.437 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:01.478 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:01.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:01.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:01.601 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:01.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:01.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 66 16 90658 90642 85.8208 58.4375 0.00900684 0.0116469 2024-03-24T12:34:01.704 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:01.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:01.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:01.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:01.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:01.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:01.869 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:01.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:01.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:01.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:01.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:02.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:02.131 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:02.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:02.151 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:34:02.151 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:34:02.151 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-24T12:34:02.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:34:02.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:02.276 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:02.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:02.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:02.321 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:02 smithi029 ceph-mon[41500]: pgmap v320: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 693 op/s; 4763/260466 objects misplaced (1.829%); 5.5 MiB/s, 88 objects/s recovering 2024-03-24T12:34:02.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:02 smithi029 ceph-mon[29919]: pgmap v320: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 693 op/s; 4763/260466 objects misplaced (1.829%); 5.5 MiB/s, 88 objects/s recovering 2024-03-24T12:34:02.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:02 smithi190 ceph-mon[33748]: pgmap v320: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 43 MiB/s wr, 693 op/s; 4763/260466 objects misplaced (1.829%); 5.5 MiB/s, 88 objects/s recovering 2024-03-24T12:34:02.410 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:02.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:02.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:02.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:02.549 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:02.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:02.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:02.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:02.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 67 16 91430 91414 85.2601 48.25 0.00681547 0.0117231 2024-03-24T12:34:02.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:02.711 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:02.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:02.780 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:34:02.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:02.816 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:34:02.821 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:34:02.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:02.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:02.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:02.886 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:34:02.929 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:34:02.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:02.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:02.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:02.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:02.976 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:34:03.014 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:34:03.055 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:34:03.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:03.095 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:34:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:34:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:34:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:34:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:34:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:34:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:34:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:34:03.107 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:34:03.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:03.118 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:03.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:03.194 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:03.194 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:03 smithi029 ceph-mon[41500]: pgmap v321: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 48 MiB/s wr, 764 op/s; 4763/268260 objects misplaced (1.776%); 4.6 MiB/s, 74 objects/s recovering 2024-03-24T12:34:03.194 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:03 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1049015201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:34:03.194 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:03.194 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:03 smithi029 ceph-mon[29919]: pgmap v321: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 48 MiB/s wr, 764 op/s; 4763/268260 objects misplaced (1.776%); 4.6 MiB/s, 74 objects/s recovering 2024-03-24T12:34:03.194 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:03 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1049015201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:34:03.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:03.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:03.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:03.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:03.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:03 smithi190 ceph-mon[33748]: pgmap v321: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 48 MiB/s wr, 764 op/s; 4763/268260 objects misplaced (1.776%); 4.6 MiB/s, 74 objects/s recovering 2024-03-24T12:34:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:03 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1049015201' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:34:03.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:03.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:03.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:03.550 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:03.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:03.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:03.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 68 16 92388 92372 84.8867 59.875 0.0124086 0.0117755 2024-03-24T12:34:03.687 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:03.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:03.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:03.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:03.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:03.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:03.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:03.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:03.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:04.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:04.104 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:04.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:04.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:04.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:04.245 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:04.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:04.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:04.385 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:04.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:04.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:04.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:04.531 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:04.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:04.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:04.656 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:04.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:04.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:04.680 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 69 16 93469 93453 84.6355 67.5625 0.00319009 0.0118013 2024-03-24T12:34:04.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:04.780 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:04.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:04.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:04.881 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:04.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:04.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:05.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:05.068 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:05.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:05.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:05.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:05.193 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:05.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:05.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:05.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:05.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:05.326 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:05.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:05.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:05.459 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:05.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:05.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:05.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:05.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:05.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:05.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 70 16 94077 94061 83.9693 38 0.0111283 0.0119041 2024-03-24T12:34:05.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:05.726 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:05.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:05.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:05.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:05.868 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:05.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:05.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:05 smithi190 ceph-mon[33748]: pgmap v322: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 899 op/s; 4532/274677 objects misplaced (1.650%); 7.1 MiB/s, 114 objects/s recovering 2024-03-24T12:34:05.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:05 smithi029 ceph-mon[41500]: pgmap v322: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 899 op/s; 4532/274677 objects misplaced (1.650%); 7.1 MiB/s, 114 objects/s recovering 2024-03-24T12:34:05.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:05 smithi029 ceph-mon[29919]: pgmap v322: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 56 MiB/s wr, 899 op/s; 4532/274677 objects misplaced (1.650%); 7.1 MiB/s, 114 objects/s recovering 2024-03-24T12:34:05.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:06.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:06.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:06.147 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:06.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:06.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:06.214 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:34:06.214 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:34:06.214 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:34:06.214 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:34:06.214 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.214 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "MOSDPGPushReply(7.10 125/112 [PushReplyOp(7:0a308258:::benchmark_data_smithi029_84282_object28192:head)])", 2024-03-24T12:34:06.214 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:29:57.422387+0000", 2024-03-24T12:34:06.214 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 246.068265897, 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.47947943900000001, 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.422387+0000", 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.422387+0000", 2024-03-24T12:34:06.215 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.422387+0000", 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4099999999999999e-07 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.422391+0000", 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7859999999999998e-06 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.216 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.422399+0000", 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2819999999999996e-06 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.422405+0000", 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6720000000000001e-06 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.422453+0000", 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7936000000000003e-05 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.217 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:57.901866+0000", 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.47941322200000003 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:141516 7.10 7:0c32a566:::benchmark_data_smithi029_84282_object40753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:52.462124+0000", 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 191.028528198, 2024-03-24T12:34:06.218 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.95534503100000001, 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 141516 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.462124+0000", 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.219 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.462124+0000", 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.462126+0000", 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.359e-06 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.462131+0000", 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.631e-06 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.220 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.462144+0000", 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2353e-05 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.462153+0000", 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6399999999999992e-06 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.462323+0000", 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000169555 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.221 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.462334+0000", 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1333e-05 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.408427+0000", 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94609323199999995 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.408551+0000", 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012332900000000001 2024-03-24T12:34:06.222 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.409294+0000", 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00074381899999999995 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.414132+0000", 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0048371480000000003 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.417309+0000", 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0031773959999999999 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.417361+0000", 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2408999999999999e-05 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.417431+0000", 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0009e-05 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.417469+0000", 2024-03-24T12:34:06.224 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7818000000000003e-05 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:141517 7.10 7:0aab7eb5:::benchmark_data_smithi029_84282_object40754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:52.465543+0000", 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 191.025109509, 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.95662581999999996, 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 141517 2024-03-24T12:34:06.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.465543+0000", 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.465543+0000", 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.226 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.465544+0000", 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.032e-06 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.465549+0000", 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5970000000000002e-06 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.465558+0000", 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7389999999999994e-06 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.227 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.465565+0000", 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8539999999999999e-06 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.465625+0000", 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9922e-05 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.465636+0000", 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1267000000000001e-05 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.228 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.411971+0000", 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94633475199999995 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.412088+0000", 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011728899999999999 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.412406+0000", 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00031773000000000001 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.417322+0000", 2024-03-24T12:34:06.229 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0049159119999999997 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.417510+0000", 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018841600000000001 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.422035+0000", 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0045242759999999998 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.230 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.422104+0000", 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9004000000000005e-05 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.422169+0000", 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5030000000000006e-05 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:141554 7.10 7:09f0f8c7:::benchmark_data_smithi029_84282_object40791:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:52.727152+0000", 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 190.763500435, 2024-03-24T12:34:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.70219545100000003, 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 141554 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.727152+0000", 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.232 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.727152+0000", 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.727154+0000", 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0109999999999999e-06 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.727159+0000", 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0679999999999996e-06 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.233 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.727173+0000", 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4100999999999999e-05 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.727183+0000", 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6479999999999998e-06 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.727305+0000", 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000122595 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.234 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.727315+0000", 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0064999999999999e-05 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.414163+0000", 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.68684735200000002 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.414275+0000", 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011166 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.235 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.414567+0000", 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029275999999999998 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.417327+0000", 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002760111 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429203+0000", 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.011875738 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.236 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429250+0000", 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6825e-05 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429303+0000", 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3445000000000001e-05 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429347+0000", 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4072000000000001e-05 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.237 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:141643 7.10 7:0b7a4696:::benchmark_data_smithi029_84282_object40880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:52.825748+0000", 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 190.66490471200001, 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.60374717499999997, 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 141643 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.238 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.825748+0000", 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.825748+0000", 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.239 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.825748+0000", 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7299999999999996e-07 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.825751+0000", 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7999999999999999e-06 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.825758+0000", 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7109999999999998e-06 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.240 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.825764+0000", 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7019999999999998e-06 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.826677+0000", 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00091300500000000004 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.826687+0000", 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.011e-05 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.241 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.417541+0000", 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.590854502 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.417652+0000", 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000111067 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.418456+0000", 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00080371200000000005 2024-03-24T12:34:06.242 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429125+0000", 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010669095 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429388+0000", 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026308100000000002 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429420+0000", 2024-03-24T12:34:06.243 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2221999999999997e-05 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429464+0000", 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3695000000000001e-05 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429495+0000", 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0899999999999999e-05 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.244 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:141691 7.10 7:0b87e867:::benchmark_data_smithi029_84282_object40928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:52.879156+0000", 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 190.61149617699999, 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.55053512900000001, 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 141691 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.879156+0000", 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.879156+0000", 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.879158+0000", 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.99e-06 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.879164+0000", 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0090000000000004e-06 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.246 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.879177+0000", 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2809000000000001e-05 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.879187+0000", 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5619999999999993e-06 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.879256+0000", 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9578000000000004e-05 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.247 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.879267+0000", 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0750999999999999e-05 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.420108+0000", 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.54084106499999995 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.420216+0000", 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000107935 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.248 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.420497+0000", 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00028134200000000002 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429131+0000", 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086333739999999992 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429528+0000", 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00039752500000000003 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429560+0000", 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2045999999999998e-05 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429651+0000", 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1168999999999994e-05 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429691+0000", 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9974e-05 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.250 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:141702 7.10 7:0fc091dc:::benchmark_data_smithi029_84282_object40939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:52.889606+0000", 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 190.601046408, 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.550149098, 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 141702 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.251 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.889606+0000", 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.889606+0000", 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.889607+0000", 2024-03-24T12:34:06.252 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7999999999999995e-07 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.889610+0000", 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.225e-06 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.889617+0000", 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2529999999999998e-06 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.253 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.889624+0000", 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5989999999999998e-06 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.889678+0000", 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4413000000000002e-05 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.889689+0000", 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0407999999999999e-05 2024-03-24T12:34:06.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.422206+0000", 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.53251789699999996 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.422329+0000", 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012283699999999999 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.422591+0000", 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026148600000000001 2024-03-24T12:34:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429730+0000", 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0071390760000000003 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429765+0000", 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5513999999999999e-05 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.439672+0000", 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099065650000000009 2024-03-24T12:34:06.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.439716+0000", 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4388999999999998e-05 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.439755+0000", 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8856000000000002e-05 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.257 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:141765 7.10 7:0c2a26d3:::benchmark_data_smithi029_84282_object41002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:52.970892+0000", 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 190.51976013199999, 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.46897168099999997, 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 141765 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.258 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.970892+0000", 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.970892+0000", 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.970894+0000", 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.584e-06 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.259 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.970899+0000", 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3630000000000002e-06 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.970918+0000", 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8298e-05 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.970927+0000", 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1209999999999996e-06 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.260 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.972487+0000", 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001560068 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.972498+0000", 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1409e-05 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.426897+0000", 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45439856499999998 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.427018+0000", 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000121753 2024-03-24T12:34:06.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.427358+0000", 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000339754 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.434407+0000", 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0070489250000000002 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.262 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.439793+0000", 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0053853920000000001 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.439802+0000", 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6779999999999995e-06 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.439834+0000", 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2085000000000002e-05 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.439864+0000", 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9686e-05 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:141787 7.10 7:0a15b122:::benchmark_data_smithi029_84282_object41024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:30:52.994729+0000", 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 190.49592365500001, 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45270114900000002, 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.264 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 141787 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.994729+0000", 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.265 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.994729+0000", 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.994730+0000", 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1969999999999999e-06 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.994734+0000", 2024-03-24T12:34:06.266 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8179999999999997e-06 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.994743+0000", 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0180000000000007e-06 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.994751+0000", 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7649999999999993e-06 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.996652+0000", 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0019015869999999999 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:52.996664+0000", 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1751e-05 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429791+0000", 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.43312690700000001 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.429896+0000", 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000104849 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.430241+0000", 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00034548800000000002 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.439870+0000", 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0096286549999999999 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.444368+0000", 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0044980159999999996 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.447323+0000", 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0029551059999999999 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.447378+0000", 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4960000000000002e-05 2024-03-24T12:34:06.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:30:53.447430+0000", 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2031999999999999e-05 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:150445 7.10 7:08125ddf:::benchmark_data_smithi029_84282_object49682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:07.044864+0000", 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 176.445788765, 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.44714555099999997, 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.271 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 150445 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044864+0000", 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044864+0000", 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.272 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044865+0000", 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5099999999999998e-07 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044868+0000", 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.41e-06 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044876+0000", 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2519999999999999e-06 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.044882+0000", 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2759999999999997e-06 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.046905+0000", 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020220429999999998 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.047030+0000", 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012532400000000001 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.047308+0000", 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000278521 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.049953+0000", 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026443640000000002 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.049971+0000", 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8440999999999998e-05 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.275 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.491901+0000", 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.44192993600000002 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.491960+0000", 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8879999999999999e-05 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:07.492009+0000", 2024-03-24T12:34:06.276 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9252999999999998e-05 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164274 7.14 7:2970b39d:::benchmark_data_smithi029_84282_object63511:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:29.237490+0000", 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 154.253162764, 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2040237389999999, 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.277 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164274 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.237490+0000", 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.237490+0000", 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.237491+0000", 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3030000000000001e-06 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.237496+0000", 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6e-06 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.237504+0000", 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7809999999999994e-06 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.237511+0000", 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7850000000000003e-06 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.238359+0000", 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00084736399999999997 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.238368+0000", 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9869999999999994e-06 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.429192+0000", 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.190823382 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.429325+0000", 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013291600000000001 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.429768+0000", 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000442912 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.437763+0000", 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079957220000000002 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.439021+0000", 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001257256 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.441428+0000", 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00240741 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.282 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.441473+0000", 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4706999999999997e-05 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.441513+0000", 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0614000000000001e-05 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164317 7.14 7:28cd1424:::benchmark_data_smithi029_84282_object63554:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-24T12:34:06.283 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:29.273829+0000", 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 154.216823224, 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1855245349999999, 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164317 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.273829+0000", 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.284 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.273829+0000", 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.273831+0000", 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.967e-06 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.273837+0000", 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3700000000000003e-06 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.273852+0000", 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5298000000000002e-05 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.273860+0000", 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3010000000000007e-06 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.277450+0000", 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0035898829999999999 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.286 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.277461+0000", 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1218e-05 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.439053+0000", 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.161591703 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.439164+0000", 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000110896 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.287 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.439528+0000", 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036453899999999999 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.444630+0000", 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0051016100000000003 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.444672+0000", 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1878000000000001e-05 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.288 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.458939+0000", 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.014267276000000001 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.459304+0000", 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036457900000000002 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.459354+0000", 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0016999999999999e-05 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.289 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164493 7.14 7:2a9c0168:::benchmark_data_smithi029_84282_object63730:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:29.414980+0000", 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 154.07567271100001, 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0476420019999999, 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.290 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164493 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.414980+0000", 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.414980+0000", 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.414980+0000", 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4199999999999996e-07 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.414983+0000", 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0759999999999999e-06 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.414993+0000", 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2310000000000002e-06 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.414999+0000", 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.319e-06 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.415176+0000", 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017670699999999999 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.415186+0000", 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0192e-05 2024-03-24T12:34:06.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.442228+0000", 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.027042531 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.442367+0000", 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013907000000000001 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.294 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.442668+0000", 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030047400000000001 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.444704+0000", 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020360270000000001 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.447196+0000", 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0024924169999999998 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.295 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462474+0000", 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.015277658 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462569+0000", 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5351000000000002e-05 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462622+0000", 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2407000000000002e-05 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.296 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164545 7.14 7:2cbc0089:::benchmark_data_smithi029_84282_object63782:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:29.453843+0000", 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 154.03680939, 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.008914785, 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164545 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.297 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.453843+0000", 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.453843+0000", 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.453844+0000", 2024-03-24T12:34:06.298 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.122e-06 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.453849+0000", 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.498e-06 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.453858+0000", 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7110000000000007e-06 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.299 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.453866+0000", 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8849999999999999e-06 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.453932+0000", 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5922999999999999e-05 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.453942+0000", 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0010000000000001e-05 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.444732+0000", 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.99078960900000002 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.444844+0000", 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011252999999999999 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.445175+0000", 2024-03-24T12:34:06.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000330537 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.451398+0000", 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0062229809999999998 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.451472+0000", 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4235999999999999e-05 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462630+0000", 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.011157897 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462715+0000", 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5272999999999995e-05 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462758+0000", 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2573e-05 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164560 7.14 7:29dedb47:::benchmark_data_smithi029_84282_object63797:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:29.537600+0000", 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.95305214300001, 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.92528761599999998, 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164560 2024-03-24T12:34:06.304 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.537600+0000", 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.537600+0000", 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.305 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.537602+0000", 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.75e-06 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.537607+0000", 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.933e-06 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.537624+0000", 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7303999999999999e-05 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.306 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.537634+0000", 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5899999999999997e-06 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.537716+0000", 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.195e-05 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.537731+0000", 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4764e-05 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.307 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.447228+0000", 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.90949726399999997 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.447333+0000", 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105235 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.447643+0000", 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030979100000000002 2024-03-24T12:34:06.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.451440+0000", 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003796649 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.453826+0000", 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0023865990000000001 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462765+0000", 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0089388030000000004 2024-03-24T12:34:06.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462842+0000", 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7521000000000005e-05 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462888+0000", 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5463000000000002e-05 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.310 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164629 7.14 7:2f69d9e7:::benchmark_data_smithi029_84282_object63866:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:29.775763+0000", 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.714889922, 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.68726730599999997, 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164629 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.311 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.775763+0000", 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.775763+0000", 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.775764+0000", 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3e-06 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.312 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.775768+0000", 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6e-06 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.775783+0000", 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.468e-05 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.775799+0000", 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5732000000000001e-05 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.313 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.779182+0000", 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003382931 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.779194+0000", 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1885000000000001e-05 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.451498+0000", 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.67230397799999997 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.314 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.451652+0000", 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015447599999999999 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.452006+0000", 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00035360799999999999 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.453869+0000", 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00186378 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.315 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.458749+0000", 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0048794160000000001 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462895+0000", 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0041459549999999998 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.462986+0000", 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1626000000000007e-05 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.463030+0000", 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3339e-05 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164671 7.14 7:29fb2bc5:::benchmark_data_smithi029_84282_object63908:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:29.810982+0000", 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.67966995399999, 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.652177118, 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.317 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164671 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.810982+0000", 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.318 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.810982+0000", 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.810984+0000", 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3149999999999999e-06 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.810989+0000", 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.327e-06 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.319 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.811021+0000", 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2189e-05 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.811035+0000", 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3743e-05 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.811194+0000", 2024-03-24T12:34:06.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000159328 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.811205+0000", 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0159e-05 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.453900+0000", 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64269524700000003 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.321 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.454002+0000", 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000102001 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.454293+0000", 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029161500000000001 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.456404+0000", 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0021110180000000001 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.322 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.458829+0000", 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0024240939999999999 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.463037+0000", 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0042080900000000003 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.463118+0000", 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1889000000000002e-05 2024-03-24T12:34:06.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.463160+0000", 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1103000000000003e-05 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164767 7.14 7:2ce38006:::benchmark_data_smithi029_84282_object64004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:29.909601+0000", 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.58105113100001, 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.558533206, 2024-03-24T12:34:06.324 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164767 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.909601+0000", 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.325 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.909601+0000", 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.909603+0000", 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2410000000000001e-06 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.909607+0000", 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.634e-06 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.909621+0000", 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3602e-05 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.909628+0000", 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5839999999999997e-06 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.909688+0000", 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9420999999999999e-05 2024-03-24T12:34:06.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.909699+0000", 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1092e-05 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.456434+0000", 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.54673547300000003 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.456545+0000", 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011078699999999999 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.456879+0000", 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00033343000000000001 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.458789+0000", 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0019101369999999999 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.329 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.463167+0000", 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0043779639999999998 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.467981+0000", 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0048140290000000001 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.468054+0000", 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3174000000000001e-05 2024-03-24T12:34:06.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.468135+0000", 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0637999999999996e-05 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164892 7.14 7:2badc996:::benchmark_data_smithi029_84282_object64129:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:30.027016+0000", 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.463636409, 2024-03-24T12:34:06.331 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.44131907300000001, 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164892 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.027016+0000", 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.332 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.027016+0000", 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.027017+0000", 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.238e-06 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.027022+0000", 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6290000000000001e-06 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.027033+0000", 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1567e-05 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.027041+0000", 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9070000000000007e-06 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.027393+0000", 2024-03-24T12:34:06.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00035175499999999998 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.027404+0000", 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0647e-05 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.459385+0000", 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.431981114 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.459503+0000", 2024-03-24T12:34:06.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000117699 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.459875+0000", 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00037191999999999998 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.467827+0000", 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0079528250000000002 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.467840+0000", 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2486e-05 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.468225+0000", 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00038554300000000002 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.468289+0000", 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3207000000000007e-05 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.468335+0000", 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6536000000000003e-05 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164895 7.14 7:2ba4616e:::benchmark_data_smithi029_84282_object64132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:30.030901+0000", 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.45975117699999, 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.44286714399999999, 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:34:06.338 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164895 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.030901+0000", 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.030901+0000", 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.030903+0000", 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5740000000000001e-06 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.030915+0000", 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2636999999999999e-05 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.030931+0000", 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5614e-05 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.030940+0000", 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6629999999999999e-06 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.031032+0000", 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2748e-05 2024-03-24T12:34:06.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.031042+0000", 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9559999999999992e-06 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.463196+0000", 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.43215363699999998 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.463755+0000", 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00055862200000000005 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.464339+0000", 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00058419799999999997 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.468182+0000", 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0038428690000000001 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.473662+0000", 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0054800830000000002 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.473674+0000", 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2265e-05 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.473720+0000", 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5676000000000001e-05 2024-03-24T12:34:06.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:06.345 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:06.345 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:34:06.345 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:30.473768+0000", 2024-03-24T12:34:06.345 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8602e-05 2024-03-24T12:34:06.345 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.345 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.345 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.345 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:06.345 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:06.345 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:34:06.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:06.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:06.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:06.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:06.523 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:06.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:06.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:06.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:06.669 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:06.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:06.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 71 16 95228 95212 83.7998 71.9375 0.0138394 0.0119283 2024-03-24T12:34:06.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:06.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:06.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:06.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:06.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:06 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:34:06.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:06 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:34:06.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:06.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:06 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:34:06.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:06 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:34:06.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:06 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:34:06.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:06 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:34:06.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:06.949 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:06.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:06.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T12:34:07.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:07.094 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:07.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:07.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:07.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:07.241 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:07.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:07.340 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:07.342 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:07.367 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:07.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:07.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:07.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:07.517 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:07.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:07.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:07.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:07.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:07.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:07.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 72 16 96263 96247 83.5343 64.6875 0.00692566 0.0119665 2024-03-24T12:34:07.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:07.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:07.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:07.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:07 smithi190 ceph-mon[33748]: pgmap v323: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 932 op/s; 4532/277098 objects misplaced (1.636%); 7.1 MiB/s, 114 objects/s recovering 2024-03-24T12:34:07.794 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:07 smithi190 ceph-mon[33748]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T12:34:07.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:07.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:07 smithi029 ceph-mon[41500]: pgmap v323: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 932 op/s; 4532/277098 objects misplaced (1.636%); 7.1 MiB/s, 114 objects/s recovering 2024-03-24T12:34:07.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:07 smithi029 ceph-mon[41500]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T12:34:07.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:07 smithi029 ceph-mon[29919]: pgmap v323: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.6 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 58 MiB/s wr, 932 op/s; 4532/277098 objects misplaced (1.636%); 7.1 MiB/s, 114 objects/s recovering 2024-03-24T12:34:07.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:07 smithi029 ceph-mon[29919]: osdmap e184: 8 total, 8 up, 8 in 2024-03-24T12:34:07.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:34:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:34:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:34:07.930 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:07.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:07.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:08.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:08.060 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:08.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:08.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:34:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:34:08.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:08.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:08.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:08.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:08.350 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:08.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:08.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:08.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:08.492 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:08.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:08.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:08.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:08.646 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:08.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:08.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 73 16 97339 97323 83.3111 67.25 0.0181918 0.0119971 2024-03-24T12:34:08.721 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:34:08.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:08.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:08.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:08.873 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:08.873 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:08.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:08.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:08.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:08.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:09.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:09.038 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:09.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:09.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:09.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:09.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:09.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:09.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:09.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:09.303 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:09.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:09.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:09.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:09.447 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:09.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:09.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:09.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:09.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:09.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:09.592 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:09.625 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:34:09.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:09 smithi190 ceph-mon[33748]: pgmap v325: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 4532/286194 objects misplaced (1.584%); 3.6 MiB/s, 57 objects/s recovering 2024-03-24T12:34:09.651 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:09 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3887295777' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:34:09.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 74 16 98237 98221 82.9437 56.125 0.0181902 0.0120511 2024-03-24T12:34:09.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:09.741 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:09.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:09.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:09.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:09 smithi029 ceph-mon[29919]: pgmap v325: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 4532/286194 objects misplaced (1.584%); 3.6 MiB/s, 57 objects/s recovering 2024-03-24T12:34:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:09 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3887295777' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:34:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:09 smithi029 ceph-mon[41500]: pgmap v325: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.00k op/s; 4532/286194 objects misplaced (1.584%); 3.6 MiB/s, 57 objects/s recovering 2024-03-24T12:34:09.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:09 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3887295777' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:34:09.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:10.033 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:10.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:10.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:34:10.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:10.149 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:34:10.149 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:34:10.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.149 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:98674 8.2 8:4362f636:::benchmark_data_smithi029_108893_object98673:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e185)", 2024-03-24T12:34:10.149 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:34:10.146709+0000", 2024-03-24T12:34:10.149 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0051774639999999997, 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052154599999999999, 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 98674 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.146709+0000", 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:10.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:10.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.146709+0000", 2024-03-24T12:34:10.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:10.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.151 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:10.151 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.146710+0000", 2024-03-24T12:34:10.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.212e-06 2024-03-24T12:34:10.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:10.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.146750+0000", 2024-03-24T12:34:10.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9271999999999999e-05 2024-03-24T12:34:10.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:10.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.146761+0000", 2024-03-24T12:34:10.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1888999999999999e-05 2024-03-24T12:34:10.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:10.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.146769+0000", 2024-03-24T12:34:10.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4529999999999996e-06 2024-03-24T12:34:10.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:10.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.146811+0000", 2024-03-24T12:34:10.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2200999999999998e-05 2024-03-24T12:34:10.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:10.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.146917+0000", 2024-03-24T12:34:10.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106352 2024-03-24T12:34:10.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.147028+0000", 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011082399999999999 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.149731+0000", 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027029350000000001 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.150305+0000", 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00057408700000000003 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:10.155 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.24815.0:98678 8.b e185/158 8:d7b8bc44:::benchmark_data_smithi029_108893_object98677:head v 185'6143, mlcod=185'6140)", 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:34:10.151152+0000", 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.00073432399999999996, 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00088528900000000004, 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.151152+0000", 2024-03-24T12:34:10.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.151152+0000", 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.151153+0000", 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3200000000000004e-07 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.151184+0000", 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1102999999999997e-05 2024-03-24T12:34:10.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.151199+0000", 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4959e-05 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.151206+0000", 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8249999999999999e-06 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.151258+0000", 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2265000000000002e-05 2024-03-24T12:34:10.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:10.151277+0000", 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9108000000000001e-05 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 2 2024-03-24T12:34:10.159 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:34:10.185 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:10.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:10.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:10.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:10.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:10.470 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:10.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:10.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:10.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:10.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:10.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:10 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:34:10.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:10 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:34:10.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 75 16 99125 99109 82.5777 55.5 0.00320469 0.0120964 2024-03-24T12:34:10.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:10.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:10.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:10.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:10.828 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:10.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:10.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:10.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:10.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:10 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:34:10.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:10 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3887295777' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:34:10.937 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:10 smithi190 ceph-mon[33748]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T12:34:10.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:10.984 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T12:34:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:11.002 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:11.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:11.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:11.140 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:11.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:11.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:10 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3887295777' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:34:11.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:10 smithi029 ceph-mon[41500]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T12:34:11.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:10 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3887295777' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:34:11.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:10 smithi029 ceph-mon[29919]: osdmap e185: 8 total, 8 up, 8 in 2024-03-24T12:34:11.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:11.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:11.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:11.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:11.397 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:11.438 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:11.439 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:11.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:11.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:11.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:11.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:11.681 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 76 16 100171 100155 82.3513 65.375 0.00326952 0.0121274 2024-03-24T12:34:11.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:11.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:11.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:11.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:11.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:11.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:11.993 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:11.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:11.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:12.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:12.137 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:12.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:12.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:12.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:12.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:12.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:12.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:12.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:11 smithi190 ceph-mon[33748]: pgmap v327: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 860 op/s; 4292/288903 objects misplaced (1.486%); 7.5 MiB/s, 119 objects/s recovering 2024-03-24T12:34:12.405 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:12.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:12.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:12.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:11 smithi029 ceph-mon[29919]: pgmap v327: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 860 op/s; 4292/288903 objects misplaced (1.486%); 7.5 MiB/s, 119 objects/s recovering 2024-03-24T12:34:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:11 smithi029 ceph-mon[41500]: pgmap v327: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 54 MiB/s wr, 860 op/s; 4292/288903 objects misplaced (1.486%); 7.5 MiB/s, 119 objects/s recovering 2024-03-24T12:34:12.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:12.553 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:12.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:12.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:12.683 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 77 16 101207 101191 82.1204 64.75 0.00450663 0.0121719 2024-03-24T12:34:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:12.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:12.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:12.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:12.823 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:12.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:12.924 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:12.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:12.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:13.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:13.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:13.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:13.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:13.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:13.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:13.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:13.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:13.335 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:34:13.336 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:34:13.336 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-03-24T12:34:13.336 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-03-24T12:34:13.336 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:34:13.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:13.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:13.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:13.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:13.501 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:13.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:13.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:13.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:13.651 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:13.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:13.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:13.684 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 78 16 102015 101999 81.715 50.5 0.00716729 0.0122325 2024-03-24T12:34:13.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:13.792 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:13.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:13.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:13.947 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:13.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:14.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T12:34:14.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:14.127 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:14.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:14.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:14.271 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:14.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:14.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:14 smithi190 ceph-mon[33748]: pgmap v328: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s; 4292/297336 objects misplaced (1.443%); 3.7 MiB/s, 58 objects/s recovering 2024-03-24T12:34:14.405 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:14.405 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:14.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:14.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:14 smithi029 ceph-mon[29919]: pgmap v328: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s; 4292/297336 objects misplaced (1.443%); 3.7 MiB/s, 58 objects/s recovering 2024-03-24T12:34:14.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:14 smithi029 ceph-mon[41500]: pgmap v328: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 945 op/s; 4292/297336 objects misplaced (1.443%); 3.7 MiB/s, 58 objects/s recovering 2024-03-24T12:34:14.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:34:14.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:14.546 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:14.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:14.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:14.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:14.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:14.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:14.686 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 79 16 103045 103029 81.4954 64.375 0.0261434 0.0122653 2024-03-24T12:34:14.697 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:34:14.766 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 185, 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "created": "2024-03-24T12:20:18.712998+0000", 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "modified": "2024-03-24T12:34:09.615233+0000", 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "last_up_change": "2024-03-24T12:25:48.676565+0000", 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "last_in_change": "2024-03-24T12:25:30.369664+0000", 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_num": 5806080, 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_set": [ 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "nodeep-scrub", 2024-03-24T12:34:14.767 INFO:teuthology.orchestra.run.smithi029.stdout: "noscrub", 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "pglog_hardlimit", 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "purged_snapdirs", 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_deletes", 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "sortbitwise" 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_version": 18, 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_snapshot": "", 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_max": 8, 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd": 8, 2024-03-24T12:34:14.768 INFO:teuthology.orchestra.run.smithi029.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "min_compat_client": "luminous", 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "require_osd_release": "squid", 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_crimson": false, 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "pools": [ 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 1, 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".mgr", 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:23:41.491597+0000", 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:34:14.769 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 1, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 1, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 1, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 1, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 1, 2024-03-24T12:34:14.770 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "22", 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:34:14.771 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:34:14.772 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:34:14.773 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_max": 32, 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_min": 1 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:34:14.774 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": {} 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 7.8899998664855957, 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 7.8899998664855957, 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.37999999523162842, 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 3, 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 3, 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.775 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 2, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "datapool", 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:01.465080+0000", 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 8193, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:34:14.776 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 3, 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 3, 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 3, 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 3, 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 3, 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.777 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "71", 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 2, 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 71, 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:34:14.778 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:34:14.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:34:14.780 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd": {} 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.6500000953674316, 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.6500000953674316, 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:34:14.781 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 3, 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".rgw.root", 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:02.100021+0000", 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:34:14.782 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:34:14.783 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:34:14.784 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:34:14.784 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:34:14.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:14.785 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:34:14.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.785 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "68", 2024-03-24T12:34:14.785 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:34:14.785 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:34:14.785 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:34:14.785 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:34:14.785 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:34:14.785 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:34:14.785 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:34:14.786 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:34:14.787 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.5, 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.5, 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.5, 2024-03-24T12:34:14.788 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.5, 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 4, 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:03.473862+0000", 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:34:14.789 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:34:14.790 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "70", 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:34:14.791 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:34:14.792 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:34:14.793 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.25, 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.25, 2024-03-24T12:34:14.794 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.25, 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2.25, 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 5, 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:05.573828+0000", 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:34:14.795 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:34:14.796 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "72", 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:34:14.797 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:34:14.798 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:34:14.799 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.25, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.25, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.25, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.25, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:34:14.800 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 6, 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:07.645083+0000", 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:34:14.801 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:34:14.802 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "74", 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:34:14.803 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:34:14.804 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:34:14.805 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_bias": 4 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.75, 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.75, 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.75, 2024-03-24T12:34:14.806 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.75, 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 8, 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "unique_pool_1", 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:32:39.584334+0000", 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:34:14.807 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 16, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 16, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 16, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 16, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 16, 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:34:14.808 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "164", 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:34:14.809 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:34:14.810 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:34:14.811 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "rados": {} 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2, 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2, 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 1, 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2, 2024-03-24T12:34:14.812 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2, 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b", 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:34:14.813 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 8, 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6801", 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6802", 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6804", 2024-03-24T12:34:14.815 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6803", 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6801/2514818464", 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6802/2514818464", 2024-03-24T12:34:14.816 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6804/2514818464", 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6803/2514818464", 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc", 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:34:14.817 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:34:14.818 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 13, 2024-03-24T12:34:14.818 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:34:14.818 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:34:14.818 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:34:14.818 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:34:14.818 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.818 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6805", 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6806", 2024-03-24T12:34:14.819 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6808", 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6807", 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6805/2923396572", 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6806/2923396572", 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6808/2923396572", 2024-03-24T12:34:14.821 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6807/2923396572", 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "c3f83505-25ee-4217-b60a-081c36092e92", 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:34:14.822 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 18, 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 142, 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6809", 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.823 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6810", 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.824 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6812", 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6811", 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6809/126032915", 2024-03-24T12:34:14.825 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6810/126032915", 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6812/126032915", 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6811/126032915", 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96", 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:34:14.826 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 27, 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 181, 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6813", 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:34:14.827 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6814", 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:34:14.828 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6816", 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6815", 2024-03-24T12:34:14.829 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6813/1080171824", 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6814/1080171824", 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6816/1080171824", 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6815/1080171824", 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "5e420304-2798-4d11-9688-d0d89482008c", 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 33, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:34:14.831 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6800", 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6801", 2024-03-24T12:34:14.832 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6803", 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6802", 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6800/3690258869", 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6801/3690258869", 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6803/3690258869", 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6802/3690258869", 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "1213431a-652c-4cbf-b32a-36335430c666", 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 42, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:34:14.835 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6804", 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6805", 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:34:14.836 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6807", 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.837 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6806", 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6804/1327220488", 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6805/1327220488", 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6807/1327220488", 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6806/1327220488", 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:34:14.839 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c", 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 51, 2024-03-24T12:34:14.840 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6808", 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6809", 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6811", 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:34:14.842 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6810", 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6808/1295064976", 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6809/1295064976", 2024-03-24T12:34:14.843 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6811/1295064976", 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6810/1295064976", 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda", 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:34:14.844 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 60, 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 160, 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6812", 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.845 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6813", 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6815", 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:34:14.846 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6814", 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6812/110963949", 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6813/110963949", 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6815/110963949", 2024-03-24T12:34:14.847 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6814/110963949", 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_xinfo": [ 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:34:14.848 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:33:40.506116+0000", 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:34:05.748585+0000", 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:34:14.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:33:39.494402+0000", 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:34:14.850 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:33:39.622360+0000", 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:34:14.851 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:33:01.570288+0000", 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:33:59.195979+0000", 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:33:16.435153+0000", 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:34:14.853 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:32:58.231586+0000", 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap": [ 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "8.0", 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: 3, 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: 5, 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: 2 2024-03-24T12:34:14.854 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_items": [ 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "3.11", 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 6, 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 5 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.b", 2024-03-24T12:34:14.855 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.f", 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.12", 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 5 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_primaries": [], 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_temp": [ 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "8.0", 2024-03-24T12:34:14.857 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: 3, 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: 5, 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: 1 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_temp": [], 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklist": { 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2633022206": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2612078930": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2365028933": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4113609697": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4130171883": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:34:14.858 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3697529912": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2929430696": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/327626505": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3715448582": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/861146182": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2109454792": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1631806715": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/852212806": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1823252077": "2024-03-25T12:21:15.151901+0000" 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "range_blocklist": {}, 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profiles": { 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "default": { 2024-03-24T12:34:14.859 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-failure-domain": "osd", 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "k": "2", 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "m": "1", 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "plugin": "jerasure", 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "technique": "reed_sol_van" 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps_queue": [], 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "new_removed_snaps": [], 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "new_purged_snaps": [], 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_node_flags": {}, 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "device_class_flags": {}, 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode": { 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_enabled": false, 2024-03-24T12:34:14.860 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_bucket_count": 0, 2024-03-24T12:34:14.861 INFO:teuthology.orchestra.run.smithi029.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:34:14.861 INFO:teuthology.orchestra.run.smithi029.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:34:14.861 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_bucket": 0 2024-03-24T12:34:14.861 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:34:14.861 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:34:14.883 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:14.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:14.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:15.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:15.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:15.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:15.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:15.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:15 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1051709602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:34:15.152 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:15.152 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:15 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1051709602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:34:15.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:15.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:15.156 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:15 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1051709602' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:34:15.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:15.301 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:15.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:15.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:15.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:15.438 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:15.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:15.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:15.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:15.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:15.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:15.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:15.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:15.684 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:34:15.688403+0000 min lat: 0.00120017 max lat: 2.54295 avg lat: 0.0122939 2024-03-24T12:34:15.684 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:34:15.684 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 80 16 104113 104097 81.311 66.75 0.0191788 0.0122939 2024-03-24T12:34:15.717 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:15.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:15.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:15.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:15.859 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:15.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:15.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:15.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:15.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:15.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:16.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:16.124 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:16.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:16.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:16.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:16.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:16.261 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:16.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:16.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:16 smithi190 ceph-mon[33748]: pgmap v329: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.11k op/s; 3987/303825 objects misplaced (1.312%); 7.7 MiB/s, 123 objects/s recovering 2024-03-24T12:34:16.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:16 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:34:16.289 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:16 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:34:16.349 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:34:16.349 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-03-24T12:34:16.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-03-24T12:34:16.349 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:34:16.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:101604 7.1 7:85914178:::benchmark_data_smithi029_84282_object841:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:29:46.183791+0000", 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 270.16545464900003, 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.353239898, 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 101604 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.350 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.183791+0000", 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.183791+0000", 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.183793+0000", 2024-03-24T12:34:16.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.491e-06 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.183799+0000", 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1550000000000004e-06 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.183817+0000", 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7992999999999999e-05 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.183825+0000", 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2369999999999992e-06 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.183886+0000", 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1470000000000006e-05 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.184011+0000", 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000124275 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.184309+0000", 2024-03-24T12:34:16.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029846499999999998 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.185601+0000", 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012920729999999999 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.185942+0000", 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034059599999999998 2024-03-24T12:34:16.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.536956+0000", 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.35101457400000002 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.536997+0000", 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0271000000000001e-05 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:46.537031+0000", 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4298000000000003e-05 2024-03-24T12:34:16.355 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:129905 7.16 7:6a89bbaf:::benchmark_data_smithi029_84282_object29142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.044494+0000", 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 221.30475131200001, 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91608335500000004, 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 129905 2024-03-24T12:34:16.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044494+0000", 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044494+0000", 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.357 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044496+0000", 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.451e-06 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044502+0000", 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4180000000000002e-06 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044517+0000", 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4985e-05 2024-03-24T12:34:16.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044525+0000", 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6510000000000001e-06 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044582+0000", 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6582999999999997e-05 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044591+0000", 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9679999999999995e-06 2024-03-24T12:34:16.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.954913+0000", 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91032239999999998 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.955039+0000", 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000126363 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.955424+0000", 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038441700000000002 2024-03-24T12:34:16.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960392+0000", 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049685270000000004 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960405+0000", 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2978e-05 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960501+0000", 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5444000000000003e-05 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960548+0000", 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7521999999999997e-05 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960578+0000", 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9648000000000001e-05 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.363 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:129941 7.16 7:6aa68b6f:::benchmark_data_smithi029_84282_object29178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.363 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.089151+0000", 2024-03-24T12:34:16.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:16.364 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 221.260094536, 2024-03-24T12:34:16.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.87456397399999997, 2024-03-24T12:34:16.364 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.364 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.364 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 129941 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089151+0000", 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089151+0000", 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089152+0000", 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.176e-06 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089157+0000", 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0440000000000003e-06 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089168+0000", 2024-03-24T12:34:16.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0969e-05 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089176+0000", 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7910000000000005e-06 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089229+0000", 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2738000000000002e-05 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089238+0000", 2024-03-24T12:34:16.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8259999999999999e-06 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.957558+0000", 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.86831984799999995 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.957655+0000", 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7683000000000006e-05 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.957971+0000", 2024-03-24T12:34:16.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031544199999999998 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960434+0000", 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024637130000000002 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960672+0000", 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000237376 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963635+0000", 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029635180000000001 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963682+0000", 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6480999999999998e-05 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963715+0000", 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3368999999999998e-05 2024-03-24T12:34:16.370 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:129994 7.16 7:6e06b474:::benchmark_data_smithi029_84282_object29231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.149614+0000", 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 221.199631337, 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82120363100000004, 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.371 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 129994 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149614+0000", 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149614+0000", 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.372 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149616+0000", 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7740000000000001e-06 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149624+0000", 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7859999999999997e-06 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149640+0000", 2024-03-24T12:34:16.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5509e-05 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149648+0000", 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8289999999999997e-06 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149723+0000", 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5117999999999995e-05 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149736+0000", 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2721000000000001e-05 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.959238+0000", 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.80950209900000003 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.959331+0000", 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2386999999999995e-05 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.959505+0000", 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017448400000000001 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963533+0000", 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040280330000000003 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963588+0000", 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5074000000000002e-05 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970720+0000", 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0071316569999999996 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970769+0000", 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9456999999999998e-05 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970818+0000", 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8702999999999998e-05 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.377 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130009 7.16 7:6b59aaed:::benchmark_data_smithi029_84282_object29246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.161268+0000", 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 221.187977577, 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82047960200000003, 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.378 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130009 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161268+0000", 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161268+0000", 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.379 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161269+0000", 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.232e-06 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161275+0000", 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1590000000000001e-06 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161285+0000", 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0713999999999999e-05 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161292+0000", 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6989999999999997e-06 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161354+0000", 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2483999999999993e-05 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161368+0000", 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3234000000000001e-05 2024-03-24T12:34:16.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960723+0000", 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.79935483699999998 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960812+0000", 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9931000000000003e-05 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.961009+0000", 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019644500000000001 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963724+0000", 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027152299999999999 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967454+0000", 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003729675 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981661+0000", 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014207054 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981698+0000", 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6838999999999997e-05 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981748+0000", 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0068999999999998e-05 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.384 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130018 7.16 7:6d78b553:::benchmark_data_smithi029_84282_object29255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.175266+0000", 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 221.17397988799999, 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.80664670699999996, 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130018 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175266+0000", 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175266+0000", 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175268+0000", 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7400000000000001e-06 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175274+0000", 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0329999999999997e-06 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175291+0000", 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7430000000000001e-05 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175301+0000", 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0370000000000001e-05 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175382+0000", 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0277000000000004e-05 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175392+0000", 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0190999999999999e-05 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.962152+0000", 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.78675967000000002 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.962267+0000", 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011578000000000001 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.962502+0000", 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023468 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967413+0000", 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049112950000000004 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967498+0000", 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4874000000000001e-05 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981793+0000", 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014295112 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981863+0000", 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9636999999999998e-05 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981913+0000", 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9617999999999998e-05 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130039 7.16 7:6daa3191:::benchmark_data_smithi029_84282_object29276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.391 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.202829+0000", 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 221.14641653000001, 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.77952253199999999, 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130039 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202829+0000", 2024-03-24T12:34:16.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202829+0000", 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202831+0000", 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3060000000000001e-06 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202835+0000", 2024-03-24T12:34:16.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7299999999999996e-06 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202854+0000", 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8416000000000001e-05 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202862+0000", 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4500000000000004e-06 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202915+0000", 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3100999999999999e-05 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202923+0000", 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1610000000000003e-06 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963748+0000", 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.76082477900000001 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963868+0000", 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000119997 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.964095+0000", 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000226758 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967465+0000", 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033697689999999999 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970870+0000", 2024-03-24T12:34:16.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003405277 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982254+0000", 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011383926000000001 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982307+0000", 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3313000000000003e-05 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982352+0000", 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4549000000000003e-05 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130093 7.16 7:6e160392:::benchmark_data_smithi029_84282_object29330:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.269634+0000", 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 221.07961170999999, 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.71289628299999996, 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.398 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130093 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269634+0000", 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269634+0000", 2024-03-24T12:34:16.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269635+0000", 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.378e-06 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269641+0000", 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8540000000000002e-06 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269662+0000", 2024-03-24T12:34:16.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0469e-05 2024-03-24T12:34:16.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269670+0000", 2024-03-24T12:34:16.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6309999999999992e-06 2024-03-24T12:34:16.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269771+0000", 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100249 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269781+0000", 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8670000000000006e-06 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.965429+0000", 2024-03-24T12:34:16.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.69564800500000001 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.965547+0000", 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000118021 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.965797+0000", 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025022800000000002 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970833+0000", 2024-03-24T12:34:16.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0050361690000000001 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970904+0000", 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1487000000000002e-05 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982425+0000", 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011520829 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982486+0000", 2024-03-24T12:34:16.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1149999999999996e-05 2024-03-24T12:34:16.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982530+0000", 2024-03-24T12:34:16.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3946000000000002e-05 2024-03-24T12:34:16.405 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130098 7.16 7:6aef2ee6:::benchmark_data_smithi029_84282_object29335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.277429+0000", 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 221.07181711300001, 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.70522084100000004, 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.406 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130098 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277429+0000", 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277429+0000", 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277430+0000", 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5600000000000001e-06 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277436+0000", 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5389999999999998e-06 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277447+0000", 2024-03-24T12:34:16.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1333e-05 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277455+0000", 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4079999999999998e-06 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277513+0000", 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7728e-05 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277522+0000", 2024-03-24T12:34:16.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6309999999999992e-06 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967527+0000", 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.69000489499999995 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967630+0000", 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010367200000000001 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967864+0000", 2024-03-24T12:34:16.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000233458 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970914+0000", 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003049997 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970949+0000", 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5073000000000001e-05 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982572+0000", 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011622937999999999 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982620+0000", 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7654999999999997e-05 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982649+0000", 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9954000000000002e-05 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.412 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130156 7.16 7:6f38da85:::benchmark_data_smithi029_84282_object29393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.463174+0000", 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 220.88607170399999, 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52157916000000004, 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130156 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463174+0000", 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463174+0000", 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463176+0000", 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.53e-06 2024-03-24T12:34:16.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463183+0000", 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3350000000000001e-06 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463202+0000", 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9423999999999999e-05 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463214+0000", 2024-03-24T12:34:16.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1616e-05 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463300+0000", 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6518000000000001e-05 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463309+0000", 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3529999999999995e-06 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.969326+0000", 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50601668300000002 2024-03-24T12:34:16.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.969419+0000", 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3696000000000006e-05 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.969625+0000", 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000205451 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970921+0000", 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012960809999999999 2024-03-24T12:34:16.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.974161+0000", 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032402080000000001 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984642+0000", 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010480835000000001 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984699+0000", 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7404999999999997e-05 2024-03-24T12:34:16.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984753+0000", 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4024999999999999e-05 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130168 7.16 7:6d007d8d:::benchmark_data_smithi029_84282_object29405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.474576+0000", 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 220.87466951799999, 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51029922000000005, 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.419 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130168 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474576+0000", 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474576+0000", 2024-03-24T12:34:16.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474578+0000", 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8080000000000001e-06 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474586+0000", 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5920000000000003e-06 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474605+0000", 2024-03-24T12:34:16.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9602999999999998e-05 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474616+0000", 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.062e-05 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474684+0000", 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8435999999999993e-05 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474694+0000", 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0186999999999999e-05 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970974+0000", 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.496279313 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.971054+0000", 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0469000000000005e-05 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.971241+0000", 2024-03-24T12:34:16.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000186369 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.974173+0000", 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029318899999999999 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.977894+0000", 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037216699999999998 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984795+0000", 2024-03-24T12:34:16.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069006240000000002 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984842+0000", 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6751e-05 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984875+0000", 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3887999999999998e-05 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.425 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130176 7.16 7:6ba24fa6:::benchmark_data_smithi029_84282_object29413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.483707+0000", 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 220.86553908400001, 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50131378900000001, 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130176 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483707+0000", 2024-03-24T12:34:16.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483707+0000", 2024-03-24T12:34:16.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483708+0000", 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.251e-06 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483712+0000", 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5340000000000001e-06 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483723+0000", 2024-03-24T12:34:16.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0831e-05 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483730+0000", 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9789999999999996e-06 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483783+0000", 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2848e-05 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483792+0000", 2024-03-24T12:34:16.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4069999999999993e-06 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.972312+0000", 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48852052499999998 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.972433+0000", 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121291 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.972611+0000", 2024-03-24T12:34:16.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017786000000000001 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.977913+0000", 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0053022870000000001 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.977983+0000", 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9985000000000002e-05 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984939+0000", 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006955363 2024-03-24T12:34:16.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984984+0000", 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4808000000000002e-05 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.985020+0000", 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6820000000000003e-05 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130190 7.16 7:693003cf:::benchmark_data_smithi029_84282_object29427:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.507558+0000", 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 220.84168783800001, 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.47759035900000002, 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130190 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507558+0000", 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507558+0000", 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507559+0000", 2024-03-24T12:34:16.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.265e-06 2024-03-24T12:34:16.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507564+0000", 2024-03-24T12:34:16.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9370000000000003e-06 2024-03-24T12:34:16.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507579+0000", 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4741e-05 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507587+0000", 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1650000000000006e-06 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507650+0000", 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3230000000000003e-05 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507660+0000", 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5380000000000008e-06 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.974217+0000", 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.46655671999999998 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.974340+0000", 2024-03-24T12:34:16.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012344600000000001 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.974619+0000", 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027936399999999999 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.979433+0000", 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048138870000000002 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.979476+0000", 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2755e-05 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.985061+0000", 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055851820000000002 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.985103+0000", 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2043999999999999e-05 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.985148+0000", 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5084999999999999e-05 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130217 7.16 7:6a8e9ff5:::benchmark_data_smithi029_84282_object29454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.554201+0000", 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 220.79504449000001, 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.435903606, 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.440 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130217 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554201+0000", 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554201+0000", 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554202+0000", 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5600000000000005e-07 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554206+0000", 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9650000000000002e-06 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554215+0000", 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4690000000000003e-06 2024-03-24T12:34:16.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554223+0000", 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3440000000000003e-06 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554291+0000", 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7779000000000003e-05 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554298+0000", 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9380000000000003e-06 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.976066+0000", 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42176839199999999 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.976209+0000", 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014275600000000001 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.976477+0000", 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000268189 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.979488+0000", 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030113319999999998 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.979527+0000", 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8349000000000002e-05 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.989983+0000", 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010456081000000001 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.990059+0000", 2024-03-24T12:34:16.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:16 smithi029 ceph-mon[41500]: pgmap v329: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.11k op/s; 3987/303825 objects misplaced (1.312%); 7.7 MiB/s, 123 objects/s recovering 2024-03-24T12:34:16.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:16 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:34:16.446 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:16 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:34:16.447 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:16 smithi029 ceph-mon[29919]: pgmap v329: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.11k op/s; 3987/303825 objects misplaced (1.312%); 7.7 MiB/s, 123 objects/s recovering 2024-03-24T12:34:16.447 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:16 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:34:16.447 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:16 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6579e-05 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.990105+0000", 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5677000000000003e-05 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.448 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130228 7.16 7:68cad5ca:::benchmark_data_smithi029_84282_object29465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.577413+0000", 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 220.771833042, 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41282179299999999, 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130228 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577413+0000", 2024-03-24T12:34:16.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577413+0000", 2024-03-24T12:34:16.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577414+0000", 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.638e-06 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577423+0000", 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3750000000000003e-06 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577443+0000", 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9769e-05 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577454+0000", 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1328000000000001e-05 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577554+0000", 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100449 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577572+0000", 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7306e-05 2024-03-24T12:34:16.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.978019+0000", 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40044787399999998 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.978095+0000", 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5640000000000001e-05 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.978278+0000", 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000182704 2024-03-24T12:34:16.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.979607+0000", 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001329521 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.980997+0000", 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013895859999999999 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.990145+0000", 2024-03-24T12:34:16.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0091478749999999998 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.990197+0000", 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2012000000000003e-05 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.990235+0000", 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7716000000000003e-05 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:142438 7.16 7:6a6aa737:::benchmark_data_smithi029_84282_object41675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:53.881235+0000", 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 202.46801042800001, 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33152354299999998, 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 142438 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881235+0000", 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881235+0000", 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881237+0000", 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5120000000000001e-06 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881243+0000", 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1519999999999997e-06 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881256+0000", 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3064999999999999e-05 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881265+0000", 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0310000000000004e-06 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881327+0000", 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2132999999999996e-05 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881436+0000", 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000108435 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881776+0000", 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000340236 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.885457+0000", 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036815620000000002 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.886281+0000", 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00082311500000000002 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:54.212681+0000", 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32640079 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:54.212716+0000", 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4938999999999999e-05 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:54.212759+0000", 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2573e-05 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:142439 7.16 7:6e5689a3:::benchmark_data_smithi029_84282_object41676:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:53.881286+0000", 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 202.46795957699999, 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33162376700000001, 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 142439 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.462 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881286+0000", 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881286+0000", 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881287+0000", 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9900000000000001e-07 2024-03-24T12:34:16.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881290+0000", 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.004e-06 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881304+0000", 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4758e-05 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.881310+0000", 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7780000000000004e-06 2024-03-24T12:34:16.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.883226+0000", 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001916334 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.883350+0000", 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012401400000000001 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.883595+0000", 2024-03-24T12:34:16.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024421700000000002 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.885552+0000", 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019572090000000001 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:53.886384+0000", 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00083252800000000002 2024-03-24T12:34:16.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:54.212805+0000", 2024-03-24T12:34:16.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32642015899999999 2024-03-24T12:34:16.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:54.212865+0000", 2024-03-24T12:34:16.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0489000000000003e-05 2024-03-24T12:34:16.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:54.212910+0000", 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4878e-05 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:150167 7.16 7:6985c70f:::benchmark_data_smithi029_84282_object49404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:06.425525+0000", 2024-03-24T12:34:16.468 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 189.92372081799999, 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.34178844800000002, 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 150167 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.425525+0000", 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.425525+0000", 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.425526+0000", 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3e-06 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.425532+0000", 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4990000000000002e-06 2024-03-24T12:34:16.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.425544+0000", 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2109999999999999e-05 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.425553+0000", 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2350000000000005e-06 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.425634+0000", 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1049000000000003e-05 2024-03-24T12:34:16.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.425781+0000", 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000146439 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.426155+0000", 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00037457000000000001 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.427951+0000", 2024-03-24T12:34:16.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0017960470000000001 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.428878+0000", 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00092714999999999996 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.767155+0000", 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.33827704400000003 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.767260+0000", 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010451 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:06.767313+0000", 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3495000000000002e-05 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:154804 7.1 7:87092e3b:::benchmark_data_smithi029_84282_object54041:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e138)", 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:13.914475+0000", 2024-03-24T12:34:16.474 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 182.43477099200001, 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.329530775, 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 154804 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.914475+0000", 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.914475+0000", 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.914476+0000", 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2240000000000001e-06 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.914481+0000", 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7110000000000003e-06 2024-03-24T12:34:16.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.914491+0000", 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8300000000000008e-06 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.914499+0000", 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.772e-06 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.914583+0000", 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3311000000000002e-05 2024-03-24T12:34:16.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.914686+0000", 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010327000000000001 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.915139+0000", 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00045280899999999998 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.921579+0000", 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064406769999999997 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:13.921654+0000", 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5085000000000004e-05 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:14.243913+0000", 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.322258143 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:14.243966+0000", 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3331000000000002e-05 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:14.244006+0000", 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9612e-05 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:163924 7.16 7:6ca9cefb:::benchmark_data_smithi029_84282_object63161:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e142)", 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:27.346472+0000", 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 169.00277385800001, 2024-03-24T12:34:16.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36607290100000001, 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163924 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.346472+0000", 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.346472+0000", 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.346473+0000", 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3200000000000003e-07 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.346476+0000", 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2270000000000002e-06 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.346484+0000", 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9249999999999995e-06 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.346491+0000", 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2250000000000003e-06 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.346534+0000", 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3081000000000003e-05 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.346634+0000", 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100059 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.346936+0000", 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030174100000000001 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.349058+0000", 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021218830000000002 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.350071+0000", 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010125240000000001 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.712417+0000", 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36234620000000001 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.712491+0000", 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4064000000000001e-05 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:16.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:16.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.712545+0000", 2024-03-24T12:34:16.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4039999999999998e-05 2024-03-24T12:34:16.486 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.486 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.486 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.486 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:16.486 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:16.486 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:34:16.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:16.487 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:16.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:16.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:16.621 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:16.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:16.684 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 81 16 105015 104999 81.0031 56.375 0.00366221 0.012337 2024-03-24T12:34:16.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:16.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:16.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:16.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:16.888 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:16.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:16.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:16.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:17.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:17.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:17.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:17.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:17.163 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:17.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:17.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:17.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:17.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:17.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:17 smithi190 ceph-mon[33748]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T12:34:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:17 smithi190 ceph-mon[33748]: pgmap v331: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 846 op/s; 3987/306507 objects misplaced (1.301%); 7.7 MiB/s, 123 objects/s recovering 2024-03-24T12:34:17.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:17.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:17 smithi029 ceph-mon[29919]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T12:34:17.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:17 smithi029 ceph-mon[29919]: pgmap v331: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 846 op/s; 3987/306507 objects misplaced (1.301%); 7.7 MiB/s, 123 objects/s recovering 2024-03-24T12:34:17.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:17 smithi029 ceph-mon[41500]: osdmap e186: 8 total, 8 up, 8 in 2024-03-24T12:34:17.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:17 smithi029 ceph-mon[41500]: pgmap v331: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.2 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 53 MiB/s wr, 846 op/s; 3987/306507 objects misplaced (1.301%); 7.7 MiB/s, 123 objects/s recovering 2024-03-24T12:34:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:17.452 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:17.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:17.530 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:34:17.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:17.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:17.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:17.684 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 82 16 106017 106001 80.7789 62.625 0.0835189 0.012369 2024-03-24T12:34:17.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:17.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:17.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:17.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:17.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:17.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:17.895 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:17.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:34:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:34:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:34:17.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:18.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:18.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:18.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:18.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:18.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:18.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:18.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:34:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:34:18.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:18.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:18.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:18.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:18.276 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:18.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:18.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:18.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:18.417 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:18.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:18.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:18.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:18.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:18.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:18.560 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:18.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:18.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:18.684 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 83 15 106902 106887 80.4728 55.375 0.0138586 0.0124217 2024-03-24T12:34:18.695 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:18.696 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:18.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:18.836 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:18.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:18.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:18.977 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:18.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:19.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:19.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:19.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:19 smithi190 ceph-mon[33748]: pgmap v332: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 3987/315795 objects misplaced (1.263%); 3.6 MiB/s, 57 objects/s recovering 2024-03-24T12:34:19.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:19.245 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:19.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:19.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:19.763 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:19 smithi029 ceph-mon[41500]: pgmap v332: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 3987/315795 objects misplaced (1.263%); 3.6 MiB/s, 57 objects/s recovering 2024-03-24T12:34:19.763 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:19 smithi029 ceph-mon[29919]: pgmap v332: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.4 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 3987/315795 objects misplaced (1.263%); 3.6 MiB/s, 57 objects/s recovering 2024-03-24T12:34:19.764 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 84 16 107728 107712 80.1286 51.5625 0.00967648 0.0124749 2024-03-24T12:34:19.802 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:19.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:19.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:19.871 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:34:19.871 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:34:19.871 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.871 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:107936 8.4 8:26c4f2b4:::benchmark_data_smithi029_108893_object107935:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e187)", 2024-03-24T12:34:19.871 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:34:19.867926+0000", 2024-03-24T12:34:19.871 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.0050323540000000002, 2024-03-24T12:34:19.871 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0050586370000000004, 2024-03-24T12:34:19.871 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:19.871 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:34:19.871 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 107936 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.867926+0000", 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.872 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.867926+0000", 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.867928+0000", 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1689999999999999e-06 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.867986+0000", 2024-03-24T12:34:19.873 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8368000000000001e-05 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.867998+0000", 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1647999999999999e-05 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.868005+0000", 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6310000000000003e-06 2024-03-24T12:34:19.874 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.868067+0000", 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1866999999999996e-05 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.868216+0000", 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000149204 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.875 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.868889+0000", 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00067245800000000002 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop_reply(client.24815.0:107936 8.4 e187/158)", 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:34:19.872021+0000", 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.00093813499999999999, 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010133970000000001, 2024-03-24T12:34:19.876 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "queued for pg", 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872021+0000", 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872021+0000", 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:19.877 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872021+0000", 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3099999999999997e-07 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872028+0000", 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6409999999999996e-06 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.878 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872039+0000", 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0761000000000001e-05 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872048+0000", 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5589999999999994e-06 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.879 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_repop(client.24815.0:107938 8.a e187/158 8:544caf75:::benchmark_data_smithi029_108893_object107937:head v 187'6619, mlcod=187'6617)", 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:34:19.872812+0000", 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 0.00014706800000000001, 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025910800000000003, 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872812+0000", 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.880 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872812+0000", 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872812+0000", 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7999999999999995e-07 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:19.881 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872860+0000", 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7091e-05 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872871+0000", 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1793e-05 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872878+0000", 2024-03-24T12:34:19.882 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1369999999999996e-06 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872953+0000", 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4103999999999994e-05 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:19.872973+0000", 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0893000000000001e-05 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:19.883 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:19.884 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:19.884 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:19.884 INFO:teuthology.orchestra.run.smithi190.stdout: ], 2024-03-24T12:34:19.884 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 3 2024-03-24T12:34:19.884 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:34:19.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:19.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:20.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:20.068 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:20.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:20.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:20.194 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:20.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:20.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:20.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:20.333 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:20.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:20.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:20 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:34:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:20 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:34:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:20 smithi190 ceph-mon[33748]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T12:34:20.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:20 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:34:20.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:20 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:34:20.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:20 smithi029 ceph-mon[41500]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T12:34:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:20 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:34:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:20 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:34:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:20 smithi029 ceph-mon[29919]: osdmap e187: 8 total, 8 up, 8 in 2024-03-24T12:34:20.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:20.469 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:20.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:20.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:20.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:20.605 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:20.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:20.684 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 85 16 108673 108657 79.8807 59.0625 0.0111665 0.012513 2024-03-24T12:34:20.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:20.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:20.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:20.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:20.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:20.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:20.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:20.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:20.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:21.007 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:21.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:21.063 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T12:34:21.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:21.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:21.156 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:21.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:21.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:21.282 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:21.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:21.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:21.374 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:21 smithi190 ceph-mon[33748]: pgmap v334: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 882 op/s; 3616/318513 objects misplaced (1.135%); 8.4 MiB/s, 134 objects/s recovering 2024-03-24T12:34:21.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:21.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:21 smithi029 ceph-mon[29919]: pgmap v334: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 882 op/s; 3616/318513 objects misplaced (1.135%); 8.4 MiB/s, 134 objects/s recovering 2024-03-24T12:34:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:21 smithi029 ceph-mon[41500]: pgmap v334: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 55 MiB/s wr, 882 op/s; 3616/318513 objects misplaced (1.135%); 8.4 MiB/s, 134 objects/s recovering 2024-03-24T12:34:21.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:21.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:21.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:21.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:21.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:21.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:21.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:21.684 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 86 16 109699 109683 79.6975 64.125 0.00687913 0.0125427 2024-03-24T12:34:21.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:21.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:21.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:21.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:21.856 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:21.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:21.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:21.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:21.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:21.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:22.097 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:22.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:22 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:34:22.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:22 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:34:22.150 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:22.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:22.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:22.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:22.283 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:22.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:22.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:22.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:22.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:22 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:34:22.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:22 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:34:22.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:22 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:34:22.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:22 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:34:22.434 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:22.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:22.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:22.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:22.570 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:22.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:22.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:22.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:22.684 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 87 15 110739 110724 79.5291 65.0625 0.0193117 0.0125689 2024-03-24T12:34:22.719 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:22.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:22.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:22.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:22.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:22.853 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:22.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:22.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:23.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:23.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:23.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:23.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:23.141 INFO:tasks.thrashosds.thrasher:j is {'epoch': 185, 'fsid': 'bc017092-e9d8-11ee-95cd-87774f69a715', 'created': '2024-03-24T12:20:18.712998+0000', 'modified': '2024-03-24T12:34:09.615233+0000', 'last_up_change': '2024-03-24T12:25:48.676565+0000', 'last_in_change': '2024-03-24T12:25:30.369664+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T12:23:41.491597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-24T12:26:01.465080+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T12:26:02.100021+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T12:26:03.473862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T12:26:05.573828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T12:26:07.645083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-03-24T12:32:39.584334+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0aa1c49b-7887-4f74-b03c-08cc0adec90b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6801', 'nonce': 2514818464}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6802', 'nonce': 2514818464}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6804', 'nonce': 2514818464}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6803', 'nonce': 2514818464}]}, 'public_addr': '172.21.15.29:6801/2514818464', 'cluster_addr': '172.21.15.29:6802/2514818464', 'heartbeat_back_addr': '172.21.15.29:6804/2514818464', 'heartbeat_front_addr': '172.21.15.29:6803/2514818464', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99d8ef8f-eb59-4882-ab51-b398e6b53bcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6805', 'nonce': 2923396572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6806', 'nonce': 2923396572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6808', 'nonce': 2923396572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6807', 'nonce': 2923396572}]}, 'public_addr': '172.21.15.29:6805/2923396572', 'cluster_addr': '172.21.15.29:6806/2923396572', 'heartbeat_back_addr': '172.21.15.29:6808/2923396572', 'heartbeat_front_addr': '172.21.15.29:6807/2923396572', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c3f83505-25ee-4217-b60a-081c36092e92', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 142, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6809', 'nonce': 126032915}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6810', 'nonce': 126032915}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6812', 'nonce': 126032915}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6811', 'nonce': 126032915}]}, 'public_addr': '172.21.15.29:6809/126032915', 'cluster_addr': '172.21.15.29:6810/126032915', 'heartbeat_back_addr': '172.21.15.29:6812/126032915', 'heartbeat_front_addr': '172.21.15.29:6811/126032915', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f5e479a8-459a-4315-a808-711a7b84bb96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 181, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6813', 'nonce': 1080171824}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6814', 'nonce': 1080171824}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6816', 'nonce': 1080171824}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6815', 'nonce': 1080171824}]}, 'public_addr': '172.21.15.29:6813/1080171824', 'cluster_addr': '172.21.15.29:6814/1080171824', 'heartbeat_back_addr': '172.21.15.29:6816/1080171824', 'heartbeat_front_addr': '172.21.15.29:6815/1080171824', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '5e420304-2798-4d11-9688-d0d89482008c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 3690258869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6801', 'nonce': 3690258869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6803', 'nonce': 3690258869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 3690258869}]}, 'public_addr': '172.21.15.190:6800/3690258869', 'cluster_addr': '172.21.15.190:6801/3690258869', 'heartbeat_back_addr': '172.21.15.190:6803/3690258869', 'heartbeat_front_addr': '172.21.15.190:6802/3690258869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1213431a-652c-4cbf-b32a-36335430c666', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 1327220488}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6805', 'nonce': 1327220488}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6807', 'nonce': 1327220488}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 1327220488}]}, 'public_addr': '172.21.15.190:6804/1327220488', 'cluster_addr': '172.21.15.190:6805/1327220488', 'heartbeat_back_addr': '172.21.15.190:6807/1327220488', 'heartbeat_front_addr': '172.21.15.190:6806/1327220488', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '0712e511-0257-44f3-a82f-ee09ab957c0c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 158, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 1295064976}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6809', 'nonce': 1295064976}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6811', 'nonce': 1295064976}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 1295064976}]}, 'public_addr': '172.21.15.190:6808/1295064976', 'cluster_addr': '172.21.15.190:6809/1295064976', 'heartbeat_back_addr': '172.21.15.190:6811/1295064976', 'heartbeat_front_addr': '172.21.15.190:6810/1295064976', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7e36bde4-14d0-4e7e-9a82-2bb97859dbda', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 160, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 110963949}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6813', 'nonce': 110963949}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6815', 'nonce': 110963949}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 110963949}]}, 'public_addr': '172.21.15.190:6812/110963949', 'cluster_addr': '172.21.15.190:6813/110963949', 'heartbeat_back_addr': '172.21.15.190:6815/110963949', 'heartbeat_front_addr': '172.21.15.190:6814/110963949', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:33:40.506116+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:34:05.748585+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:33:39.494402+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:33:39.622360+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:33:01.570288+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:33:59.195979+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:33:16.435153+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:32:58.231586+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '8.0', 'osds': [3, 5, 2]}], 'pg_upmap_items': [{'pgid': '3.11', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '8.0', 'osds': [3, 5, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.29:0/2633022206': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2612078930': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2365028933': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4113609697': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4130171883': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/3697529912': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2929430696': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/327626505': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/3715448582': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/861146182': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2109454792': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/1631806715': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:6800/852212806': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/1823252077': '2024-03-25T12:21:15.151901+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:34:23.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:34:23.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:23.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:23.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:23.245 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:23 smithi190 ceph-mon[33748]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T12:34:23.245 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:23 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:34:23.245 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:23 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:34:23.245 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:23 smithi190 ceph-mon[33748]: pgmap v336: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 3616/326211 objects misplaced (1.108%); 5.4 MiB/s, 86 objects/s recovering 2024-03-24T12:34:23.329 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:34:23.329 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:34:23.330 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-03-24T12:34:23.330 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-03-24T12:34:23.330 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:34:23.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:23.384 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:23.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:23.411 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:23 smithi029 ceph-mon[41500]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T12:34:23.411 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:23 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:34:23.411 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:23 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:34:23.412 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:23 smithi029 ceph-mon[41500]: pgmap v336: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 3616/326211 objects misplaced (1.108%); 5.4 MiB/s, 86 objects/s recovering 2024-03-24T12:34:23.412 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:23 smithi029 ceph-mon[29919]: osdmap e188: 8 total, 8 up, 8 in 2024-03-24T12:34:23.412 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:23 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:34:23.412 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:23 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:34:23.412 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:23 smithi029 ceph-mon[29919]: pgmap v336: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 3616/326211 objects misplaced (1.108%); 5.4 MiB/s, 86 objects/s recovering 2024-03-24T12:34:23.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:23.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:23.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:23.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:23.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:23.672 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:23.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:23.685 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 88 16 111658 111642 79.2767 57.375 0.0367918 0.0126089 2024-03-24T12:34:23.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:23.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:23.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:23.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:23.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:23.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:23.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:23.965 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:34:24.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:24.097 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:24.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:24.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T12:34:24.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:24.245 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:24.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:24.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:24.383 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:24.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:24.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:24 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:34:24.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:24 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:34:24.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:24 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:34:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:24 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:34:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:24 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:34:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:24 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:34:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:24 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:34:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:24 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:34:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:24 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:34:24.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:24.509 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:24.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:24.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:24.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:24.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:24.656 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:24.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:24.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:24.686 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 89 16 112765 112749 79.1624 69.1875 0.0108928 0.0126274 2024-03-24T12:34:24.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:24.798 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:24.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:24.899 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:24.943 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:24.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:25.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:25.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:25.085 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:25.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:25.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:25.223 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:25.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:25.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:25.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:25.374 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:25.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:25.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:25.514 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:25.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:25.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:25.657 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:25.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:25.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:25.686 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Total time run: 90.0095 2024-03-24T12:34:25.686 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Total writes made: 113857 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Write size: 65536 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Object size: 65536 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Bandwidth (MB/sec): 79.059 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev Bandwidth: 36.9107 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max bandwidth (MB/sec): 216.25 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min bandwidth (MB/sec): 0 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Average IOPS: 1264 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev IOPS: 590.583 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max IOPS: 3460 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min IOPS: 0 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Average Latency(s): 0.0126444 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev Latency(s): 0.0355948 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max latency(s): 2.54295 2024-03-24T12:34:25.687 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min latency(s): 0.00120017 2024-03-24T12:34:25.693 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T12:34:25.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:25.799 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:25.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:25.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:25.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:25.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:26.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:26.049 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:34:26.089 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:26.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:26.095 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:34:26.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:25 smithi029 ceph-mon[29919]: pgmap v337: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 3236/332601 objects misplaced (0.973%); 8.4 MiB/s, 135 objects/s recovering 2024-03-24T12:34:26.096 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:25 smithi029 ceph-mon[41500]: pgmap v337: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 3236/332601 objects misplaced (0.973%); 8.4 MiB/s, 135 objects/s recovering 2024-03-24T12:34:26.136 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:34:26.179 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:34:26.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:26.220 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:34:26.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:26.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:26.259 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:34:26.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:26.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:26.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:26.350 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:34:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:25 smithi190 ceph-mon[33748]: pgmap v337: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s; 3236/332601 objects misplaced (0.973%); 8.4 MiB/s, 135 objects/s recovering 2024-03-24T12:34:26.389 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:34:26.394 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:34:26.394 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:34:26.394 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:34:26.394 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:34:26.394 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:34:26.395 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:34:26.395 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:34:26.395 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:34:26.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:26.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:26.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:26.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:26.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:26.605 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:26.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:26.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:26.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:26.742 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:26.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:26.844 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:26.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:26.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:26.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:27.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:27.009 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:27.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:27.032 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:26 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2632699368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:34:27.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:27.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:27.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:27.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:34:27.174 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:34:27.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:27.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:27.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:27.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:27.387 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:26 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2632699368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:34:27.389 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:26 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2632699368' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:34:27.416 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:27.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:27.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:27.555 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:27.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:27.657 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:27.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:34:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:34:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:34:27.678 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:34:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:27.705 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:27.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:27.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:27.856 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:27.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:27.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:27.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T12:34:27.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:27.997 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:27.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:27.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:28.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:28.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:27 smithi190 ceph-mon[33748]: pgmap v338: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 821 op/s; 3236/335352 objects misplaced (0.965%); 8.4 MiB/s, 135 objects/s recovering 2024-03-24T12:34:28.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:34:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:34:28.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:27 smithi029 ceph-mon[29919]: pgmap v338: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 821 op/s; 3236/335352 objects misplaced (0.965%); 8.4 MiB/s, 135 objects/s recovering 2024-03-24T12:34:28.145 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:28.145 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:27 smithi029 ceph-mon[41500]: pgmap v338: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 51 MiB/s wr, 821 op/s; 3236/335352 objects misplaced (0.965%); 8.4 MiB/s, 135 objects/s recovering 2024-03-24T12:34:28.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:28.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:28.246 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:28.283 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:28.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:28.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:28.303 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:28.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:28.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:28.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:28.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:28.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:28.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:28.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:28.704 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:28.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:28.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:28.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:28.845 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:28.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:28.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:28.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:28.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:28.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:29.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:29.111 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:29.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:29.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:29.251 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:29.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:29.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:29.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:29.370 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:29 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:29.389 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:29.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:29.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:29.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:29 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:29.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:29 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:29.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:29.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:29.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:29.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:29.656 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:29.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:29.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:29.798 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:29.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:29.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:29.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:29.926 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:29.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:29.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:30.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:30.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:30.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:30.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:30.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:30.191 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:34:30.201 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:30.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:30.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:30 smithi190 ceph-mon[33748]: pgmap v339: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 1.03k op/s; 3236/337368 objects misplaced (0.959%); 7.1 MiB/s, 113 objects/s recovering 2024-03-24T12:34:30.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:30.341 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:30.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:30.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:30 smithi029 ceph-mon[29919]: pgmap v339: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 1.03k op/s; 3236/337368 objects misplaced (0.959%); 7.1 MiB/s, 113 objects/s recovering 2024-03-24T12:34:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:30 smithi029 ceph-mon[41500]: pgmap v339: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 1.03k op/s; 3236/337368 objects misplaced (0.959%); 7.1 MiB/s, 113 objects/s recovering 2024-03-24T12:34:30.442 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:30.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:30.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:30.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:30.495 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:34:30.495 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:34:30.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:30.610 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:30.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:30.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:30.735 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:30.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:30.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:30.869 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:30.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:30.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:30.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:31.005 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:31.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:31.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:31.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:31.140 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:31.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:31.200 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:31 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4017053067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:34:31.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T12:34:31.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:31.279 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:31.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:31.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:31.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:31.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:31.419 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:31.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:31.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:31 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4017053067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:34:31.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:31 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4017053067' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:34:31.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:31.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:31.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:31.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:31.567 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:31.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:31.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:31.693 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:31.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:31.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:31.820 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:31.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:31.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:31.857 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:34:31.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:31.952 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:31.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:31.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:32.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:32.080 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:32.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:32.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:32.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:32 smithi029 ceph-mon[41500]: pgmap v340: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 977 op/s; 2609/336522 objects misplaced (0.775%); 6.9 MiB/s, 110 objects/s recovering 2024-03-24T12:34:32.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:32 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:34:32.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:32 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4017053067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:34:32.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:32 smithi029 ceph-mon[41500]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T12:34:32.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:32 smithi029 ceph-mon[41500]: 8.1 scrub starts 2024-03-24T12:34:32.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:32 smithi029 ceph-mon[29919]: pgmap v340: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 977 op/s; 2609/336522 objects misplaced (0.775%); 6.9 MiB/s, 110 objects/s recovering 2024-03-24T12:34:32.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:32 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:34:32.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:32 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4017053067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:34:32.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:32 smithi029 ceph-mon[29919]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T12:34:32.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:32 smithi029 ceph-mon[29919]: 8.1 scrub starts 2024-03-24T12:34:32.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:32.221 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:32.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:32.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:32.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:32.356 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:32.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:32 smithi190 ceph-mon[33748]: pgmap v340: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.8 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 977 op/s; 2609/336522 objects misplaced (0.775%); 6.9 MiB/s, 110 objects/s recovering 2024-03-24T12:34:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:32 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:34:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:32 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4017053067' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:34:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:32 smithi190 ceph-mon[33748]: osdmap e189: 8 total, 8 up, 8 in 2024-03-24T12:34:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:32 smithi190 ceph-mon[33748]: 8.1 scrub starts 2024-03-24T12:34:32.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:32.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:32.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:32.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:32.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:32.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:32.630 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:32.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:32.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:32.761 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:32.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:32.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:32.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:32.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:32.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:33.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:33.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:33.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:33.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:33.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:33.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:33.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:33.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:33 smithi029 ceph-mon[41500]: 8.c scrub starts 2024-03-24T12:34:33.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:33.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:33 smithi029 ceph-mon[29919]: 8.c scrub starts 2024-03-24T12:34:33.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:33.289 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:33.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:33 smithi190 ceph-mon[33748]: 8.c scrub starts 2024-03-24T12:34:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:33.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:33.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:33.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:33.584 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:33.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:33.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:33.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:33.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:33.718 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:33.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:33.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:33.850 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:33.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:33.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:33.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:33.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:34.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:34.126 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:34.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:34.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:34.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:34 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:34:34.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:34 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:34:34.189 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:34 smithi190 ceph-mon[33748]: pgmap v342: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 988 op/s; 2609/328503 objects misplaced (0.794%); 6.9 MiB/s, 110 objects/s recovering 2024-03-24T12:34:34.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:34.269 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:34.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:34.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:34.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:34.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:34 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:34:34.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:34 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:34:34.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:34 smithi029 ceph-mon[29919]: pgmap v342: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 988 op/s; 2609/328503 objects misplaced (0.794%); 6.9 MiB/s, 110 objects/s recovering 2024-03-24T12:34:34.374 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:34 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:34:34.374 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:34 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:34:34.375 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:34 smithi029 ceph-mon[41500]: pgmap v342: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 6.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 33 MiB/s wr, 988 op/s; 2609/328503 objects misplaced (0.794%); 6.9 MiB/s, 110 objects/s recovering 2024-03-24T12:34:34.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:34.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:34.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:34.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:34.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:34.544 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:34.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:34.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:34.686 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:34.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:34.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:34.826 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:34.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:34.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:34.920 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:34:34.920 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:34:34.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:34.958 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:34:34.973 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:34.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:34.974 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:35.075 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:35.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:35.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:35.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:35.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:35 smithi190 ceph-mon[33748]: 8.4 scrub starts 2024-03-24T12:34:35.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:35 smithi190 ceph-mon[33748]: 8.1 scrub ok 2024-03-24T12:34:35.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:35 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:34:35.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:35 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:34:35.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:35 smithi190 ceph-mon[33748]: pgmap v343: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 146 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 20 MiB/s wr, 964 op/s; 1928/322827 objects misplaced (0.597%); 7.4 MiB/s, 118 objects/s recovering 2024-03-24T12:34:35.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:35 smithi029 ceph-mon[41500]: 8.4 scrub starts 2024-03-24T12:34:35.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:35 smithi029 ceph-mon[41500]: 8.1 scrub ok 2024-03-24T12:34:35.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:35 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:34:35.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:35 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:34:35.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:35 smithi029 ceph-mon[41500]: pgmap v343: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 146 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 20 MiB/s wr, 964 op/s; 1928/322827 objects misplaced (0.597%); 7.4 MiB/s, 118 objects/s recovering 2024-03-24T12:34:35.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:35 smithi029 ceph-mon[29919]: 8.4 scrub starts 2024-03-24T12:34:35.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:35 smithi029 ceph-mon[29919]: 8.1 scrub ok 2024-03-24T12:34:35.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:35 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:34:35.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:35 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:34:35.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:35 smithi029 ceph-mon[29919]: pgmap v343: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 146 active+clean; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 20 MiB/s wr, 964 op/s; 1928/322827 objects misplaced (0.597%); 7.4 MiB/s, 118 objects/s recovering 2024-03-24T12:34:35.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:35.259 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:35.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:35.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:35.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:35.401 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:35.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:35.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:35.503 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:35.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:35.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:35.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:35.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T12:34:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:35.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:35.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:35.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:35.812 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:35.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:35.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:35.958 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:35.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:36.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:36.089 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:36.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:36.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:36.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:36.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:36.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:36.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:36.328 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:34:36.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[29919]: 8.c scrub ok 2024-03-24T12:34:36.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[29919]: 5.1 scrub starts 2024-03-24T12:34:36.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[29919]: 5.1 scrub ok 2024-03-24T12:34:36.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[29919]: 8.4 scrub ok 2024-03-24T12:34:36.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[29919]: 4.6 scrub starts 2024-03-24T12:34:36.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[29919]: 4.6 scrub ok 2024-03-24T12:34:36.330 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[41500]: 8.c scrub ok 2024-03-24T12:34:36.330 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[41500]: 5.1 scrub starts 2024-03-24T12:34:36.330 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[41500]: 5.1 scrub ok 2024-03-24T12:34:36.330 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[41500]: 8.4 scrub ok 2024-03-24T12:34:36.330 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[41500]: 4.6 scrub starts 2024-03-24T12:34:36.330 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:36 smithi029 ceph-mon[41500]: 4.6 scrub ok 2024-03-24T12:34:36.337 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:34:36.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:36.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:36.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:36 smithi190 ceph-mon[33748]: 8.c scrub ok 2024-03-24T12:34:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:36 smithi190 ceph-mon[33748]: 5.1 scrub starts 2024-03-24T12:34:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:36 smithi190 ceph-mon[33748]: 5.1 scrub ok 2024-03-24T12:34:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:36 smithi190 ceph-mon[33748]: 8.4 scrub ok 2024-03-24T12:34:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:36 smithi190 ceph-mon[33748]: 4.6 scrub starts 2024-03-24T12:34:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:36 smithi190 ceph-mon[33748]: 4.6 scrub ok 2024-03-24T12:34:36.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:34:36.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:36.515 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:36.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:36.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:36.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:36.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:36.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:36.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:36.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:36.776 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:36.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:36.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:36.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:36.919 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:36.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:37.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:37.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:37.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:37.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:37.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:37.187 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:37.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:37.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:37.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:37.329 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:37.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:37 smithi190 ceph-mon[33748]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:34:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:37 smithi190 ceph-mon[33748]: pgmap v344: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 146 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 962 op/s; 1928/319992 objects misplaced (0.603%); 7.4 MiB/s, 118 objects/s recovering 2024-03-24T12:34:37.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:37 smithi029 ceph-mon[29919]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:34:37.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:37 smithi029 ceph-mon[29919]: pgmap v344: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 146 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 962 op/s; 1928/319992 objects misplaced (0.603%); 7.4 MiB/s, 118 objects/s recovering 2024-03-24T12:34:37.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:37 smithi029 ceph-mon[41500]: from='client.24863 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:34:37.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:37 smithi029 ceph-mon[41500]: pgmap v344: 148 pgs: 1 active+clean+scrubbing, 1 active+remapped+backfilling, 146 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 14 MiB/s wr, 962 op/s; 1928/319992 objects misplaced (0.603%); 7.4 MiB/s, 118 objects/s recovering 2024-03-24T12:34:37.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:37.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:37.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:37.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:37.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:37.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:37.702 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:37.734 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:37.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:37.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:37.860 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:37.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:37.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:34:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:34:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:34:37.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:37.991 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:37.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:37.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:38.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:38.128 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:38.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:38.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:38 smithi190 ceph-mon[33748]: 8.8 scrub starts 2024-03-24T12:34:38.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:38 smithi190 ceph-mon[33748]: 8.7 scrub starts 2024-03-24T12:34:38.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:38 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:38.134 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:34:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:34:38.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:38.266 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:38.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:38.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:38.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:38.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:38.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:38.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:38 smithi029 ceph-mon[41500]: 8.8 scrub starts 2024-03-24T12:34:38.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:38 smithi029 ceph-mon[41500]: 8.7 scrub starts 2024-03-24T12:34:38.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:38 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:38 smithi029 ceph-mon[29919]: 8.8 scrub starts 2024-03-24T12:34:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:38 smithi029 ceph-mon[29919]: 8.7 scrub starts 2024-03-24T12:34:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:38 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:38.494 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:38.525 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:38.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:38.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:38.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:38.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:38.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:38.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:38.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:38.793 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:38.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:38.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:38.921 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:38.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:39.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:39.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:39.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:39.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:39.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:39 smithi190 ceph-mon[33748]: 8.6 scrub starts 2024-03-24T12:34:39.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:39 smithi190 ceph-mon[33748]: pgmap v345: 148 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 2.0 MiB/s wr, 965 op/s; 1928/310332 objects misplaced (0.621%); 7.4 MiB/s, 118 objects/s recovering 2024-03-24T12:34:39.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:39.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:39 smithi029 ceph-mon[41500]: 8.6 scrub starts 2024-03-24T12:34:39.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:39 smithi029 ceph-mon[41500]: pgmap v345: 148 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 2.0 MiB/s wr, 965 op/s; 1928/310332 objects misplaced (0.621%); 7.4 MiB/s, 118 objects/s recovering 2024-03-24T12:34:39.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:39 smithi029 ceph-mon[29919]: 8.6 scrub starts 2024-03-24T12:34:39.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:39 smithi029 ceph-mon[29919]: pgmap v345: 148 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 145 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 2.0 MiB/s wr, 965 op/s; 1928/310332 objects misplaced (0.621%); 7.4 MiB/s, 118 objects/s recovering 2024-03-24T12:34:39.183 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:39.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:39.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:39.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:39.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:39.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:39.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:39.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:39.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:39.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:39.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:39.570 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:39.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:39.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:39.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:39.710 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:39.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:39.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:39.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:39.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:39.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:39.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:39.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:39.994 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:39.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:40.086 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:34:40.087 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:34:40.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:40.132 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:40.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:40.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:40.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:40.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:40.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:40.303 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:40 smithi190 ceph-mon[33748]: 8.8 scrub ok 2024-03-24T12:34:40.375 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:40.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:40.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:40.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:40 smithi029 ceph-mon[29919]: 8.8 scrub ok 2024-03-24T12:34:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:40 smithi029 ceph-mon[41500]: 8.8 scrub ok 2024-03-24T12:34:40.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:40.534 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:40.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:40.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:40.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:40.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:40.669 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:40.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:40.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:40.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:40.803 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:40.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:40.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:34:40.892 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-03-24T12:34:34.538386+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7039818102,"num_objects":107609,"num_object_clones":0,"num_object_copies":322827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1928,"num_objects_unfound":0,"num_objects_dirty":107609,"num_whiteouts":0,"num_read":1437,"num_read_kb":1283,"num_write":120920,"num_write_kb":7289310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4508,"num_bytes_recovered":295831466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86171,"ondisk_log_size":86171,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":22995816,"kb_used_data":20992604,"kb_used_omap":12,"kb_used_meta":2002739,"kb_avail":726965400,"statfs":{"total":767960285184,"available":744412569600,"internally_reserved":0,"allocated":21496426496,"data_stored":21493171555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2050805297},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-213581800,"num_objects":-3258,"num_object_clones":0,"num_object_copies":-9774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1308,"num_objects_unfound":0,"num_objects_dirty":-3258,"num_whiteouts":0,"num_read":11,"num_read_kb":11,"num_write":9636,"num_write_kb":204033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1187,"num_bytes_recovered":77791232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003176"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030632+0000","last_change":"2024-03-24T12:32:55.993821+0000","last_active":"2024-03-24T12:34:31.030632+0000","last_peered":"2024-03-24T12:34:31.030632+0000","last_clean":"2024-03-24T12:34:31.030632+0000","last_became_active":"2024-03-24T12:26:09.350521+0000","last_became_peered":"2024-03-24T12:26:09.350521+0000","last_unstale":"2024-03-24T12:34:31.030632+0000","last_undegraded":"2024-03-24T12:34:31.030632+0000","last_fullsized":"2024-03-24T12:34:31.030632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:55.993649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.993649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379363+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.040645+0000","last_change":"2024-03-24T12:33:01.510002+0000","last_active":"2024-03-24T12:34:31.040645+0000","last_peered":"2024-03-24T12:34:31.040645+0000","last_clean":"2024-03-24T12:34:31.040645+0000","last_became_active":"2024-03-24T12:26:07.335517+0000","last_became_peered":"2024-03-24T12:26:07.335517+0000","last_unstale":"2024-03-24T12:34:31.040645+0000","last_undegraded":"2024-03-24T12:34:31.040645+0000","last_fullsized":"2024-03-24T12:34:31.040645+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:01.509739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:01.509739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":273,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.081460+0000","last_change":"2024-03-24T12:33:31.312660+0000","last_active":"2024-03-24T12:34:22.081460+0000","last_peered":"2024-03-24T12:34:22.081460+0000","last_clean":"2024-03-24T12:34:22.081460+0000","last_became_active":"2024-03-24T12:26:03.170822+0000","last_became_peered":"2024-03-24T12:26:03.170822+0000","last_unstale":"2024-03-24T12:34:22.081460+0000","last_undegraded":"2024-03-24T12:34:22.081460+0000","last_fullsized":"2024-03-24T12:34:22.081460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:31.312530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:31.312530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.425303+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":294,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.026066+0000","last_change":"2024-03-24T12:33:12.325849+0000","last_active":"2024-03-24T12:34:31.026066+0000","last_peered":"2024-03-24T12:34:31.026066+0000","last_clean":"2024-03-24T12:34:31.026066+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:34:31.026066+0000","last_undegraded":"2024-03-24T12:34:31.026066+0000","last_fullsized":"2024-03-24T12:34:31.026066+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:12.325694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.325694+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:40.240287+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.046191+0000","last_change":"2024-03-24T12:32:56.130518+0000","last_active":"2024-03-24T12:34:31.046191+0000","last_peered":"2024-03-24T12:34:31.046191+0000","last_clean":"2024-03-24T12:34:31.046191+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:34:31.046191+0000","last_undegraded":"2024-03-24T12:34:31.046191+0000","last_fullsized":"2024-03-24T12:34:31.046191+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:56.130373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:56.130373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.026153+0000","last_change":"2024-03-24T12:32:53.023292+0000","last_active":"2024-03-24T12:34:31.026153+0000","last_peered":"2024-03-24T12:34:31.026153+0000","last_clean":"2024-03-24T12:34:31.026153+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:34:31.026153+0000","last_undegraded":"2024-03-24T12:34:31.026153+0000","last_fullsized":"2024-03-24T12:34:31.026153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:53.023160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:53.023160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379378+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":275,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.049587+0000","last_change":"2024-03-24T12:33:09.263042+0000","last_active":"2024-03-24T12:34:31.049587+0000","last_peered":"2024-03-24T12:34:31.049587+0000","last_clean":"2024-03-24T12:34:31.049587+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:34:31.049587+0000","last_undegraded":"2024-03-24T12:34:31.049587+0000","last_fullsized":"2024-03-24T12:34:31.049587+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:09.262901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:09.262901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"90'22","reported_seq":310,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.041414+0000","last_change":"2024-03-24T12:33:36.263856+0000","last_active":"2024-03-24T12:34:31.041414+0000","last_peered":"2024-03-24T12:34:31.041414+0000","last_clean":"2024-03-24T12:34:31.041414+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:34:31.041414+0000","last_undegraded":"2024-03-24T12:34:31.041414+0000","last_fullsized":"2024-03-24T12:34:31.041414+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:33:36.263707+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.263707+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:59.187313+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":268,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.080048+0000","last_change":"2024-03-24T12:33:14.363693+0000","last_active":"2024-03-24T12:34:22.080048+0000","last_peered":"2024-03-24T12:34:22.080048+0000","last_clean":"2024-03-24T12:34:22.080048+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:34:22.080048+0000","last_undegraded":"2024-03-24T12:34:22.080048+0000","last_fullsized":"2024-03-24T12:34:22.080048+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:14.363553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:33:14.363553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":215,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":266,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030597+0000","last_change":"2024-03-24T12:32:51.275896+0000","last_active":"2024-03-24T12:34:31.030597+0000","last_peered":"2024-03-24T12:34:31.030597+0000","last_clean":"2024-03-24T12:34:31.030597+0000","last_became_active":"2024-03-24T12:26:07.331562+0000","last_became_peered":"2024-03-24T12:26:07.331562+0000","last_unstale":"2024-03-24T12:34:31.030597+0000","last_undegraded":"2024-03-24T12:34:31.030597+0000","last_fullsized":"2024-03-24T12:34:31.030597+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:51.275761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:51.275761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.231258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.031359+0000","last_change":"2024-03-24T12:33:33.178127+0000","last_active":"2024-03-24T12:34:31.031359+0000","last_peered":"2024-03-24T12:34:31.031359+0000","last_clean":"2024-03-24T12:34:31.031359+0000","last_became_active":"2024-03-24T12:26:03.161544+0000","last_became_peered":"2024-03-24T12:26:03.161544+0000","last_unstale":"2024-03-24T12:34:31.031359+0000","last_undegraded":"2024-03-24T12:34:31.031359+0000","last_fullsized":"2024-03-24T12:34:31.031359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:33.177938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:33.177938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:49.571945+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":294,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030858+0000","last_change":"2024-03-24T12:33:34.398616+0000","last_active":"2024-03-24T12:34:31.030858+0000","last_peered":"2024-03-24T12:34:31.030858+0000","last_clean":"2024-03-24T12:34:31.030858+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:34:31.030858+0000","last_undegraded":"2024-03-24T12:34:31.030858+0000","last_fullsized":"2024-03-24T12:34:31.030858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:34.398401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.398401+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:55.759200+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030967+0000","last_change":"2024-03-24T12:33:21.155477+0000","last_active":"2024-03-24T12:34:31.030967+0000","last_peered":"2024-03-24T12:34:31.030967+0000","last_clean":"2024-03-24T12:34:31.030967+0000","last_became_active":"2024-03-24T12:26:09.350721+0000","last_became_peered":"2024-03-24T12:26:09.350721+0000","last_unstale":"2024-03-24T12:34:31.030967+0000","last_undegraded":"2024-03-24T12:34:31.030967+0000","last_fullsized":"2024-03-24T12:34:31.030967+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.155345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.155345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:50.784827+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.033589+0000","last_change":"2024-03-24T12:33:37.143601+0000","last_active":"2024-03-24T12:34:31.033589+0000","last_peered":"2024-03-24T12:34:31.033589+0000","last_clean":"2024-03-24T12:34:31.033589+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:34:31.033589+0000","last_undegraded":"2024-03-24T12:34:31.033589+0000","last_fullsized":"2024-03-24T12:34:31.033589+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:37.143487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:33:37.143487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:58.049705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.031056+0000","last_change":"2024-03-24T12:33:23.095826+0000","last_active":"2024-03-24T12:34:31.031056+0000","last_peered":"2024-03-24T12:34:31.031056+0000","last_clean":"2024-03-24T12:34:31.031056+0000","last_became_active":"2024-03-24T12:26:03.159997+0000","last_became_peered":"2024-03-24T12:26:03.159997+0000","last_unstale":"2024-03-24T12:34:31.031056+0000","last_undegraded":"2024-03-24T12:34:31.031056+0000","last_fullsized":"2024-03-24T12:34:31.031056+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:23.095706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:33:23.095706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:52.991000+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":296,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:32.424617+0000","last_change":"2024-03-24T12:34:32.424665+0000","last_active":"2024-03-24T12:34:32.424617+0000","last_peered":"2024-03-24T12:34:32.424617+0000","last_clean":"2024-03-24T12:34:32.424617+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:34:32.424617+0000","last_undegraded":"2024-03-24T12:34:32.424617+0000","last_fullsized":"2024-03-24T12:34:32.424617+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T12:34:32.424537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:32.424537+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:00.556829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.028157+0000","last_change":"2024-03-24T12:31:23.766176+0000","last_active":"2024-03-24T12:34:31.028157+0000","last_peered":"2024-03-24T12:34:31.028157+0000","last_clean":"2024-03-24T12:34:31.028157+0000","last_became_active":"2024-03-24T12:26:09.643671+0000","last_became_peered":"2024-03-24T12:26:09.643671+0000","last_unstale":"2024-03-24T12:34:31.028157+0000","last_undegraded":"2024-03-24T12:34:31.028157+0000","last_fullsized":"2024-03-24T12:34:31.028157+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:23.766078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:31:23.766078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.038986+0000","last_change":"2024-03-24T12:33:14.795387+0000","last_active":"2024-03-24T12:34:31.038986+0000","last_peered":"2024-03-24T12:34:31.038986+0000","last_clean":"2024-03-24T12:34:31.038986+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:34:31.038986+0000","last_undegraded":"2024-03-24T12:34:31.038986+0000","last_fullsized":"2024-03-24T12:34:31.038986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:14.795243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:14.795243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":272,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.081048+0000","last_change":"2024-03-24T12:32:49.778838+0000","last_active":"2024-03-24T12:34:22.081048+0000","last_peered":"2024-03-24T12:34:22.081048+0000","last_clean":"2024-03-24T12:34:22.081048+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:34:22.081048+0000","last_undegraded":"2024-03-24T12:34:22.081048+0000","last_fullsized":"2024-03-24T12:34:22.081048+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:49.778759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:32:49.778759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:22.086749+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":303,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.027819+0000","last_change":"2024-03-24T12:32:55.627790+0000","last_active":"2024-03-24T12:34:31.027819+0000","last_peered":"2024-03-24T12:34:31.027819+0000","last_clean":"2024-03-24T12:34:31.027819+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:34:31.027819+0000","last_undegraded":"2024-03-24T12:34:31.027819+0000","last_fullsized":"2024-03-24T12:34:31.027819+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:32:55.627693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.627693+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":263,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.050652+0000","last_change":"2024-03-24T12:29:13.161639+0000","last_active":"2024-03-24T12:34:31.050652+0000","last_peered":"2024-03-24T12:34:31.050652+0000","last_clean":"2024-03-24T12:34:31.050652+0000","last_became_active":"2024-03-24T12:26:09.643738+0000","last_became_peered":"2024-03-24T12:26:09.643738+0000","last_unstale":"2024-03-24T12:34:31.050652+0000","last_undegraded":"2024-03-24T12:34:31.050652+0000","last_fullsized":"2024-03-24T12:34:31.050652+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:13.161510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:29:13.161510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.027056+0000","last_change":"2024-03-24T12:33:01.021282+0000","last_active":"2024-03-24T12:34:31.027056+0000","last_peered":"2024-03-24T12:34:31.027056+0000","last_clean":"2024-03-24T12:34:31.027056+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:34:31.027056+0000","last_undegraded":"2024-03-24T12:34:31.027056+0000","last_fullsized":"2024-03-24T12:34:31.027056+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:01.021148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:33:01.021148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379331+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030947+0000","last_change":"2024-03-24T12:33:18.258903+0000","last_active":"2024-03-24T12:34:31.030947+0000","last_peered":"2024-03-24T12:34:31.030947+0000","last_clean":"2024-03-24T12:34:31.030947+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:34:31.030947+0000","last_undegraded":"2024-03-24T12:34:31.030947+0000","last_fullsized":"2024-03-24T12:34:31.030947+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:18.258782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.258782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:38.365035+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":293,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.049562+0000","last_change":"2024-03-24T12:33:07.191584+0000","last_active":"2024-03-24T12:34:31.049562+0000","last_peered":"2024-03-24T12:34:31.049562+0000","last_clean":"2024-03-24T12:34:31.049562+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:34:31.049562+0000","last_undegraded":"2024-03-24T12:34:31.049562+0000","last_fullsized":"2024-03-24T12:34:31.049562+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:33:07.191431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.191431+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":263,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.034145+0000","last_change":"2024-03-24T12:32:57.372314+0000","last_active":"2024-03-24T12:34:31.034145+0000","last_peered":"2024-03-24T12:34:31.034145+0000","last_clean":"2024-03-24T12:34:31.034145+0000","last_became_active":"2024-03-24T12:26:09.343786+0000","last_became_peered":"2024-03-24T12:26:09.343786+0000","last_unstale":"2024-03-24T12:34:31.034145+0000","last_undegraded":"2024-03-24T12:34:31.034145+0000","last_fullsized":"2024-03-24T12:34:31.034145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:32:57.372183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:57.372183+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.868939+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":388,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.038063+0000","last_change":"2024-03-24T12:33:07.975256+0000","last_active":"2024-03-24T12:34:31.038063+0000","last_peered":"2024-03-24T12:34:31.038063+0000","last_clean":"2024-03-24T12:34:31.038063+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:34:31.038063+0000","last_undegraded":"2024-03-24T12:34:31.038063+0000","last_fullsized":"2024-03-24T12:34:31.038063+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:33:07.975118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.975118+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.277735+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":291,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.040547+0000","last_change":"2024-03-24T12:33:10.675622+0000","last_active":"2024-03-24T12:34:31.040547+0000","last_peered":"2024-03-24T12:34:31.040547+0000","last_clean":"2024-03-24T12:34:31.040547+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:34:31.040547+0000","last_undegraded":"2024-03-24T12:34:31.040547+0000","last_fullsized":"2024-03-24T12:34:31.040547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:10.675383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:10.675383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":292,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.092860+0000","last_change":"2024-03-24T12:33:21.442809+0000","last_active":"2024-03-24T12:34:22.092860+0000","last_peered":"2024-03-24T12:34:22.092860+0000","last_clean":"2024-03-24T12:34:22.092860+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:34:22.092860+0000","last_undegraded":"2024-03-24T12:34:22.092860+0000","last_fullsized":"2024-03-24T12:34:22.092860+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:33:21.442670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.442670+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:30.230404+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":278,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.039149+0000","last_change":"2024-03-24T12:33:03.556099+0000","last_active":"2024-03-24T12:34:31.039149+0000","last_peered":"2024-03-24T12:34:31.039149+0000","last_clean":"2024-03-24T12:34:31.039149+0000","last_became_active":"2024-03-24T12:26:09.348355+0000","last_became_peered":"2024-03-24T12:26:09.348355+0000","last_unstale":"2024-03-24T12:34:31.039149+0000","last_undegraded":"2024-03-24T12:34:31.039149+0000","last_fullsized":"2024-03-24T12:34:31.039149+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:03.555975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.555975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":265,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.093111+0000","last_change":"2024-03-24T12:33:15.244131+0000","last_active":"2024-03-24T12:34:22.093111+0000","last_peered":"2024-03-24T12:34:22.093111+0000","last_clean":"2024-03-24T12:34:22.093111+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:34:22.093111+0000","last_undegraded":"2024-03-24T12:34:22.093111+0000","last_fullsized":"2024-03-24T12:34:22.093111+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:15.244006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:15.244006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:42.717046+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.046275+0000","last_change":"2024-03-24T12:33:28.354892+0000","last_active":"2024-03-24T12:34:31.046275+0000","last_peered":"2024-03-24T12:34:31.046275+0000","last_clean":"2024-03-24T12:34:31.046275+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:34:31.046275+0000","last_undegraded":"2024-03-24T12:34:31.046275+0000","last_fullsized":"2024-03-24T12:34:31.046275+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:28.354367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.354367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:56.014723+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":298,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.037962+0000","last_change":"2024-03-24T12:32:55.882888+0000","last_active":"2024-03-24T12:34:31.037962+0000","last_peered":"2024-03-24T12:34:31.037962+0000","last_clean":"2024-03-24T12:34:31.037962+0000","last_became_active":"2024-03-24T12:26:05.317587+0000","last_became_peered":"2024-03-24T12:26:05.317587+0000","last_unstale":"2024-03-24T12:34:31.037962+0000","last_undegraded":"2024-03-24T12:34:31.037962+0000","last_fullsized":"2024-03-24T12:34:31.037962+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:32:55.882733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.882733+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.277650+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.033556+0000","last_change":"2024-03-24T12:33:01.322667+0000","last_active":"2024-03-24T12:34:31.033556+0000","last_peered":"2024-03-24T12:34:31.033556+0000","last_clean":"2024-03-24T12:34:31.033556+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:34:31.033556+0000","last_undegraded":"2024-03-24T12:34:31.033556+0000","last_fullsized":"2024-03-24T12:34:31.033556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:01.322531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:01.322531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.037058+0000","last_change":"2024-03-24T12:33:12.983236+0000","last_active":"2024-03-24T12:34:31.037058+0000","last_peered":"2024-03-24T12:34:31.037058+0000","last_clean":"2024-03-24T12:34:31.037058+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:34:31.037058+0000","last_undegraded":"2024-03-24T12:34:31.037058+0000","last_fullsized":"2024-03-24T12:34:31.037058+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:12.983157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.983157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:34.212572+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":272,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.081650+0000","last_change":"2024-03-24T12:33:00.920226+0000","last_active":"2024-03-24T12:34:22.081650+0000","last_peered":"2024-03-24T12:34:22.081650+0000","last_clean":"2024-03-24T12:34:22.081650+0000","last_became_active":"2024-03-24T12:26:03.170395+0000","last_became_peered":"2024-03-24T12:26:03.170395+0000","last_unstale":"2024-03-24T12:34:22.081650+0000","last_undegraded":"2024-03-24T12:34:22.081650+0000","last_fullsized":"2024-03-24T12:34:22.081650+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:00.920105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:00.920105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:27.249347+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":284,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.046364+0000","last_change":"2024-03-24T12:33:27.357494+0000","last_active":"2024-03-24T12:34:31.046364+0000","last_peered":"2024-03-24T12:34:31.046364+0000","last_clean":"2024-03-24T12:34:31.046364+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:34:31.046364+0000","last_undegraded":"2024-03-24T12:34:31.046364+0000","last_fullsized":"2024-03-24T12:34:31.046364+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:33:27.355578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:27.355578+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:46.925782+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.049924+0000","last_change":"2024-03-24T12:31:29.996318+0000","last_active":"2024-03-24T12:34:31.049924+0000","last_peered":"2024-03-24T12:34:31.049924+0000","last_clean":"2024-03-24T12:34:31.049924+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:34:31.049924+0000","last_undegraded":"2024-03-24T12:34:31.049924+0000","last_fullsized":"2024-03-24T12:34:31.049924+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:31:29.996197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.033459+0000","last_change":"2024-03-24T12:33:26.230030+0000","last_active":"2024-03-24T12:34:31.033459+0000","last_peered":"2024-03-24T12:34:31.033459+0000","last_clean":"2024-03-24T12:34:31.033459+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:34:31.033459+0000","last_undegraded":"2024-03-24T12:34:31.033459+0000","last_fullsized":"2024-03-24T12:34:31.033459+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:26.229911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:26.229911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.062544+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":288,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030659+0000","last_change":"2024-03-24T12:33:28.059411+0000","last_active":"2024-03-24T12:34:31.030659+0000","last_peered":"2024-03-24T12:34:31.030659+0000","last_clean":"2024-03-24T12:34:31.030659+0000","last_became_active":"2024-03-24T12:26:03.165083+0000","last_became_peered":"2024-03-24T12:26:03.165083+0000","last_unstale":"2024-03-24T12:34:31.030659+0000","last_undegraded":"2024-03-24T12:34:31.030659+0000","last_fullsized":"2024-03-24T12:34:31.030659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:28.059288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.059288+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:53.397278+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":289,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.041314+0000","last_change":"2024-03-24T12:31:29.027987+0000","last_active":"2024-03-24T12:34:31.041314+0000","last_peered":"2024-03-24T12:34:31.041314+0000","last_clean":"2024-03-24T12:34:31.041314+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:34:31.041314+0000","last_undegraded":"2024-03-24T12:34:31.041314+0000","last_fullsized":"2024-03-24T12:34:31.041314+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:31:29.027885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:31:29.027885+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":260,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.079780+0000","last_change":"2024-03-24T12:32:53.875297+0000","last_active":"2024-03-24T12:34:22.079780+0000","last_peered":"2024-03-24T12:34:22.079780+0000","last_clean":"2024-03-24T12:34:22.079780+0000","last_became_active":"2024-03-24T12:26:09.350656+0000","last_became_peered":"2024-03-24T12:26:09.350656+0000","last_unstale":"2024-03-24T12:34:22.079780+0000","last_undegraded":"2024-03-24T12:34:22.079780+0000","last_fullsized":"2024-03-24T12:34:22.079780+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:53.875206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:32:53.875206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.046102+0000","last_change":"2024-03-24T12:33:31.383451+0000","last_active":"2024-03-24T12:34:31.046102+0000","last_peered":"2024-03-24T12:34:31.046102+0000","last_clean":"2024-03-24T12:34:31.046102+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:34:31.046102+0000","last_undegraded":"2024-03-24T12:34:31.046102+0000","last_fullsized":"2024-03-24T12:34:31.046102+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_clean_scrub_stamp":"2024-03-24T12:33:31.383330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:53.917455+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.034229+0000","last_change":"2024-03-24T12:33:12.283498+0000","last_active":"2024-03-24T12:34:31.034229+0000","last_peered":"2024-03-24T12:34:31.034229+0000","last_clean":"2024-03-24T12:34:31.034229+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:34:31.034229+0000","last_undegraded":"2024-03-24T12:34:31.034229+0000","last_fullsized":"2024-03-24T12:34:31.034229+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:12.283417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.283417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":264,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030290+0000","last_change":"2024-03-24T12:33:03.187495+0000","last_active":"2024-03-24T12:34:31.030290+0000","last_peered":"2024-03-24T12:34:31.030290+0000","last_clean":"2024-03-24T12:34:31.030290+0000","last_became_active":"2024-03-24T12:26:16.874170+0000","last_became_peered":"2024-03-24T12:26:16.874170+0000","last_unstale":"2024-03-24T12:34:31.030290+0000","last_undegraded":"2024-03-24T12:34:31.030290+0000","last_fullsized":"2024-03-24T12:34:31.030290+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:03.187410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.187410+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.231398+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.040966+0000","last_change":"2024-03-24T12:29:05.266028+0000","last_active":"2024-03-24T12:34:31.040966+0000","last_peered":"2024-03-24T12:34:31.040966+0000","last_clean":"2024-03-24T12:34:31.040966+0000","last_became_active":"2024-03-24T12:26:09.343296+0000","last_became_peered":"2024-03-24T12:26:09.343296+0000","last_unstale":"2024-03-24T12:34:31.040966+0000","last_undegraded":"2024-03-24T12:34:31.040966+0000","last_fullsized":"2024-03-24T12:34:31.040966+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:29:05.265893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:29:05.265893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.041400+0000","last_change":"2024-03-24T12:33:21.145032+0000","last_active":"2024-03-24T12:34:31.041400+0000","last_peered":"2024-03-24T12:34:31.041400+0000","last_clean":"2024-03-24T12:34:31.041400+0000","last_became_active":"2024-03-24T12:26:07.330811+0000","last_became_peered":"2024-03-24T12:26:07.330811+0000","last_unstale":"2024-03-24T12:34:31.041400+0000","last_undegraded":"2024-03-24T12:34:31.041400+0000","last_fullsized":"2024-03-24T12:34:31.041400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.144883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.144883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":290,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.026900+0000","last_change":"2024-03-24T12:33:17.214730+0000","last_active":"2024-03-24T12:34:31.026900+0000","last_peered":"2024-03-24T12:34:31.026900+0000","last_clean":"2024-03-24T12:34:31.026900+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:34:31.026900+0000","last_undegraded":"2024-03-24T12:34:31.026900+0000","last_fullsized":"2024-03-24T12:34:31.026900+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:17.214655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:17.214655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:43.429599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"88'16","reported_seq":310,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.026904+0000","last_change":"2024-03-24T12:32:50.139927+0000","last_active":"2024-03-24T12:34:31.026904+0000","last_peered":"2024-03-24T12:34:31.026904+0000","last_clean":"2024-03-24T12:34:31.026904+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:34:31.026904+0000","last_undegraded":"2024-03-24T12:34:31.026904+0000","last_fullsized":"2024-03-24T12:34:31.026904+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:32:50.139785+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:32:50.139785+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379348+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":261,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.092573+0000","last_change":"2024-03-24T12:33:08.045743+0000","last_active":"2024-03-24T12:34:22.092573+0000","last_peered":"2024-03-24T12:34:22.092573+0000","last_clean":"2024-03-24T12:34:22.092573+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:34:22.092573+0000","last_undegraded":"2024-03-24T12:34:22.092573+0000","last_fullsized":"2024-03-24T12:34:22.092573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:33:08.045492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.045492+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":376,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.031469+0000","last_change":"2024-03-24T12:32:57.178519+0000","last_active":"2024-03-24T12:34:31.031469+0000","last_peered":"2024-03-24T12:34:31.031469+0000","last_clean":"2024-03-24T12:34:31.031469+0000","last_became_active":"2024-03-24T12:26:07.329487+0000","last_became_peered":"2024-03-24T12:26:07.329487+0000","last_unstale":"2024-03-24T12:34:31.031469+0000","last_undegraded":"2024-03-24T12:34:31.031469+0000","last_fullsized":"2024-03-24T12:34:31.031469+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:32:57.177894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:57.177894+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.231458+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.033196+0000","last_change":"2024-03-24T12:33:06.290383+0000","last_active":"2024-03-24T12:34:31.033196+0000","last_peered":"2024-03-24T12:34:31.033196+0000","last_clean":"2024-03-24T12:34:31.033196+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:34:31.033196+0000","last_undegraded":"2024-03-24T12:34:31.033196+0000","last_fullsized":"2024-03-24T12:34:31.033196+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:06.290232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:06.290232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":285,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.049687+0000","last_change":"2024-03-24T12:29:15.283980+0000","last_active":"2024-03-24T12:34:31.049687+0000","last_peered":"2024-03-24T12:34:31.049687+0000","last_clean":"2024-03-24T12:34:31.049687+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:34:31.049687+0000","last_undegraded":"2024-03-24T12:34:31.049687+0000","last_fullsized":"2024-03-24T12:34:31.049687+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:29:15.283868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:29:15.283868+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.026385+0000","last_change":"2024-03-24T12:32:52.071488+0000","last_active":"2024-03-24T12:34:31.026385+0000","last_peered":"2024-03-24T12:34:31.026385+0000","last_clean":"2024-03-24T12:34:31.026385+0000","last_became_active":"2024-03-24T12:26:09.346062+0000","last_became_peered":"2024-03-24T12:26:09.346062+0000","last_unstale":"2024-03-24T12:34:31.026385+0000","last_undegraded":"2024-03-24T12:34:31.026385+0000","last_fullsized":"2024-03-24T12:34:31.026385+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:52.071390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:52.071390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379316+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"93'9","reported_seq":378,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.050006+0000","last_change":"2024-03-24T12:33:06.165605+0000","last_active":"2024-03-24T12:34:31.050006+0000","last_peered":"2024-03-24T12:34:31.050006+0000","last_clean":"2024-03-24T12:34:31.050006+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:34:31.050006+0000","last_undegraded":"2024-03-24T12:34:31.050006+0000","last_fullsized":"2024-03-24T12:34:31.050006+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'9","last_scrub_stamp":"2024-03-24T12:33:06.165466+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:33:06.165466+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":280,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.081494+0000","last_change":"2024-03-24T12:33:26.284194+0000","last_active":"2024-03-24T12:34:22.081494+0000","last_peered":"2024-03-24T12:34:22.081494+0000","last_clean":"2024-03-24T12:34:22.081494+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:34:22.081494+0000","last_undegraded":"2024-03-24T12:34:22.081494+0000","last_fullsized":"2024-03-24T12:34:22.081494+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:26.283885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:26.283885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:40.908213+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'13","reported_seq":298,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.031665+0000","last_change":"2024-03-24T12:33:09.042723+0000","last_active":"2024-03-24T12:34:31.031665+0000","last_peered":"2024-03-24T12:34:31.031665+0000","last_clean":"2024-03-24T12:34:31.031665+0000","last_became_active":"2024-03-24T12:26:05.358213+0000","last_became_peered":"2024-03-24T12:26:05.358213+0000","last_unstale":"2024-03-24T12:34:31.031665+0000","last_undegraded":"2024-03-24T12:34:31.031665+0000","last_fullsized":"2024-03-24T12:34:31.031665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:33:09.042629+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:33:09.042629+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.231427+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.028137+0000","last_change":"2024-03-24T12:33:00.629949+0000","last_active":"2024-03-24T12:34:31.028137+0000","last_peered":"2024-03-24T12:34:31.028137+0000","last_clean":"2024-03-24T12:34:31.028137+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:34:31.028137+0000","last_undegraded":"2024-03-24T12:34:31.028137+0000","last_fullsized":"2024-03-24T12:34:31.028137+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:00.629793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:33:00.629793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":266,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.050537+0000","last_change":"2024-03-24T12:33:18.355672+0000","last_active":"2024-03-24T12:34:31.050537+0000","last_peered":"2024-03-24T12:34:31.050537+0000","last_clean":"2024-03-24T12:34:31.050537+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:34:31.050537+0000","last_undegraded":"2024-03-24T12:34:31.050537+0000","last_fullsized":"2024-03-24T12:34:31.050537+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:18.355449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.355449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:44.546629+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":288,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.033318+0000","last_change":"2024-03-24T12:33:11.251158+0000","last_active":"2024-03-24T12:34:31.033318+0000","last_peered":"2024-03-24T12:34:31.033318+0000","last_clean":"2024-03-24T12:34:31.033318+0000","last_became_active":"2024-03-24T12:32:43.256833+0000","last_became_peered":"2024-03-24T12:32:43.256833+0000","last_unstale":"2024-03-24T12:34:31.033318+0000","last_undegraded":"2024-03-24T12:34:31.033318+0000","last_fullsized":"2024-03-24T12:34:31.033318+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:11.251080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:11.251080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.869025+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":292,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.092962+0000","last_change":"2024-03-24T12:33:18.443409+0000","last_active":"2024-03-24T12:34:22.092962+0000","last_peered":"2024-03-24T12:34:22.092962+0000","last_clean":"2024-03-24T12:34:22.092962+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:34:22.092962+0000","last_undegraded":"2024-03-24T12:34:22.092962+0000","last_fullsized":"2024-03-24T12:34:22.092962+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:18.443276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.443276+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:40.854569+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.034043+0000","last_change":"2024-03-24T12:33:03.328835+0000","last_active":"2024-03-24T12:34:31.034043+0000","last_peered":"2024-03-24T12:34:31.034043+0000","last_clean":"2024-03-24T12:34:31.034043+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:34:31.034043+0000","last_undegraded":"2024-03-24T12:34:31.034043+0000","last_fullsized":"2024-03-24T12:34:31.034043+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:03.328728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.328728+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":266,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030767+0000","last_change":"2024-03-24T12:32:59.274958+0000","last_active":"2024-03-24T12:34:31.030767+0000","last_peered":"2024-03-24T12:34:31.030767+0000","last_clean":"2024-03-24T12:34:31.030767+0000","last_became_active":"2024-03-24T12:26:07.329117+0000","last_became_peered":"2024-03-24T12:26:07.329117+0000","last_unstale":"2024-03-24T12:34:31.030767+0000","last_undegraded":"2024-03-24T12:34:31.030767+0000","last_fullsized":"2024-03-24T12:34:31.030767+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:59.274820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:59.274820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.231488+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.036338+0000","last_change":"2024-03-24T12:32:52.926826+0000","last_active":"2024-03-24T12:34:31.036338+0000","last_peered":"2024-03-24T12:34:31.036338+0000","last_clean":"2024-03-24T12:34:31.036338+0000","last_became_active":"2024-03-24T12:26:03.161414+0000","last_became_peered":"2024-03-24T12:26:03.161414+0000","last_unstale":"2024-03-24T12:34:31.036338+0000","last_undegraded":"2024-03-24T12:34:31.036338+0000","last_fullsized":"2024-03-24T12:34:31.036338+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:52.926693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:52.926693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.277705+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.050697+0000","last_change":"2024-03-24T12:32:58.025762+0000","last_active":"2024-03-24T12:34:31.050697+0000","last_peered":"2024-03-24T12:34:31.050697+0000","last_clean":"2024-03-24T12:34:31.050697+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:34:31.050697+0000","last_undegraded":"2024-03-24T12:34:31.050697+0000","last_fullsized":"2024-03-24T12:34:31.050697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T12:32:58.023697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:58.023697+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"189'7661","reported_seq":7733,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:32.465908+0000","last_change":"2024-03-24T12:32:41.565569+0000","last_active":"2024-03-24T12:34:32.465908+0000","last_peered":"2024-03-24T12:34:32.465908+0000","last_clean":"2024-03-24T12:34:32.465908+0000","last_became_active":"2024-03-24T12:32:41.565218+0000","last_became_peered":"2024-03-24T12:32:41.565218+0000","last_unstale":"2024-03-24T12:34:32.465908+0000","last_undegraded":"2024-03-24T12:34:32.465908+0000","last_fullsized":"2024-03-24T12:34:32.465908+0000","mapping_epoch":158,"log_start":"170'2300","ondisk_log_start":"170'2300","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":5361,"log_dups_size":0,"ondisk_log_size":5361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446234624,"num_objects":6809,"num_object_clones":0,"num_object_copies":20427,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6809,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7661,"num_write_kb":463040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030747+0000","last_change":"2024-03-24T12:33:34.170601+0000","last_active":"2024-03-24T12:34:31.030747+0000","last_peered":"2024-03-24T12:34:31.030747+0000","last_clean":"2024-03-24T12:34:31.030747+0000","last_became_active":"2024-03-24T12:26:09.643602+0000","last_became_peered":"2024-03-24T12:26:09.643602+0000","last_unstale":"2024-03-24T12:34:31.030747+0000","last_undegraded":"2024-03-24T12:34:31.030747+0000","last_fullsized":"2024-03-24T12:34:31.030747+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.170473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.170473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:53.672932+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":286,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.033942+0000","last_change":"2024-03-24T12:29:13.102973+0000","last_active":"2024-03-24T12:34:31.033942+0000","last_peered":"2024-03-24T12:34:31.033942+0000","last_clean":"2024-03-24T12:34:31.033942+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:34:31.033942+0000","last_undegraded":"2024-03-24T12:34:31.033942+0000","last_fullsized":"2024-03-24T12:34:31.033942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:29:13.102846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:29:13.102846+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.868904+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":303,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.031758+0000","last_change":"2024-03-24T12:33:26.095777+0000","last_active":"2024-03-24T12:34:31.031758+0000","last_peered":"2024-03-24T12:34:31.031758+0000","last_clean":"2024-03-24T12:34:31.031758+0000","last_became_active":"2024-03-24T12:26:05.358217+0000","last_became_peered":"2024-03-24T12:26:05.358217+0000","last_unstale":"2024-03-24T12:34:31.031758+0000","last_undegraded":"2024-03-24T12:34:31.031758+0000","last_fullsized":"2024-03-24T12:34:31.031758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:33:26.094785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:26.094785+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:51.228338+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":387,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.033520+0000","last_change":"2024-03-24T12:33:22.181670+0000","last_active":"2024-03-24T12:34:31.033520+0000","last_peered":"2024-03-24T12:34:31.033520+0000","last_clean":"2024-03-24T12:34:31.033520+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:34:31.033520+0000","last_undegraded":"2024-03-24T12:34:31.033520+0000","last_fullsized":"2024-03-24T12:34:31.033520+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:33:22.181560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:22.181560+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"189'7392","reported_seq":7464,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:32.462609+0000","last_change":"2024-03-24T12:32:41.045588+0000","last_active":"2024-03-24T12:34:32.462609+0000","last_peered":"2024-03-24T12:34:32.462609+0000","last_clean":"2024-03-24T12:34:32.462609+0000","last_became_active":"2024-03-24T12:32:41.045244+0000","last_became_peered":"2024-03-24T12:32:41.045244+0000","last_unstale":"2024-03-24T12:34:32.462609+0000","last_undegraded":"2024-03-24T12:34:32.462609+0000","last_fullsized":"2024-03-24T12:34:32.462609+0000","mapping_epoch":158,"log_start":"167'1000","ondisk_log_start":"167'1000","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":6392,"log_dups_size":0,"ondisk_log_size":6392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.277614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":433324032,"num_objects":6612,"num_object_clones":0,"num_object_copies":19836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7392,"num_write_kb":448128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.049812+0000","last_change":"2024-03-24T12:33:20.675885+0000","last_active":"2024-03-24T12:34:31.049812+0000","last_peered":"2024-03-24T12:34:31.049812+0000","last_clean":"2024-03-24T12:34:31.049812+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:34:31.049812+0000","last_undegraded":"2024-03-24T12:34:31.049812+0000","last_fullsized":"2024-03-24T12:34:31.049812+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:20.675649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:20.675649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:34.909534+0000","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.034257+0000","last_change":"2024-03-24T12:32:59.313688+0000","last_active":"2024-03-24T12:34:31.034257+0000","last_peered":"2024-03-24T12:34:31.034257+0000","last_clean":"2024-03-24T12:34:31.034257+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:34:31.034257+0000","last_undegraded":"2024-03-24T12:34:31.034257+0000","last_fullsized":"2024-03-24T12:34:31.034257+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:59.313589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:59.313589+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"140'17","reported_seq":314,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.025967+0000","last_change":"2024-03-24T12:33:30.333141+0000","last_active":"2024-03-24T12:34:31.025967+0000","last_peered":"2024-03-24T12:34:31.025967+0000","last_clean":"2024-03-24T12:34:31.025967+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:34:31.025967+0000","last_undegraded":"2024-03-24T12:34:31.025967+0000","last_fullsized":"2024-03-24T12:34:31.025967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'17","last_scrub_stamp":"2024-03-24T12:33:30.332974+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:33:30.332974+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:42.623286+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.027807+0000","last_change":"2024-03-24T12:33:13.794657+0000","last_active":"2024-03-24T12:34:31.027807+0000","last_peered":"2024-03-24T12:34:31.027807+0000","last_clean":"2024-03-24T12:34:31.027807+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:34:31.027807+0000","last_undegraded":"2024-03-24T12:34:31.027807+0000","last_fullsized":"2024-03-24T12:34:31.027807+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:13.794520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:13.794520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":120,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"188'7430","reported_seq":7499,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:30.805657+0000","last_change":"2024-03-24T12:32:41.041630+0000","last_active":"2024-03-24T12:34:30.805657+0000","last_peered":"2024-03-24T12:34:30.805657+0000","last_clean":"2024-03-24T12:34:30.805657+0000","last_became_active":"2024-03-24T12:32:41.041303+0000","last_became_peered":"2024-03-24T12:32:41.041303+0000","last_unstale":"2024-03-24T12:34:30.805657+0000","last_undegraded":"2024-03-24T12:34:30.805657+0000","last_fullsized":"2024-03-24T12:34:30.805657+0000","mapping_epoch":158,"log_start":"169'1600","ondisk_log_start":"169'1600","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":5830,"log_dups_size":0,"ondisk_log_size":5830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447873024,"num_objects":6834,"num_object_clones":0,"num_object_copies":20502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7430,"num_write_kb":456448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.033121+0000","last_change":"2024-03-24T12:33:07.283118+0000","last_active":"2024-03-24T12:34:31.033121+0000","last_peered":"2024-03-24T12:34:31.033121+0000","last_clean":"2024-03-24T12:34:31.033121+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:34:31.033121+0000","last_undegraded":"2024-03-24T12:34:31.033121+0000","last_fullsized":"2024-03-24T12:34:31.033121+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:07.282989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.282989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.868968+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.027350+0000","last_change":"2024-03-24T12:33:06.631005+0000","last_active":"2024-03-24T12:34:31.027350+0000","last_peered":"2024-03-24T12:34:31.027350+0000","last_clean":"2024-03-24T12:34:31.027350+0000","last_became_active":"2024-03-24T12:26:07.331416+0000","last_became_peered":"2024-03-24T12:26:07.331416+0000","last_unstale":"2024-03-24T12:34:31.027350+0000","last_undegraded":"2024-03-24T12:34:31.027350+0000","last_fullsized":"2024-03-24T12:34:31.027350+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:06.630869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:06.630869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.817810+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.034405+0000","last_change":"2024-03-24T12:33:34.204979+0000","last_active":"2024-03-24T12:34:31.034405+0000","last_peered":"2024-03-24T12:34:31.034405+0000","last_clean":"2024-03-24T12:34:31.034405+0000","last_became_active":"2024-03-24T12:26:03.165214+0000","last_became_peered":"2024-03-24T12:26:03.165214+0000","last_unstale":"2024-03-24T12:34:31.034405+0000","last_undegraded":"2024-03-24T12:34:31.034405+0000","last_fullsized":"2024-03-24T12:34:31.034405+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.204829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:54.365709+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":309,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.036941+0000","last_change":"2024-03-24T12:33:01.910428+0000","last_active":"2024-03-24T12:34:31.036941+0000","last_peered":"2024-03-24T12:34:31.036941+0000","last_clean":"2024-03-24T12:34:31.036941+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:34:31.036941+0000","last_undegraded":"2024-03-24T12:34:31.036941+0000","last_fullsized":"2024-03-24T12:34:31.036941+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:33:01.910310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:01.910310+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.277790+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"189'7625","reported_seq":7697,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:33.125666+0000","last_change":"2024-03-24T12:32:41.042352+0000","last_active":"2024-03-24T12:34:33.125666+0000","last_peered":"2024-03-24T12:34:33.125666+0000","last_clean":"2024-03-24T12:34:33.125666+0000","last_became_active":"2024-03-24T12:32:41.042042+0000","last_became_peered":"2024-03-24T12:32:41.042042+0000","last_unstale":"2024-03-24T12:34:33.125666+0000","last_undegraded":"2024-03-24T12:34:33.125666+0000","last_fullsized":"2024-03-24T12:34:33.125666+0000","mapping_epoch":158,"log_start":"172'2700","ondisk_log_start":"172'2700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440205312,"num_objects":6717,"num_object_clones":0,"num_object_copies":20151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7625,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":260,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.092802+0000","last_change":"2024-03-24T12:33:13.176741+0000","last_active":"2024-03-24T12:34:22.092802+0000","last_peered":"2024-03-24T12:34:22.092802+0000","last_clean":"2024-03-24T12:34:22.092802+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:34:22.092802+0000","last_undegraded":"2024-03-24T12:34:22.092802+0000","last_fullsized":"2024-03-24T12:34:22.092802+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:13.176371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:13.176371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.337454+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.027319+0000","last_change":"2024-03-24T12:33:16.756497+0000","last_active":"2024-03-24T12:34:31.027319+0000","last_peered":"2024-03-24T12:34:31.027319+0000","last_clean":"2024-03-24T12:34:31.027319+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:34:31.027319+0000","last_undegraded":"2024-03-24T12:34:31.027319+0000","last_fullsized":"2024-03-24T12:34:31.027319+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:16.756102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:33:16.756102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":289,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.031147+0000","last_change":"2024-03-24T12:33:08.075378+0000","last_active":"2024-03-24T12:34:31.031147+0000","last_peered":"2024-03-24T12:34:31.031147+0000","last_clean":"2024-03-24T12:34:31.031147+0000","last_became_active":"2024-03-24T12:26:03.162562+0000","last_became_peered":"2024-03-24T12:26:03.162562+0000","last_unstale":"2024-03-24T12:34:31.031147+0000","last_undegraded":"2024-03-24T12:34:31.031147+0000","last_fullsized":"2024-03-24T12:34:31.031147+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:33:08.075093+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.075093+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":300,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.080850+0000","last_change":"2024-03-24T12:33:17.435894+0000","last_active":"2024-03-24T12:34:22.080850+0000","last_peered":"2024-03-24T12:34:22.080850+0000","last_clean":"2024-03-24T12:34:22.080850+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:34:22.080850+0000","last_undegraded":"2024-03-24T12:34:22.080850+0000","last_fullsized":"2024-03-24T12:34:22.080850+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:17.435756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:17.435756+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.934217+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"189'7469","reported_seq":7541,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:32.479741+0000","last_change":"2024-03-24T12:32:41.044116+0000","last_active":"2024-03-24T12:34:32.479741+0000","last_peered":"2024-03-24T12:34:32.479741+0000","last_clean":"2024-03-24T12:34:32.479741+0000","last_became_active":"2024-03-24T12:32:41.043709+0000","last_became_peered":"2024-03-24T12:32:41.043709+0000","last_unstale":"2024-03-24T12:34:32.479741+0000","last_undegraded":"2024-03-24T12:34:32.479741+0000","last_fullsized":"2024-03-24T12:34:32.479741+0000","mapping_epoch":158,"log_start":"170'2200","ondisk_log_start":"170'2200","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":5269,"log_dups_size":0,"ondisk_log_size":5269,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436797440,"num_objects":6665,"num_object_clones":0,"num_object_copies":19995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7469,"num_write_kb":452288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.049874+0000","last_change":"2024-03-24T12:32:57.050467+0000","last_active":"2024-03-24T12:34:31.049874+0000","last_peered":"2024-03-24T12:34:31.049874+0000","last_clean":"2024-03-24T12:34:31.049874+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:34:31.049874+0000","last_undegraded":"2024-03-24T12:34:31.049874+0000","last_fullsized":"2024-03-24T12:34:31.049874+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:57.050342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:32:57.050342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":284,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030502+0000","last_change":"2024-03-24T12:33:06.156977+0000","last_active":"2024-03-24T12:34:31.030502+0000","last_peered":"2024-03-24T12:34:31.030502+0000","last_clean":"2024-03-24T12:34:31.030502+0000","last_became_active":"2024-03-24T12:26:03.165771+0000","last_became_peered":"2024-03-24T12:26:03.165771+0000","last_unstale":"2024-03-24T12:34:31.030502+0000","last_undegraded":"2024-03-24T12:34:31.030502+0000","last_fullsized":"2024-03-24T12:34:31.030502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:33:06.156811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:06.156811+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.231344+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":283,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.026703+0000","last_change":"2024-03-24T12:32:54.061412+0000","last_active":"2024-03-24T12:34:31.026703+0000","last_peered":"2024-03-24T12:34:31.026703+0000","last_clean":"2024-03-24T12:34:31.026703+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:34:31.026703+0000","last_undegraded":"2024-03-24T12:34:31.026703+0000","last_fullsized":"2024-03-24T12:34:31.026703+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:54.061297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:54.061297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379283+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"90'38","reported_seq":439,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.039006+0000","last_change":"2024-03-24T12:32:55.403983+0000","last_active":"2024-03-24T12:34:31.039006+0000","last_peered":"2024-03-24T12:34:31.039006+0000","last_clean":"2024-03-24T12:34:31.039006+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:34:31.039006+0000","last_undegraded":"2024-03-24T12:34:31.039006+0000","last_fullsized":"2024-03-24T12:34:31.039006+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'38","last_scrub_stamp":"2024-03-24T12:32:55.403873+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.403873+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":264,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.081150+0000","last_change":"2024-03-24T12:32:52.817749+0000","last_active":"2024-03-24T12:34:22.081150+0000","last_peered":"2024-03-24T12:34:22.081150+0000","last_clean":"2024-03-24T12:34:22.081150+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:34:22.081150+0000","last_undegraded":"2024-03-24T12:34:22.081150+0000","last_fullsized":"2024-03-24T12:34:22.081150+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:52.817660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:52.817660+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:22.086721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"189'7560","reported_seq":7632,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:34.051191+0000","last_change":"2024-03-24T12:32:41.045390+0000","last_active":"2024-03-24T12:34:34.051191+0000","last_peered":"2024-03-24T12:34:34.051191+0000","last_clean":"2024-03-24T12:34:34.051191+0000","last_became_active":"2024-03-24T12:32:41.045113+0000","last_became_peered":"2024-03-24T12:32:41.045113+0000","last_unstale":"2024-03-24T12:34:34.051191+0000","last_undegraded":"2024-03-24T12:34:34.051191+0000","last_fullsized":"2024-03-24T12:34:34.051191+0000","mapping_epoch":158,"log_start":"173'2700","ondisk_log_start":"173'2700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":4860,"log_dups_size":0,"ondisk_log_size":4860,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437125120,"num_objects":6670,"num_object_clones":0,"num_object_copies":20010,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6670,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7560,"num_write_kb":455360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.034509+0000","last_change":"2024-03-24T12:33:35.181634+0000","last_active":"2024-03-24T12:34:31.034509+0000","last_peered":"2024-03-24T12:34:31.034509+0000","last_clean":"2024-03-24T12:34:31.034509+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:34:31.034509+0000","last_undegraded":"2024-03-24T12:34:31.034509+0000","last_fullsized":"2024-03-24T12:34:31.034509+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:35.181345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:35.181345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:48.071612+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":291,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.039865+0000","last_change":"2024-03-24T12:33:19.170350+0000","last_active":"2024-03-24T12:34:31.039865+0000","last_peered":"2024-03-24T12:34:31.039865+0000","last_clean":"2024-03-24T12:34:31.039865+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:34:31.039865+0000","last_undegraded":"2024-03-24T12:34:31.039865+0000","last_fullsized":"2024-03-24T12:34:31.039865+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:19.170147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.170147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:40.775761+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":290,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.027490+0000","last_change":"2024-03-24T12:32:58.630218+0000","last_active":"2024-03-24T12:34:31.027490+0000","last_peered":"2024-03-24T12:34:31.027490+0000","last_clean":"2024-03-24T12:34:31.027490+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:34:31.027490+0000","last_undegraded":"2024-03-24T12:34:31.027490+0000","last_fullsized":"2024-03-24T12:34:31.027490+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:58.630094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:58.630094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":318,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.039464+0000","last_change":"2024-03-24T12:33:02.509385+0000","last_active":"2024-03-24T12:34:31.039464+0000","last_peered":"2024-03-24T12:34:31.039464+0000","last_clean":"2024-03-24T12:34:31.039464+0000","last_became_active":"2024-03-24T12:26:05.315639+0000","last_became_peered":"2024-03-24T12:26:05.315639+0000","last_unstale":"2024-03-24T12:34:31.039464+0000","last_undegraded":"2024-03-24T12:34:31.039464+0000","last_fullsized":"2024-03-24T12:34:31.039464+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:33:02.509218+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:31:08.237387+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.509218+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.027930+0000","last_change":"2024-03-24T12:33:09.679795+0000","last_active":"2024-03-24T12:34:31.027930+0000","last_peered":"2024-03-24T12:34:31.027930+0000","last_clean":"2024-03-24T12:34:31.027930+0000","last_became_active":"2024-03-24T12:26:07.343010+0000","last_became_peered":"2024-03-24T12:26:07.343010+0000","last_unstale":"2024-03-24T12:34:31.027930+0000","last_undegraded":"2024-03-24T12:34:31.027930+0000","last_fullsized":"2024-03-24T12:34:31.027930+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:09.679721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:09.679721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.507477+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"189'7520","reported_seq":7592,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:32.489220+0000","last_change":"2024-03-24T12:32:41.042784+0000","last_active":"2024-03-24T12:34:32.489220+0000","last_peered":"2024-03-24T12:34:32.489220+0000","last_clean":"2024-03-24T12:34:32.489220+0000","last_became_active":"2024-03-24T12:32:41.042370+0000","last_became_peered":"2024-03-24T12:32:41.042370+0000","last_unstale":"2024-03-24T12:34:32.489220+0000","last_undegraded":"2024-03-24T12:34:32.489220+0000","last_fullsized":"2024-03-24T12:34:32.489220+0000","mapping_epoch":158,"log_start":"167'1100","ondisk_log_start":"167'1100","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":6420,"log_dups_size":0,"ondisk_log_size":6420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.276382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436469760,"num_objects":6660,"num_object_clones":0,"num_object_copies":19980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7520,"num_write_kb":453760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.032227+0000","last_change":"2024-03-24T12:33:18.134199+0000","last_active":"2024-03-24T12:34:31.032227+0000","last_peered":"2024-03-24T12:34:31.032227+0000","last_clean":"2024-03-24T12:34:31.032227+0000","last_became_active":"2024-03-24T12:26:09.347821+0000","last_became_peered":"2024-03-24T12:26:09.347821+0000","last_unstale":"2024-03-24T12:34:31.032227+0000","last_undegraded":"2024-03-24T12:34:31.032227+0000","last_fullsized":"2024-03-24T12:34:31.032227+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:18.134111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.134111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:39.370943+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":536,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.033304+0000","last_change":"2024-03-24T12:33:14.339596+0000","last_active":"2024-03-24T12:34:31.033304+0000","last_peered":"2024-03-24T12:34:31.033304+0000","last_clean":"2024-03-24T12:34:31.033304+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:34:31.033304+0000","last_undegraded":"2024-03-24T12:34:31.033304+0000","last_fullsized":"2024-03-24T12:34:31.033304+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:33:14.339456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:33:14.339456+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:40.390099+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":283,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.041633+0000","last_change":"2024-03-24T12:33:24.345396+0000","last_active":"2024-03-24T12:34:31.041633+0000","last_peered":"2024-03-24T12:34:31.041633+0000","last_clean":"2024-03-24T12:34:31.041633+0000","last_became_active":"2024-03-24T12:26:03.166467+0000","last_became_peered":"2024-03-24T12:26:03.166467+0000","last_unstale":"2024-03-24T12:34:31.041633+0000","last_undegraded":"2024-03-24T12:34:31.041633+0000","last_fullsized":"2024-03-24T12:34:31.041633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:24.345279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:24.345279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:46.053687+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":403,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.037480+0000","last_change":"2024-03-24T12:32:59.916431+0000","last_active":"2024-03-24T12:34:31.037480+0000","last_peered":"2024-03-24T12:34:31.037480+0000","last_clean":"2024-03-24T12:34:31.037480+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:34:31.037480+0000","last_undegraded":"2024-03-24T12:34:31.037480+0000","last_fullsized":"2024-03-24T12:34:31.037480+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:32:59.916303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:59.916303+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.277826+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":267,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.034262+0000","last_change":"2024-03-24T12:33:08.288494+0000","last_active":"2024-03-24T12:34:31.034262+0000","last_peered":"2024-03-24T12:34:31.034262+0000","last_clean":"2024-03-24T12:34:31.034262+0000","last_became_active":"2024-03-24T12:26:07.337548+0000","last_became_peered":"2024-03-24T12:26:07.337548+0000","last_unstale":"2024-03-24T12:34:31.034262+0000","last_undegraded":"2024-03-24T12:34:31.034262+0000","last_fullsized":"2024-03-24T12:34:31.034262+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:08.288377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.288377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"189'7385","reported_seq":7457,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:33.194583+0000","last_change":"2024-03-24T12:32:41.565670+0000","last_active":"2024-03-24T12:34:33.194583+0000","last_peered":"2024-03-24T12:34:33.194583+0000","last_clean":"2024-03-24T12:34:33.194583+0000","last_became_active":"2024-03-24T12:32:41.565379+0000","last_became_peered":"2024-03-24T12:32:41.565379+0000","last_unstale":"2024-03-24T12:34:33.194583+0000","last_undegraded":"2024-03-24T12:34:33.194583+0000","last_fullsized":"2024-03-24T12:34:33.194583+0000","mapping_epoch":158,"log_start":"171'2400","ondisk_log_start":"171'2400","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":4985,"log_dups_size":0,"ondisk_log_size":4985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430374912,"num_objects":6567,"num_object_clones":0,"num_object_copies":19701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7385,"num_write_kb":446464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":279,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.039257+0000","last_change":"2024-03-24T12:33:09.517074+0000","last_active":"2024-03-24T12:34:31.039257+0000","last_peered":"2024-03-24T12:34:31.039257+0000","last_clean":"2024-03-24T12:34:31.039257+0000","last_became_active":"2024-03-24T12:26:09.349435+0000","last_became_peered":"2024-03-24T12:26:09.349435+0000","last_unstale":"2024-03-24T12:34:31.039257+0000","last_undegraded":"2024-03-24T12:34:31.039257+0000","last_fullsized":"2024-03-24T12:34:31.039257+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:09.516956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:09.516956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":281,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.092427+0000","last_change":"2024-03-24T12:33:34.251764+0000","last_active":"2024-03-24T12:34:22.092427+0000","last_peered":"2024-03-24T12:34:22.092427+0000","last_clean":"2024-03-24T12:34:22.092427+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:34:22.092427+0000","last_undegraded":"2024-03-24T12:34:22.092427+0000","last_fullsized":"2024-03-24T12:34:22.092427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.251636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.251636+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.251636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:58.299034+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":795,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.035248+0000","last_change":"2024-03-24T12:33:32.221302+0000","last_active":"2024-03-24T12:34:31.035248+0000","last_peered":"2024-03-24T12:34:31.035248+0000","last_clean":"2024-03-24T12:34:31.035248+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:34:31.035248+0000","last_undegraded":"2024-03-24T12:34:31.035248+0000","last_fullsized":"2024-03-24T12:34:31.035248+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:33:32.221181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:32.221181+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:32.456473+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":502,"num_read_kb":497,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":286,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.092674+0000","last_change":"2024-03-24T12:32:50.752283+0000","last_active":"2024-03-24T12:34:22.092674+0000","last_peered":"2024-03-24T12:34:22.092674+0000","last_clean":"2024-03-24T12:34:22.092674+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:34:22.092674+0000","last_undegraded":"2024-03-24T12:34:22.092674+0000","last_fullsized":"2024-03-24T12:34:22.092674+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:32:50.752146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:50.752146+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.032534+0000","last_change":"2024-03-24T12:33:10.102515+0000","last_active":"2024-03-24T12:34:31.032534+0000","last_peered":"2024-03-24T12:34:31.032534+0000","last_clean":"2024-03-24T12:34:31.032534+0000","last_became_active":"2024-03-24T12:26:07.326457+0000","last_became_peered":"2024-03-24T12:26:07.326457+0000","last_unstale":"2024-03-24T12:34:31.032534+0000","last_undegraded":"2024-03-24T12:34:31.032534+0000","last_fullsized":"2024-03-24T12:34:31.032534+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:10.102394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:10.102394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.231314+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"189'7644","reported_seq":7716,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:34.064232+0000","last_change":"2024-03-24T12:32:41.563958+0000","last_active":"2024-03-24T12:34:34.064232+0000","last_peered":"2024-03-24T12:34:34.064232+0000","last_clean":"2024-03-24T12:34:34.064232+0000","last_became_active":"2024-03-24T12:32:41.563612+0000","last_became_peered":"2024-03-24T12:32:41.563612+0000","last_unstale":"2024-03-24T12:34:34.064232+0000","last_undegraded":"2024-03-24T12:34:34.064232+0000","last_fullsized":"2024-03-24T12:34:34.064232+0000","mapping_epoch":158,"log_start":"172'2800","ondisk_log_start":"172'2800","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":4844,"log_dups_size":0,"ondisk_log_size":4844,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438042624,"num_objects":6684,"num_object_clones":0,"num_object_copies":20052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7644,"num_write_kb":458496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.040804+0000","last_change":"2024-03-24T12:32:57.476369+0000","last_active":"2024-03-24T12:34:31.040804+0000","last_peered":"2024-03-24T12:34:31.040804+0000","last_clean":"2024-03-24T12:34:31.040804+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:34:31.040804+0000","last_undegraded":"2024-03-24T12:34:31.040804+0000","last_fullsized":"2024-03-24T12:34:31.040804+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:57.476212+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:57.476212+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":283,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.038889+0000","last_change":"2024-03-24T12:32:53.384154+0000","last_active":"2024-03-24T12:34:31.038889+0000","last_peered":"2024-03-24T12:34:31.038889+0000","last_clean":"2024-03-24T12:34:31.038889+0000","last_became_active":"2024-03-24T12:26:03.167244+0000","last_became_peered":"2024-03-24T12:26:03.167244+0000","last_unstale":"2024-03-24T12:34:31.038889+0000","last_undegraded":"2024-03-24T12:34:31.038889+0000","last_fullsized":"2024-03-24T12:34:31.038889+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:53.384075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:53.384075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":305,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.080996+0000","last_change":"2024-03-24T12:33:19.405563+0000","last_active":"2024-03-24T12:34:22.080996+0000","last_peered":"2024-03-24T12:34:22.080996+0000","last_clean":"2024-03-24T12:34:22.080996+0000","last_became_active":"2024-03-24T12:26:05.356745+0000","last_became_peered":"2024-03-24T12:26:05.356745+0000","last_unstale":"2024-03-24T12:34:22.080996+0000","last_undegraded":"2024-03-24T12:34:22.080996+0000","last_fullsized":"2024-03-24T12:34:22.080996+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:33:19.405401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.405401+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.845660+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":275,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.037034+0000","last_change":"2024-03-24T12:32:50.951383+0000","last_active":"2024-03-24T12:34:31.037034+0000","last_peered":"2024-03-24T12:34:31.037034+0000","last_clean":"2024-03-24T12:34:31.037034+0000","last_became_active":"2024-03-24T12:26:07.337303+0000","last_became_peered":"2024-03-24T12:26:07.337303+0000","last_unstale":"2024-03-24T12:34:31.037034+0000","last_undegraded":"2024-03-24T12:34:31.037034+0000","last_fullsized":"2024-03-24T12:34:31.037034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:50.951269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:32:50.951269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.277678+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"188'7571","reported_seq":7641,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:30.752978+0000","last_change":"2024-03-24T12:32:41.565895+0000","last_active":"2024-03-24T12:34:30.752978+0000","last_peered":"2024-03-24T12:34:30.752978+0000","last_clean":"2024-03-24T12:34:30.752978+0000","last_became_active":"2024-03-24T12:32:41.565563+0000","last_became_peered":"2024-03-24T12:32:41.565563+0000","last_unstale":"2024-03-24T12:34:30.752978+0000","last_undegraded":"2024-03-24T12:34:30.752978+0000","last_fullsized":"2024-03-24T12:34:30.752978+0000","mapping_epoch":158,"log_start":"169'1800","ondisk_log_start":"169'1800","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456065024,"num_objects":6959,"num_object_clones":0,"num_object_copies":20877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6959,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7571,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":261,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.079886+0000","last_change":"2024-03-24T12:33:12.332284+0000","last_active":"2024-03-24T12:34:22.079886+0000","last_peered":"2024-03-24T12:34:22.079886+0000","last_clean":"2024-03-24T12:34:22.079886+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:34:22.079886+0000","last_undegraded":"2024-03-24T12:34:22.079886+0000","last_fullsized":"2024-03-24T12:34:22.079886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:12.332162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.332162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:34.976750+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":283,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.031581+0000","last_change":"2024-03-24T12:33:11.083121+0000","last_active":"2024-03-24T12:34:31.031581+0000","last_peered":"2024-03-24T12:34:31.031581+0000","last_clean":"2024-03-24T12:34:31.031581+0000","last_became_active":"2024-03-24T12:26:03.168184+0000","last_became_peered":"2024-03-24T12:26:03.168184+0000","last_unstale":"2024-03-24T12:34:31.031581+0000","last_undegraded":"2024-03-24T12:34:31.031581+0000","last_fullsized":"2024-03-24T12:34:31.031581+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:11.083006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:11.083006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.231370+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"138'15","reported_seq":307,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.041486+0000","last_change":"2024-03-24T12:33:07.518493+0000","last_active":"2024-03-24T12:34:31.041486+0000","last_peered":"2024-03-24T12:34:31.041486+0000","last_clean":"2024-03-24T12:34:31.041486+0000","last_became_active":"2024-03-24T12:26:05.316739+0000","last_became_peered":"2024-03-24T12:26:05.316739+0000","last_unstale":"2024-03-24T12:34:31.041486+0000","last_undegraded":"2024-03-24T12:34:31.041486+0000","last_fullsized":"2024-03-24T12:34:31.041486+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:33:07.518368+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.518368+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.802570+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":374,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:23.309820+0000","last_change":"2024-03-24T12:32:54.818766+0000","last_active":"2024-03-24T12:34:23.309820+0000","last_peered":"2024-03-24T12:34:23.309820+0000","last_clean":"2024-03-24T12:34:23.309820+0000","last_became_active":"2024-03-24T12:26:07.336080+0000","last_became_peered":"2024-03-24T12:26:07.336080+0000","last_unstale":"2024-03-24T12:34:23.309820+0000","last_undegraded":"2024-03-24T12:34:23.309820+0000","last_fullsized":"2024-03-24T12:34:23.309820+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:32:54.818624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:54.818624+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"189'7448","reported_seq":7521,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:32.466675+0000","last_change":"2024-03-24T12:32:41.564142+0000","last_active":"2024-03-24T12:34:32.466675+0000","last_peered":"2024-03-24T12:34:32.466675+0000","last_clean":"2024-03-24T12:34:32.466675+0000","last_became_active":"2024-03-24T12:32:41.563736+0000","last_became_peered":"2024-03-24T12:32:41.563736+0000","last_unstale":"2024-03-24T12:34:32.466675+0000","last_undegraded":"2024-03-24T12:34:32.466675+0000","last_fullsized":"2024-03-24T12:34:32.466675+0000","mapping_epoch":158,"log_start":"170'2100","ondisk_log_start":"170'2100","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":5348,"log_dups_size":0,"ondisk_log_size":5348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":438763544,"num_objects":6696,"num_object_clones":0,"num_object_copies":20088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6696,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7448,"num_write_kb":452545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.034109+0000","last_change":"2024-03-24T12:33:38.172365+0000","last_active":"2024-03-24T12:34:31.034109+0000","last_peered":"2024-03-24T12:34:31.034109+0000","last_clean":"2024-03-24T12:34:31.034109+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:34:31.034109+0000","last_undegraded":"2024-03-24T12:34:31.034109+0000","last_fullsized":"2024-03-24T12:34:31.034109+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:38.172239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:38.172239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:58.289331+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":281,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:22.092765+0000","last_change":"2024-03-24T12:33:29.273583+0000","last_active":"2024-03-24T12:34:22.092765+0000","last_peered":"2024-03-24T12:34:22.092765+0000","last_clean":"2024-03-24T12:34:22.092765+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:34:22.092765+0000","last_undegraded":"2024-03-24T12:34:22.092765+0000","last_fullsized":"2024-03-24T12:34:22.092765+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:29.273325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:29.273325+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:49.061356+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"93'21","reported_seq":307,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.025786+0000","last_change":"2024-03-24T12:33:20.268750+0000","last_active":"2024-03-24T12:34:31.025786+0000","last_peered":"2024-03-24T12:34:31.025786+0000","last_clean":"2024-03-24T12:34:31.025786+0000","last_became_active":"2024-03-24T12:26:05.315454+0000","last_became_peered":"2024-03-24T12:26:05.315454+0000","last_unstale":"2024-03-24T12:34:31.025786+0000","last_undegraded":"2024-03-24T12:34:31.025786+0000","last_fullsized":"2024-03-24T12:34:31.025786+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'21","last_scrub_stamp":"2024-03-24T12:33:20.268587+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:33:20.268587+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379395+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":386,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.046693+0000","last_change":"2024-03-24T12:33:35.289329+0000","last_active":"2024-03-24T12:34:31.046693+0000","last_peered":"2024-03-24T12:34:31.046693+0000","last_clean":"2024-03-24T12:34:31.046693+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:34:31.046693+0000","last_undegraded":"2024-03-24T12:34:31.046693+0000","last_fullsized":"2024-03-24T12:34:31.046693+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:33:35.288932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:35.288932+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:02.021358+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"188'7341","reported_seq":7411,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-03-24T12:34:30.793592+0000","last_change":"2024-03-24T12:32:41.036940+0000","last_active":"2024-03-24T12:34:30.793592+0000","last_peered":"2024-03-24T12:34:30.793592+0000","last_clean":"2024-03-24T12:34:30.793592+0000","last_became_active":"2024-03-24T12:32:41.036596+0000","last_became_peered":"2024-03-24T12:32:41.036596+0000","last_unstale":"2024-03-24T12:34:30.793592+0000","last_undegraded":"2024-03-24T12:34:30.793592+0000","last_fullsized":"2024-03-24T12:34:30.793592+0000","mapping_epoch":158,"log_start":"169'1500","ondisk_log_start":"169'1500","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":5841,"log_dups_size":0,"ondisk_log_size":5841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":441909248,"num_objects":6743,"num_object_clones":0,"num_object_copies":20229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7341,"num_write_kb":450688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":271,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.030758+0000","last_change":"2024-03-24T12:33:35.381730+0000","last_active":"2024-03-24T12:34:31.030758+0000","last_peered":"2024-03-24T12:34:31.030758+0000","last_clean":"2024-03-24T12:34:31.030758+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:34:31.030758+0000","last_undegraded":"2024-03-24T12:34:31.030758+0000","last_fullsized":"2024-03-24T12:34:31.030758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:35.381614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:35.381614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:36.772645+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.050333+0000","last_change":"2024-03-24T12:33:15.388058+0000","last_active":"2024-03-24T12:34:31.050333+0000","last_peered":"2024-03-24T12:34:31.050333+0000","last_clean":"2024-03-24T12:34:31.050333+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:34:31.050333+0000","last_undegraded":"2024-03-24T12:34:31.050333+0000","last_fullsized":"2024-03-24T12:34:31.050333+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:15.387796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:15.387796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":311,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.046549+0000","last_change":"2024-03-24T12:33:33.420555+0000","last_active":"2024-03-24T12:34:31.046549+0000","last_peered":"2024-03-24T12:34:31.046549+0000","last_clean":"2024-03-24T12:34:31.046549+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:34:31.046549+0000","last_undegraded":"2024-03-24T12:34:31.046549+0000","last_fullsized":"2024-03-24T12:34:31.046549+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T12:33:33.420394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:33.420394+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:45.052567+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":275,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.025552+0000","last_change":"2024-03-24T12:33:02.044769+0000","last_active":"2024-03-24T12:34:31.025552+0000","last_peered":"2024-03-24T12:34:31.025552+0000","last_clean":"2024-03-24T12:34:31.025552+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:34:31.025552+0000","last_undegraded":"2024-03-24T12:34:31.025552+0000","last_fullsized":"2024-03-24T12:34:31.025552+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:02.044666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.044666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.491534+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"189'7576","reported_seq":7648,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:33.165627+0000","last_change":"2024-03-24T12:32:41.043057+0000","last_active":"2024-03-24T12:34:33.165627+0000","last_peered":"2024-03-24T12:34:33.165627+0000","last_clean":"2024-03-24T12:34:33.165627+0000","last_became_active":"2024-03-24T12:32:41.042732+0000","last_became_peered":"2024-03-24T12:32:41.042732+0000","last_unstale":"2024-03-24T12:34:33.165627+0000","last_undegraded":"2024-03-24T12:34:33.165627+0000","last_fullsized":"2024-03-24T12:34:33.165627+0000","mapping_epoch":158,"log_start":"171'2600","ondisk_log_start":"171'2600","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":4976,"log_dups_size":0,"ondisk_log_size":4976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":442368000,"num_objects":6750,"num_object_clones":0,"num_object_copies":20250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7576,"num_write_kb":458432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.031267+0000","last_change":"2024-03-24T12:33:36.210705+0000","last_active":"2024-03-24T12:34:31.031267+0000","last_peered":"2024-03-24T12:34:31.031267+0000","last_clean":"2024-03-24T12:34:31.031267+0000","last_became_active":"2024-03-24T12:26:09.351061+0000","last_became_peered":"2024-03-24T12:26:09.351061+0000","last_unstale":"2024-03-24T12:34:31.031267+0000","last_undegraded":"2024-03-24T12:34:31.031267+0000","last_fullsized":"2024-03-24T12:34:31.031267+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:36.210440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.210440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:48.424406+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"140'9","reported_seq":385,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.028235+0000","last_change":"2024-03-24T12:31:21.743353+0000","last_active":"2024-03-24T12:34:31.028235+0000","last_peered":"2024-03-24T12:34:31.028235+0000","last_clean":"2024-03-24T12:34:31.028235+0000","last_became_active":"2024-03-24T12:26:07.328595+0000","last_became_peered":"2024-03-24T12:26:07.328595+0000","last_unstale":"2024-03-24T12:34:31.028235+0000","last_undegraded":"2024-03-24T12:34:31.028235+0000","last_fullsized":"2024-03-24T12:34:31.028235+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'9","last_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:31:21.743220+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379218+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.046460+0000","last_change":"2024-03-24T12:33:08.247372+0000","last_active":"2024-03-24T12:34:31.046460+0000","last_peered":"2024-03-24T12:34:31.046460+0000","last_clean":"2024-03-24T12:34:31.046460+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:34:31.046460+0000","last_undegraded":"2024-03-24T12:34:31.046460+0000","last_fullsized":"2024-03-24T12:34:31.046460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:08.247253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.247253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"92'22","reported_seq":285,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.028862+0000","last_change":"2024-03-24T12:33:02.599767+0000","last_active":"2024-03-24T12:34:31.028862+0000","last_peered":"2024-03-24T12:34:31.028862+0000","last_clean":"2024-03-24T12:34:31.028862+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:34:31.028862+0000","last_undegraded":"2024-03-24T12:34:31.028862+0000","last_fullsized":"2024-03-24T12:34:31.028862+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-24T12:33:02.599612+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.599612+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"189'7487","reported_seq":7559,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:32.420365+0000","last_change":"2024-03-24T12:32:41.044757+0000","last_active":"2024-03-24T12:34:32.420365+0000","last_peered":"2024-03-24T12:34:32.420365+0000","last_clean":"2024-03-24T12:34:32.420365+0000","last_became_active":"2024-03-24T12:32:41.044207+0000","last_became_peered":"2024-03-24T12:32:41.044207+0000","last_unstale":"2024-03-24T12:34:32.420365+0000","last_undegraded":"2024-03-24T12:34:32.420365+0000","last_fullsized":"2024-03-24T12:34:32.420365+0000","mapping_epoch":158,"log_start":"172'2700","ondisk_log_start":"172'2700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":4787,"log_dups_size":0,"ondisk_log_size":4787,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436666368,"num_objects":6663,"num_object_clones":0,"num_object_copies":19989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6663,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7487,"num_write_kb":452800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":262,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.049482+0000","last_change":"2024-03-24T12:33:19.377017+0000","last_active":"2024-03-24T12:34:31.049482+0000","last_peered":"2024-03-24T12:34:31.049482+0000","last_clean":"2024-03-24T12:34:31.049482+0000","last_became_active":"2024-03-24T12:26:09.643739+0000","last_became_peered":"2024-03-24T12:26:09.643739+0000","last_unstale":"2024-03-24T12:34:31.049482+0000","last_undegraded":"2024-03-24T12:34:31.049482+0000","last_fullsized":"2024-03-24T12:34:31.049482+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:19.376894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.376894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.032474+0000","last_change":"2024-03-24T12:33:12.115065+0000","last_active":"2024-03-24T12:34:31.032474+0000","last_peered":"2024-03-24T12:34:31.032474+0000","last_clean":"2024-03-24T12:34:31.032474+0000","last_became_active":"2024-03-24T12:26:07.332766+0000","last_became_peered":"2024-03-24T12:26:07.332766+0000","last_unstale":"2024-03-24T12:34:31.032474+0000","last_undegraded":"2024-03-24T12:34:31.032474+0000","last_fullsized":"2024-03-24T12:34:31.032474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:12.114959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.114959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:31.231516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.025649+0000","last_change":"2024-03-24T12:33:08.165239+0000","last_active":"2024-03-24T12:34:31.025649+0000","last_peered":"2024-03-24T12:34:31.025649+0000","last_clean":"2024-03-24T12:34:31.025649+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:34:31.025649+0000","last_undegraded":"2024-03-24T12:34:31.025649+0000","last_fullsized":"2024-03-24T12:34:31.025649+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:08.165107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.165107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:33.648796+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":297,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.028070+0000","last_change":"2024-03-24T12:32:55.028177+0000","last_active":"2024-03-24T12:34:31.028070+0000","last_peered":"2024-03-24T12:34:31.028070+0000","last_clean":"2024-03-24T12:34:31.028070+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:34:31.028070+0000","last_undegraded":"2024-03-24T12:34:31.028070+0000","last_fullsized":"2024-03-24T12:34:31.028070+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:32:55.028060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.028060+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379267+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"189'7578","reported_seq":7699,"reported_epoch":189,"state":"active+clean+scrubbing","last_fresh":"2024-03-24T12:34:33.200828+0000","last_change":"2024-03-24T12:34:31.526318+0000","last_active":"2024-03-24T12:34:33.200828+0000","last_peered":"2024-03-24T12:34:33.200828+0000","last_clean":"2024-03-24T12:34:33.200828+0000","last_became_active":"2024-03-24T12:32:41.563886+0000","last_became_peered":"2024-03-24T12:32:41.563886+0000","last_unstale":"2024-03-24T12:34:33.200828+0000","last_undegraded":"2024-03-24T12:34:33.200828+0000","last_fullsized":"2024-03-24T12:34:33.200828+0000","mapping_epoch":158,"log_start":"171'2600","ondisk_log_start":"171'2600","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":4799,"log_size":4978,"log_dups_size":0,"ondisk_log_size":4978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 1s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439615488,"num_objects":6708,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7578,"num_write_kb":457152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":270,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.029025+0000","last_change":"2024-03-24T12:32:56.667268+0000","last_active":"2024-03-24T12:34:31.029025+0000","last_peered":"2024-03-24T12:34:31.029025+0000","last_clean":"2024-03-24T12:34:31.029025+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:34:31.029025+0000","last_undegraded":"2024-03-24T12:34:31.029025+0000","last_fullsized":"2024-03-24T12:34:31.029025+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:56.667140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:32:56.667140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.039162+0000","last_change":"2024-03-24T12:33:13.797360+0000","last_active":"2024-03-24T12:34:31.039162+0000","last_peered":"2024-03-24T12:34:31.039162+0000","last_clean":"2024-03-24T12:34:31.039162+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:34:31.039162+0000","last_undegraded":"2024-03-24T12:34:31.039162+0000","last_fullsized":"2024-03-24T12:34:31.039162+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:13.797243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:13.797243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":282,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.036783+0000","last_change":"2024-03-24T12:32:54.886366+0000","last_active":"2024-03-24T12:34:31.036783+0000","last_peered":"2024-03-24T12:34:31.036783+0000","last_clean":"2024-03-24T12:34:31.036783+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:34:31.036783+0000","last_undegraded":"2024-03-24T12:34:31.036783+0000","last_fullsized":"2024-03-24T12:34:31.036783+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:54.886269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:32:54.886269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.277761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"94'25","reported_seq":306,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.025809+0000","last_change":"2024-03-24T12:31:09.668493+0000","last_active":"2024-03-24T12:34:31.025809+0000","last_peered":"2024-03-24T12:34:31.025809+0000","last_clean":"2024-03-24T12:34:31.025809+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:34:31.025809+0000","last_undegraded":"2024-03-24T12:34:31.025809+0000","last_fullsized":"2024-03-24T12:34:31.025809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'25","last_scrub_stamp":"2024-03-24T12:31:09.668392+0000","last_deep_scrub":"94'25","last_deep_scrub_stamp":"2024-03-24T12:28:02.436198+0000","last_clean_scrub_stamp":"2024-03-24T12:31:09.668392+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379198+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"189'7618","reported_seq":16670,"reported_epoch":189,"state":"active+remapped+backfilling","last_fresh":"2024-03-24T12:34:34.097615+0000","last_change":"2024-03-24T12:33:52.465268+0000","last_active":"2024-03-24T12:34:34.097615+0000","last_peered":"2024-03-24T12:34:34.097615+0000","last_clean":"2024-03-24T12:33:50.009597+0000","last_became_active":"2024-03-24T12:33:52.453752+0000","last_became_peered":"2024-03-24T12:33:52.453752+0000","last_unstale":"2024-03-24T12:34:34.097615+0000","last_undegraded":"2024-03-24T12:34:34.097615+0000","last_fullsized":"2024-03-24T12:34:34.097615+0000","mapping_epoch":181,"log_start":"172'2700","ondisk_log_start":"172'2700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:32:40.019252+0000","objects_scrubbed":0,"log_size":4918,"log_dups_size":0,"ondisk_log_size":4918,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437387264,"num_objects":6674,"num_object_clones":0,"num_object_copies":20022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1928,"num_objects_unfound":0,"num_objects_dirty":6674,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7618,"num_write_kb":457344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4487,"num_bytes_recovered":294060032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,1],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":6674}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":278,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.028316+0000","last_change":"2024-03-24T12:33:10.193697+0000","last_active":"2024-03-24T12:34:31.028316+0000","last_peered":"2024-03-24T12:34:31.028316+0000","last_clean":"2024-03-24T12:34:31.028316+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:34:31.028316+0000","last_undegraded":"2024-03-24T12:34:31.028316+0000","last_fullsized":"2024-03-24T12:34:31.028316+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:10.193581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:10.193581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379411+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":385,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.028340+0000","last_change":"2024-03-24T12:32:57.614709+0000","last_active":"2024-03-24T12:34:31.028340+0000","last_peered":"2024-03-24T12:34:31.028340+0000","last_clean":"2024-03-24T12:34:31.028340+0000","last_became_active":"2024-03-24T12:26:07.334798+0000","last_became_peered":"2024-03-24T12:26:07.334798+0000","last_unstale":"2024-03-24T12:34:31.028340+0000","last_undegraded":"2024-03-24T12:34:31.028340+0000","last_fullsized":"2024-03-24T12:34:31.028340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:32:57.614572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:32:57.614572+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":274,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.046006+0000","last_change":"2024-03-24T12:33:21.383831+0000","last_active":"2024-03-24T12:34:31.046006+0000","last_peered":"2024-03-24T12:34:31.046006+0000","last_clean":"2024-03-24T12:34:31.046006+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:34:31.046006+0000","last_undegraded":"2024-03-24T12:34:31.046006+0000","last_fullsized":"2024-03-24T12:34:31.046006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.383703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.383703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":293,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-03-24T12:34:31.027055+0000","last_change":"2024-03-24T12:31:17.834752+0000","last_active":"2024-03-24T12:34:31.027055+0000","last_peered":"2024-03-24T12:34:31.027055+0000","last_clean":"2024-03-24T12:34:31.027055+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:34:31.027055+0000","last_undegraded":"2024-03-24T12:34:31.027055+0000","last_fullsized":"2024-03-24T12:34:31.027055+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:31:17.834622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:31:17.834622+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:34:35.379235+0000","scrub_duration":100,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":7039221784,"num_objects":107411,"num_object_clones":0,"num_object_copies":322233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1928,"num_objects_unfound":0,"num_objects_dirty":107411,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":120305,"num_write_kb":7286849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4487,"num_bytes_recovered":294060032,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21417308160,"data_stored":21417295944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85505,"ondisk_log_size":85505,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":502,"num_read_kb":497,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037866,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2375488,"kb_used_data":2145780,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":91369664,"statfs":{"total":95995035648,"available":93562535936,"internally_reserved":0,"allocated":2197278720,"data_stored":2196864653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":235207114},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332208,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2367392,"kb_used_data":2151576,"kb_used_omap":1,"kb_used_meta":215806,"kb_avail":91377760,"statfs":{"total":95995035648,"available":93570826240,"internally_reserved":0,"allocated":2203213824,"data_stored":2202827817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":220985799},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574987,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3271824,"kb_used_data":3024008,"kb_used_omap":1,"kb_used_meta":247742,"kb_avail":90473328,"statfs":{"total":95995035648,"available":92644687872,"internally_reserved":0,"allocated":3096584192,"data_stored":3096149467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":253688261},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738511,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2963180,"kb_used_data":2634852,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":90781972,"statfs":{"total":95995035648,"available":92960739328,"internally_reserved":0,"allocated":2698088448,"data_stored":2697683166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411461,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3193188,"kb_used_data":2877144,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":90551964,"statfs":{"total":95995035648,"available":92725211136,"internally_reserved":0,"allocated":2946195456,"data_stored":2945808389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":323353029},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117118,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2842884,"kb_used_data":2554876,"kb_used_omap":1,"kb_used_meta":287998,"kb_avail":90902268,"statfs":{"total":95995035648,"available":93083922432,"internally_reserved":0,"allocated":2616193024,"data_stored":2615770513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":294910407},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920890,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2334968,"kb_used_data":2159404,"kb_used_omap":1,"kb_used_meta":175550,"kb_avail":91410184,"statfs":{"total":95995035648,"available":93604028416,"internally_reserved":0,"allocated":2211229696,"data_stored":2210822395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":179763653},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626549,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3646892,"kb_used_data":3444964,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":90098260,"statfs":{"total":95995035648,"available":92260618240,"internally_reserved":0,"allocated":3527643136,"data_stored":3527245155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2679963648,"data_stored":2679963648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3076325376,"data_stored":3076325376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2942046208,"data_stored":2942042136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2607742976,"data_stored":2607742976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2209480704,"data_stored":2209480704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3516928000,"data_stored":3516923928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2198798336,"data_stored":2198798336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2186022912,"data_stored":2186018840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:34:40.900 INFO:tasks.thrashosds.thrasher:Setting 8.b to [7, 3, 6, 5, 1, 4] 2024-03-24T12:34:40.900 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '8.b', '7', '3', '6', '5', '1', '4'] 2024-03-24T12:34:40.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pg-upmap-items 8.b 7 3 6 5 1 4 2024-03-24T12:34:40.904 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:40.950 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:40.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:41.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:41.073 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:34:41.103 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:41.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:41.104 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:41.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:41 smithi029 ceph-mon[29919]: 8.9 scrub starts 2024-03-24T12:34:41.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:41 smithi029 ceph-mon[29919]: pgmap v346: 148 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 145 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 961 op/s; 1076/307707 objects misplaced (0.350%); 8.6 MiB/s, 138 objects/s recovering 2024-03-24T12:34:41.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:41 smithi029 ceph-mon[29919]: 8.7 scrub ok 2024-03-24T12:34:41.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:41 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:34:41.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:41 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:34:41.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:41 smithi029 ceph-mon[41500]: 8.9 scrub starts 2024-03-24T12:34:41.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:41 smithi029 ceph-mon[41500]: pgmap v346: 148 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 145 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 961 op/s; 1076/307707 objects misplaced (0.350%); 8.6 MiB/s, 138 objects/s recovering 2024-03-24T12:34:41.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:41 smithi029 ceph-mon[41500]: 8.7 scrub ok 2024-03-24T12:34:41.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:41 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:34:41.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:41 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:34:41.185 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:41.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:41.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:41.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:41.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:41.344 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:41 smithi190 ceph-mon[33748]: 8.9 scrub starts 2024-03-24T12:34:41.344 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:41 smithi190 ceph-mon[33748]: pgmap v346: 148 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 145 active+clean; 6.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 961 op/s; 1076/307707 objects misplaced (0.350%); 8.6 MiB/s, 138 objects/s recovering 2024-03-24T12:34:41.344 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:41 smithi190 ceph-mon[33748]: 8.7 scrub ok 2024-03-24T12:34:41.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:41 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:34:41.345 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:41 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:34:41.373 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:41.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:41.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:41.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:41.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:41.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:41.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:41.663 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:41.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:41.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:41.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:41.813 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:41.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:41.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:41.915 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:41.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:41.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:42.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:42.102 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:42.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:42.163 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:42 smithi029 ceph-mon[41500]: 8.5 scrub starts 2024-03-24T12:34:42.163 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:42 smithi029 ceph-mon[29919]: 8.5 scrub starts 2024-03-24T12:34:42.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:42.238 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:42.239 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:42.265 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:34:42.304 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:34:42.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:42.347 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:34:42.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:42.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:42 smithi190 ceph-mon[33748]: 8.5 scrub starts 2024-03-24T12:34:42.454 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:34:42.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:42.494 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:34:42.511 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:42.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:42.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:42.535 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:34:42.578 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:34:42.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:42.621 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:34:42.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:34:42.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:34:42.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:34:42.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:34:42.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:34:42.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:34:42.624 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:34:42.625 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:34:42.655 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:42.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:42.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:42.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:42.780 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:42.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:42.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:42.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:42.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:43.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:43.059 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:43.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:43.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:43.161 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:43.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:43.189 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:43.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:43.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:43.308 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:43 smithi190 ceph-mon[33748]: 8.6 scrub ok 2024-03-24T12:34:43.308 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:43 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2119350504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:34:43.308 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:43 smithi190 ceph-mon[33748]: 8.5 scrub ok 2024-03-24T12:34:43.308 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:43 smithi190 ceph-mon[33748]: pgmap v347: 148 pgs: 1 active+remapped+backfilling, 3 active+clean+scrubbing, 144 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 953 op/s; 1076/300996 objects misplaced (0.357%); 7.5 MiB/s, 119 objects/s recovering 2024-03-24T12:34:43.308 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:43 smithi190 ceph-mon[33748]: 8.9 scrub ok 2024-03-24T12:34:43.309 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:43 smithi190 ceph-mon[33748]: 6.1c scrub starts 2024-03-24T12:34:43.309 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:43 smithi190 ceph-mon[33748]: 6.1c scrub ok 2024-03-24T12:34:43.314 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:43.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:43.328 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:34:43.328 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:34:43.328 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-03-24T12:34:43.328 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:34:43.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:43.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[41500]: 8.6 scrub ok 2024-03-24T12:34:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2119350504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:34:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[41500]: 8.5 scrub ok 2024-03-24T12:34:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[41500]: pgmap v347: 148 pgs: 1 active+remapped+backfilling, 3 active+clean+scrubbing, 144 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 953 op/s; 1076/300996 objects misplaced (0.357%); 7.5 MiB/s, 119 objects/s recovering 2024-03-24T12:34:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[41500]: 8.9 scrub ok 2024-03-24T12:34:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[41500]: 6.1c scrub starts 2024-03-24T12:34:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[41500]: 6.1c scrub ok 2024-03-24T12:34:43.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[29919]: 8.6 scrub ok 2024-03-24T12:34:43.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2119350504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:34:43.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[29919]: 8.5 scrub ok 2024-03-24T12:34:43.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[29919]: pgmap v347: 148 pgs: 1 active+remapped+backfilling, 3 active+clean+scrubbing, 144 active+clean; 6.1 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 953 op/s; 1076/300996 objects misplaced (0.357%); 7.5 MiB/s, 119 objects/s recovering 2024-03-24T12:34:43.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[29919]: 8.9 scrub ok 2024-03-24T12:34:43.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[29919]: 6.1c scrub starts 2024-03-24T12:34:43.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:43 smithi029 ceph-mon[29919]: 6.1c scrub ok 2024-03-24T12:34:43.443 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:43.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:43.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:43.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:43.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:43.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:43.716 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:43.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:43.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:43.818 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:43.840 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:43.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:43.942 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:43.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:43.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:44.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:44.115 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:44.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:44.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:34:44.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:44.254 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:44.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:44.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:44.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:44.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:44 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:34:44.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:44 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:34:44.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:44.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:44.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:44.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:44 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:34:44.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:44 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:34:44.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:44 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:34:44.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:44 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:34:44.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:44.502 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:44.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:44.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:44.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:44.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:44.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:44.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:44.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:44.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:44.786 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:44.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:44.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:44.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:44.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:44.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:44.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:45.025 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:45.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:45.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:45.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:45.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:45.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:45.333 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:45.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:45.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:45.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:45 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:34:45.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:45 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:34:45.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:45 smithi190 ceph-mon[33748]: 4.d scrub starts 2024-03-24T12:34:45.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:45 smithi190 ceph-mon[33748]: 4.d scrub ok 2024-03-24T12:34:45.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:45 smithi190 ceph-mon[33748]: 5.2 scrub starts 2024-03-24T12:34:45.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:45 smithi190 ceph-mon[33748]: 5.2 scrub ok 2024-03-24T12:34:45.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:45 smithi190 ceph-mon[33748]: pgmap v348: 148 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 145 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 907 op/s; 337/295848 objects misplaced (0.114%); 11 MiB/s, 173 objects/s recovering 2024-03-24T12:34:45.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:34:45.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:34:45.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[29919]: 4.d scrub starts 2024-03-24T12:34:45.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[29919]: 4.d scrub ok 2024-03-24T12:34:45.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[29919]: 5.2 scrub starts 2024-03-24T12:34:45.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[29919]: 5.2 scrub ok 2024-03-24T12:34:45.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[29919]: pgmap v348: 148 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 145 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 907 op/s; 337/295848 objects misplaced (0.114%); 11 MiB/s, 173 objects/s recovering 2024-03-24T12:34:45.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:34:45.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:34:45.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[41500]: 4.d scrub starts 2024-03-24T12:34:45.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[41500]: 4.d scrub ok 2024-03-24T12:34:45.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[41500]: 5.2 scrub starts 2024-03-24T12:34:45.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[41500]: 5.2 scrub ok 2024-03-24T12:34:45.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:45 smithi029 ceph-mon[41500]: pgmap v348: 148 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 145 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 907 op/s; 337/295848 objects misplaced (0.114%); 11 MiB/s, 173 objects/s recovering 2024-03-24T12:34:45.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:45.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:45.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:45.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:45.592 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:45.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:45.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:45.718 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:45.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:45.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:45.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:45.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:45.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:45.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:45.998 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:45.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:45.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:46.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:46.124 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:46.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:46.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:46.248 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:46.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:46.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:46.361 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:34:46.361 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:34:46.361 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-03-24T12:34:46.362 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-03-24T12:34:46.362 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:34:46.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:46 smithi190 ceph-mon[33748]: 5.1b scrub starts 2024-03-24T12:34:46.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:46 smithi190 ceph-mon[33748]: 5.1b scrub ok 2024-03-24T12:34:46.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:46 smithi190 ceph-mon[33748]: 5.e scrub starts 2024-03-24T12:34:46.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:46 smithi190 ceph-mon[33748]: 5.e scrub ok 2024-03-24T12:34:46.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:46 smithi190 ceph-mon[33748]: 8.3 scrub starts 2024-03-24T12:34:46.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:46 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:34:46.362 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:46 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:34:46.373 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:46.375 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[29919]: 5.1b scrub starts 2024-03-24T12:34:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[29919]: 5.1b scrub ok 2024-03-24T12:34:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[29919]: 5.e scrub starts 2024-03-24T12:34:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[29919]: 5.e scrub ok 2024-03-24T12:34:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[29919]: 8.3 scrub starts 2024-03-24T12:34:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:34:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:34:46.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[41500]: 5.1b scrub starts 2024-03-24T12:34:46.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[41500]: 5.1b scrub ok 2024-03-24T12:34:46.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[41500]: 5.e scrub starts 2024-03-24T12:34:46.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[41500]: 5.e scrub ok 2024-03-24T12:34:46.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[41500]: 8.3 scrub starts 2024-03-24T12:34:46.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:34:46.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:46 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:34:46.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:46.518 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:46.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:46.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:46.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:46.657 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:46.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:46.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:46.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:46.798 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:46.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:46.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:46.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:46.938 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:46.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:46.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:47.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:47.063 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:47.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:47.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:34:47.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:47.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:47.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:47.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:47.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:47.359 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:47.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:47.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:47 smithi190 ceph-mon[33748]: pgmap v349: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 814 op/s; 337/293541 objects misplaced (0.115%); 7.7 MiB/s, 123 objects/s recovering 2024-03-24T12:34:47.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:47 smithi029 ceph-mon[41500]: pgmap v349: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 814 op/s; 337/293541 objects misplaced (0.115%); 7.7 MiB/s, 123 objects/s recovering 2024-03-24T12:34:47.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:47 smithi029 ceph-mon[29919]: pgmap v349: 148 pgs: 1 active+remapped+backfilling, 1 active+clean+scrubbing, 146 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 814 op/s; 337/293541 objects misplaced (0.115%); 7.7 MiB/s, 123 objects/s recovering 2024-03-24T12:34:47.441 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:34:47.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:47.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:47.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:47.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:47.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:47.615 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:47.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:47.716 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:47.747 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:47.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:47.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:47.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:47.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:47.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:47.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:47.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:34:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:34:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:34:47.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:48.037 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:48.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:48.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:48.110 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:34:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:34:48.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:48.163 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:48.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:48.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:48.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:48.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:48.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:48.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:48.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 8.d scrub starts 2024-03-24T12:34:48.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 3.1b scrub starts 2024-03-24T12:34:48.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 3.1b scrub ok 2024-03-24T12:34:48.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 3.10 scrub starts 2024-03-24T12:34:48.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 3.10 scrub ok 2024-03-24T12:34:48.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:34:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:34:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 5.4 scrub starts 2024-03-24T12:34:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 5.4 scrub ok 2024-03-24T12:34:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 6.11 scrub starts 2024-03-24T12:34:48.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:48 smithi190 ceph-mon[33748]: 6.11 scrub ok 2024-03-24T12:34:48.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:48.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 8.d scrub starts 2024-03-24T12:34:48.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 3.1b scrub starts 2024-03-24T12:34:48.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 3.1b scrub ok 2024-03-24T12:34:48.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 3.10 scrub starts 2024-03-24T12:34:48.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 3.10 scrub ok 2024-03-24T12:34:48.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 5.4 scrub starts 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 5.4 scrub ok 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 6.11 scrub starts 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[41500]: 6.11 scrub ok 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 8.d scrub starts 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 3.1b scrub starts 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 3.1b scrub ok 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 3.10 scrub starts 2024-03-24T12:34:48.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 3.10 scrub ok 2024-03-24T12:34:48.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:34:48.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:34:48.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:34:48.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 5.4 scrub starts 2024-03-24T12:34:48.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 5.4 scrub ok 2024-03-24T12:34:48.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:48.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 6.11 scrub starts 2024-03-24T12:34:48.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:48 smithi029 ceph-mon[29919]: 6.11 scrub ok 2024-03-24T12:34:48.427 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:48.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:48.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:48.554 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:48.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:48.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:48.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:48.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:48.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:48.805 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:48.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:48.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:48.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:48.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:48.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:48.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:49.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:49.055 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:49.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:49.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:49.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:49.184 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:49.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:49.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:49.313 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:49.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:49.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: 8.d scrub ok 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: 8.3 scrub ok 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: pgmap v351: 148 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 145 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 811 op/s; 337/286002 objects misplaced (0.118%); 9.3 MiB/s, 148 objects/s recovering 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: 5.19 scrub starts 2024-03-24T12:34:49.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:49 smithi190 ceph-mon[33748]: 5.19 scrub ok 2024-03-24T12:34:49.410 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158103 7.11 7:88d08e65:::benchmark_data_smithi029_84282_object57340:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.558928+0000", 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.851937467, 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1526137300000001, 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.411 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158103 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.558928+0000", 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.558928+0000", 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.558929+0000", 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8599999999999997e-07 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.558933+0000", 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6290000000000001e-06 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.558948+0000", 2024-03-24T12:34:49.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4945e-05 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.558957+0000", 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4360000000000002e-06 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.559003+0000", 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6779000000000001e-05 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.559013+0000", 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5540000000000003e-06 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.704773+0000", 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1457596350000001 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.704895+0000", 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012264900000000001 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.705260+0000", 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036436999999999998 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.711303+0000", 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0060431089999999996 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.711363+0000", 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0349000000000001e-05 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.711421+0000", 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8387999999999997e-05 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.711482+0000", 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0411999999999997e-05 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.711541+0000", 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9589000000000002e-05 2024-03-24T12:34:49.417 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158124 7.11 7:8bff5d4b:::benchmark_data_smithi029_84282_object57361:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.584646+0000", 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.82621910699999, 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1319555400000001, 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158124 2024-03-24T12:34:49.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.584646+0000", 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.584646+0000", 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.584647+0000", 2024-03-24T12:34:49.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2109999999999999e-06 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.584652+0000", 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5079999999999999e-06 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.584663+0000", 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1647999999999999e-05 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.584670+0000", 2024-03-24T12:34:49.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8020000000000003e-06 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.584722+0000", 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2148999999999998e-05 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.584733+0000", 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0169999999999999e-05 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.707760+0000", 2024-03-24T12:34:49.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1230271570000001 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.707908+0000", 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014825999999999999 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.708215+0000", 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000307084 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.711632+0000", 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003417064 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.711917+0000", 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000284543 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716488+0000", 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045709790000000002 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716549+0000", 2024-03-24T12:34:49.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1295000000000002e-05 2024-03-24T12:34:49.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:49.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716602+0000", 2024-03-24T12:34:49.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2670000000000002e-05 2024-03-24T12:34:49.425 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.425 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.425 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.425 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158210 7.11 7:8bc670cf:::benchmark_data_smithi029_84282_object57447:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.425 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.680750+0000", 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.73011473099999, 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0361326909999999, 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158210 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.680750+0000", 2024-03-24T12:34:49.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.680750+0000", 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.680752+0000", 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2160000000000001e-06 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.680756+0000", 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4429999999999996e-06 2024-03-24T12:34:49.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.680768+0000", 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1788e-05 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.680775+0000", 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9929999999999998e-06 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.680829+0000", 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4589000000000002e-05 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.680851+0000", 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1605000000000001e-05 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.709821+0000", 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.028969909 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.709939+0000", 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011789400000000001 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.710177+0000", 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000238194 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.711933+0000", 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001756262 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716432+0000", 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044981689999999998 2024-03-24T12:34:49.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716721+0000", 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028966000000000002 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716816+0000", 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5044999999999995e-05 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716883+0000", 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6923999999999998e-05 2024-03-24T12:34:49.431 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158244 7.11 7:8d77eef5:::benchmark_data_smithi029_84282_object57481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.716767+0000", 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.694097836, 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.004186231, 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158244 2024-03-24T12:34:49.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.716767+0000", 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.716767+0000", 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.716768+0000", 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.099e-06 2024-03-24T12:34:49.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.716773+0000", 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2860000000000001e-06 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.716783+0000", 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.011e-05 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.716789+0000", 2024-03-24T12:34:49.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.601e-06 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.716845+0000", 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5615000000000002e-05 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.716852+0000", 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0099999999999998e-06 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.711966+0000", 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.99511360699999996 2024-03-24T12:34:49.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.712081+0000", 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011534799999999999 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.712342+0000", 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026139199999999999 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716447+0000", 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041046870000000001 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716664+0000", 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000216622 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.720877+0000", 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042135860000000001 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.720910+0000", 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2529e-05 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.720954+0000", 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3739000000000003e-05 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158255 7.11 7:8a18a15b:::benchmark_data_smithi029_84282_object57492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.726541+0000", 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.68432460700001, 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.99451952399999999, 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.438 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158255 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.726541+0000", 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.726541+0000", 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.726542+0000", 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.438e-06 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.726546+0000", 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6780000000000002e-06 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.726557+0000", 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1240000000000001e-05 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.726563+0000", 2024-03-24T12:34:49.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8119999999999996e-06 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.726619+0000", 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6150000000000003e-05 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.726628+0000", 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0389999999999993e-06 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.714237+0000", 2024-03-24T12:34:49.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.98760933799999995 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.714368+0000", 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013031499999999999 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.714667+0000", 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029938099999999999 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716899+0000", 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022317700000000001 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.716939+0000", 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0425e-05 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.721010+0000", 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040705029999999996 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.721038+0000", 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8470999999999999e-05 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.721060+0000", 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1963999999999998e-05 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158259 7.11 7:8ed9278f:::benchmark_data_smithi029_84282_object57496:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.729459+0000", 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.681406649, 2024-03-24T12:34:49.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.99801850599999997, 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158259 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.729459+0000", 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.729459+0000", 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.729460+0000", 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.274e-06 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.729465+0000", 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1340000000000003e-06 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.729480+0000", 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4599e-05 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.729488+0000", 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3850000000000002e-06 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.729548+0000", 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9984e-05 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.729557+0000", 2024-03-24T12:34:49.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.67e-06 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.717000+0000", 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.98744316700000001 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.717131+0000", 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000130987 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.717391+0000", 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026039200000000002 2024-03-24T12:34:49.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.721071+0000", 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036800320000000002 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.721107+0000", 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5759000000000001e-05 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727333+0000", 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0062258319999999997 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727425+0000", 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2714999999999996e-05 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727477+0000", 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1576000000000002e-05 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158279 7.11 7:8f46a358:::benchmark_data_smithi029_84282_object57516:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.450 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.746728+0000", 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.66413765499999, 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.98088313199999999, 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158279 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.746728+0000", 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.746728+0000", 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.746729+0000", 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9800000000000002e-07 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.746732+0000", 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9020000000000001e-06 2024-03-24T12:34:49.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.746743+0000", 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0417e-05 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.746749+0000", 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1959999999999996e-06 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.746802+0000", 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2864000000000001e-05 2024-03-24T12:34:49.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.746808+0000", 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1380000000000004e-06 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.719377+0000", 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.97256884799999999 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.719470+0000", 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3209999999999997e-05 2024-03-24T12:34:49.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.719685+0000", 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021444500000000001 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.723026+0000", 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033415749999999998 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.723101+0000", 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5376000000000005e-05 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727535+0000", 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044339030000000003 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727577+0000", 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1801999999999997e-05 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727611+0000", 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3457999999999997e-05 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158297 7.11 7:8cddb0c6:::benchmark_data_smithi029_84282_object57534:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.770963+0000", 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.63990169499999, 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.95681925899999998, 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158297 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.770963+0000", 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.770963+0000", 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.770965+0000", 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.049e-06 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.770968+0000", 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.439e-06 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.770978+0000", 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8169999999999994e-06 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.770986+0000", 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9999999999999996e-06 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.771041+0000", 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5040000000000002e-05 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.771049+0000", 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3760000000000008e-06 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.721160+0000", 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.95011102400000003 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.721270+0000", 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010955699999999999 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.721457+0000", 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000187042 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727502+0000", 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0060455719999999999 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727658+0000", 2024-03-24T12:34:49.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015606499999999999 2024-03-24T12:34:49.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727707+0000", 2024-03-24T12:34:49.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8807000000000002e-05 2024-03-24T12:34:49.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727749+0000", 2024-03-24T12:34:49.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1779000000000001e-05 2024-03-24T12:34:49.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727783+0000", 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3692000000000002e-05 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158301 7.11 7:8c321b10:::benchmark_data_smithi029_84282_object57538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.772554+0000", 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.63831082999999, 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.95776772399999999, 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.464 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158301 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.772554+0000", 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.772554+0000", 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.772555+0000", 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0699999999999996e-07 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.772559+0000", 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8809999999999998e-06 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.772569+0000", 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0484e-05 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.772577+0000", 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7370000000000006e-06 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.772620+0000", 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2877000000000003e-05 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.772627+0000", 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8730000000000001e-06 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.723130+0000", 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.95050296000000001 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.723225+0000", 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4821999999999994e-05 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.723446+0000", 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022086600000000001 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727671+0000", 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042253489999999998 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730190+0000", 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025190249999999998 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730245+0000", 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5009999999999997e-05 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730280+0000", 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5471e-05 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730322+0000", 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1662000000000002e-05 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158340 7.11 7:8a2c264d:::benchmark_data_smithi029_84282_object57577:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.470 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.819277+0000", 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.59158779500001, 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91116639399999999, 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158340 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.819277+0000", 2024-03-24T12:34:49.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.819277+0000", 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.819279+0000", 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1620000000000001e-06 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.819282+0000", 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9090000000000002e-06 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.819294+0000", 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1671999999999999e-05 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.819301+0000", 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1620000000000002e-06 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.819349+0000", 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7926999999999997e-05 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.819357+0000", 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8690000000000004e-06 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.724902+0000", 2024-03-24T12:34:49.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.90554537300000004 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.724992+0000", 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9845000000000005e-05 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.725177+0000", 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000184215 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727677+0000", 2024-03-24T12:34:49.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025003780000000001 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730225+0000", 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025480279999999999 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730364+0000", 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000139572 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730409+0000", 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4991999999999999e-05 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730444+0000", 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4289999999999999e-05 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158377 7.11 7:8e379ecf:::benchmark_data_smithi029_84282_object57614:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:18.988694+0000", 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.422171542, 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.75544624999999999, 2024-03-24T12:34:49.477 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158377 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.988694+0000", 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.988694+0000", 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.988695+0000", 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.094e-06 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.988698+0000", 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7579999999999998e-06 2024-03-24T12:34:49.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.988708+0000", 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0015e-05 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.988715+0000", 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1800000000000001e-06 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.988771+0000", 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6480999999999997e-05 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:18.988778+0000", 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8480000000000003e-06 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727809+0000", 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.73903122399999999 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.727923+0000", 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000113619 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.728244+0000", 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032151899999999998 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.734657+0000", 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064127460000000004 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.734699+0000", 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1798999999999997e-05 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.744034+0000", 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0093346269999999999 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.744088+0000", 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4449e-05 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.744140+0000", 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1891000000000002e-05 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.483 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:158403 7.11 7:8b60a5be:::benchmark_data_smithi029_84282_object57640:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:19.016206+0000", 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 210.39465902699999, 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.73155653099999995, 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 158403 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.016206+0000", 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.016206+0000", 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.016207+0000", 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1689999999999999e-06 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.016212+0000", 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5970000000000002e-06 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.016223+0000", 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1317e-05 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.016260+0000", 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6952000000000001e-05 2024-03-24T12:34:49.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.016326+0000", 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6112000000000007e-05 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.016336+0000", 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0147e-05 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730490+0000", 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.714153802 2024-03-24T12:34:49.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730604+0000", 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011373499999999999 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.730922+0000", 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031838100000000001 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.734710+0000", 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037880090000000002 2024-03-24T12:34:49.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.734747+0000", 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6856000000000001e-05 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.747658+0000", 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012911267000000001 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.747713+0000", 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4364000000000003e-05 2024-03-24T12:34:49.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:19.747763+0000", 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9823000000000001e-05 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:163334 7.17 7:e92ccd1a:::benchmark_data_smithi029_84282_object62571:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:26.040900+0000", 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 203.36996540999999, 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0548734790000001, 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.490 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163334 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.040900+0000", 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.040900+0000", 2024-03-24T12:34:49.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.040901+0000", 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7320000000000001e-06 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.040907+0000", 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.181e-06 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.040918+0000", 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.182e-05 2024-03-24T12:34:49.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.040927+0000", 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4619999999999996e-06 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.041098+0000", 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017138099999999999 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.041108+0000", 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3500000000000003e-06 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.089801+0000", 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0486929300000001 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.089955+0000", 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015414199999999999 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.090772+0000", 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00081692899999999998 2024-03-24T12:34:49.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.095630+0000", 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048585410000000001 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.095642+0000", 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1669e-05 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.095679+0000", 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7438000000000003e-05 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.095733+0000", 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3570999999999999e-05 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.095773+0000", 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0333000000000002e-05 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.496 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:163354 7.17 7:ee0e0516:::benchmark_data_smithi029_84282_object62591:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:26.065128+0000", 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 203.34573742000001, 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0381393860000001, 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163354 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.065128+0000", 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.065128+0000", 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.065129+0000", 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0729999999999999e-06 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.065133+0000", 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7579999999999998e-06 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.065142+0000", 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4900000000000006e-06 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.065150+0000", 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9049999999999997e-06 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.065229+0000", 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9054000000000006e-05 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.065240+0000", 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0742e-05 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.095804+0000", 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0305645830000001 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.095911+0000", 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010683699999999999 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.096293+0000", 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038208699999999998 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.100678+0000", 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0043848569999999998 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.103120+0000", 2024-03-24T12:34:49.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002441679 2024-03-24T12:34:49.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.103149+0000", 2024-03-24T12:34:49.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9074999999999999e-05 2024-03-24T12:34:49.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.103218+0000", 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8993999999999997e-05 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.103267+0000", 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9252000000000002e-05 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:163378 7.17 7:e9099786:::benchmark_data_smithi029_84282_object62615:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:26.087399+0000", 2024-03-24T12:34:49.504 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 203.32346575099999, 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0218722929999999, 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163378 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.087399+0000", 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.087399+0000", 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.087401+0000", 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1090000000000001e-06 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.087404+0000", 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.771e-06 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.087414+0000", 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7450000000000007e-06 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.087422+0000", 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7649999999999993e-06 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.087489+0000", 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7057999999999994e-05 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.087500+0000", 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1146e-05 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.098395+0000", 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.010895399 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.098518+0000", 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012257899999999999 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.098803+0000", 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028493600000000001 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.103097+0000", 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042941580000000002 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.106347+0000", 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003250143 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109078+0000", 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027311359999999999 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109172+0000", 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3743e-05 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109272+0000", 2024-03-24T12:34:49.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9605000000000006e-05 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:163393 7.17 7:ee910b58:::benchmark_data_smithi029_84282_object62630:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:26.104509+0000", 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 203.30635595199999, 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.004866662, 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.511 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163393 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.104509+0000", 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.104509+0000", 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.104510+0000", 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1659999999999999e-06 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.104514+0000", 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0169999999999998e-06 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.104526+0000", 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1697e-05 2024-03-24T12:34:49.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.104533+0000", 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4119999999999998e-06 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.104589+0000", 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6447999999999999e-05 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.104598+0000", 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0880000000000001e-06 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.100710+0000", 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.99611219200000001 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.100828+0000", 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117914 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.101221+0000", 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039268400000000002 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.108897+0000", 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00767594 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.108918+0000", 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1299999999999999e-05 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109315+0000", 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039670100000000001 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109340+0000", 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5534e-05 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109376+0000", 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5568999999999998e-05 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.517 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:163397 7.17 7:ed376353:::benchmark_data_smithi029_84282_object62634:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:26.107287+0000", 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 203.30357814499999, 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0021557059999999, 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.518 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163397 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.107287+0000", 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.107287+0000", 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.107288+0000", 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1480000000000001e-06 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.107293+0000", 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6739999999999996e-06 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.107304+0000", 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1071e-05 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.107311+0000", 2024-03-24T12:34:49.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2470000000000002e-06 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.107369+0000", 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7411999999999999e-05 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.107380+0000", 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.111e-05 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.103314+0000", 2024-03-24T12:34:49.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.99593454400000003 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.103436+0000", 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012207200000000001 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.103720+0000", 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028326499999999999 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.108987+0000", 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0052671059999999997 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109288+0000", 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000301382 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109401+0000", 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112926 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109425+0000", 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3668000000000001e-05 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109443+0000", 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8080999999999998e-05 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:163425 7.17 7:ebc5377e:::benchmark_data_smithi029_84282_object62662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:26.332520+0000", 2024-03-24T12:34:49.524 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 203.07834560200001, 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.79219982099999997, 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163425 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.332520+0000", 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.332520+0000", 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.332521+0000", 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4419999999999999e-06 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.332526+0000", 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5059999999999997e-06 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.332540+0000", 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4273999999999999e-05 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.332548+0000", 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8909999999999995e-06 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.332603+0000", 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5764999999999999e-05 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.332617+0000", 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3057000000000001e-05 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.106377+0000", 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.773760789 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.106490+0000", 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011228800000000001 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.106848+0000", 2024-03-24T12:34:49.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000358029 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109381+0000", 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025335359999999999 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109464+0000", 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2489000000000003e-05 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.124628+0000", 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.015164086 2024-03-24T12:34:49.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.124679+0000", 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0807000000000003e-05 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.124719+0000", 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0862000000000003e-05 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.530 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:163447 7.17 7:ef7c21c3:::benchmark_data_smithi029_84282_object62684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:26.358164+0000", 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 203.052700813, 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.766818624, 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163447 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.358164+0000", 2024-03-24T12:34:49.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.358164+0000", 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.358166+0000", 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.282e-06 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.358170+0000", 2024-03-24T12:34:49.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0729999999999998e-06 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.358180+0000", 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9459999999999993e-06 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.358186+0000", 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2890000000000003e-06 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.358256+0000", 2024-03-24T12:34:49.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9821999999999997e-05 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.358268+0000", 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2112e-05 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109480+0000", 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.75121200899999996 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109570+0000", 2024-03-24T12:34:49.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0070999999999999e-05 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.109826+0000", 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000256259 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.116190+0000", 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063642079999999997 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.120579+0000", 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0043889910000000001 2024-03-24T12:34:49.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.124879+0000", 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042998079999999996 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.124934+0000", 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4521000000000002e-05 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.124983+0000", 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9233000000000001e-05 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.536 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:163510 7.17 7:e9d4a9df:::benchmark_data_smithi029_84282_object62747:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:26.418263+0000", 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 202.99260253599999, 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.71425771100000002, 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 163510 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:34:49.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.418263+0000", 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.418263+0000", 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.418264+0000", 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.141e-06 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.418268+0000", 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.014e-06 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.418280+0000", 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2627e-05 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.418289+0000", 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3699999999999995e-06 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.418357+0000", 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8448999999999995e-05 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:26.418373+0000", 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5696e-05 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.113967+0000", 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.6955945 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.114087+0000", 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000119214 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.114409+0000", 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032228200000000002 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.124733+0000", 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010323575 2024-03-24T12:34:49.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.124824+0000", 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1873000000000006e-05 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.132401+0000", 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0075771010000000001 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.132472+0000", 2024-03-24T12:34:49.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0358999999999995e-05 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:27.132520+0000", 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8510000000000001e-05 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:34:49.543 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:34:49.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:49.544 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:49.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:49.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:49.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:34:49.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: 8.d scrub ok 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: 8.3 scrub ok 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: pgmap v351: 148 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 145 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 811 op/s; 337/286002 objects misplaced (0.118%); 9.3 MiB/s, 148 objects/s recovering 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: 5.19 scrub starts 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[29919]: 5.19 scrub ok 2024-03-24T12:34:49.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: osdmap e190: 8 total, 8 up, 8 in 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: 8.d scrub ok 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: 8.3 scrub ok 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: pgmap v351: 148 pgs: 2 active+clean+scrubbing, 1 active+remapped+backfilling, 145 active+clean; 5.8 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 811 op/s; 337/286002 objects misplaced (0.118%); 9.3 MiB/s, 148 objects/s recovering 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: 5.19 scrub starts 2024-03-24T12:34:49.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:49 smithi029 ceph-mon[41500]: 5.19 scrub ok 2024-03-24T12:34:49.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:49.670 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:49.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:49.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:49.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:49.805 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:49.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:49.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:49.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:49.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:49.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:50.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:50.096 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:50.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:50.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:50.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:50.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:50.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:50.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:50.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:50.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:34:50.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:50.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:50.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:50.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:50.500 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:50.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:50.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:50.623 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:50.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:50.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:50 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:34:50.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:50 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:34:50.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:50 smithi190 ceph-mon[33748]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T12:34:50.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:50 smithi190 ceph-mon[33748]: 6.1e scrub starts 2024-03-24T12:34:50.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:50 smithi190 ceph-mon[33748]: 6.1e scrub ok 2024-03-24T12:34:50.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:50 smithi190 ceph-mon[33748]: 8.a scrub starts 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[29919]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[29919]: 6.1e scrub starts 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[29919]: 6.1e scrub ok 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[29919]: 8.a scrub starts 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[41500]: osdmap e191: 8 total, 8 up, 8 in 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[41500]: 6.1e scrub starts 2024-03-24T12:34:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[41500]: 6.1e scrub ok 2024-03-24T12:34:50.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:50 smithi029 ceph-mon[41500]: 8.a scrub starts 2024-03-24T12:34:50.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:50.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:50.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:50.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:50.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:50.902 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:50.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:50.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:51.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:51.050 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:51.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:51.152 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:51.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:51.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:51.287 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:51.313 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:51.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:51.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:51.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:51.440 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:51.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:51.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:51.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:51.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:51.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:51.574 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:51.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:51 smithi190 ceph-mon[33748]: 8.e scrub starts 2024-03-24T12:34:51.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:51 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:34:51.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:51 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:34:51.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:51 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:34:51.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:51 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:34:51.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:51 smithi190 ceph-mon[33748]: pgmap v353: 148 pgs: 1 peering, 2 active+clean+scrubbing, 145 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 704 op/s; 8.1 MiB/s, 128 objects/s recovering 2024-03-24T12:34:51.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:51 smithi190 ceph-mon[33748]: 3.11 scrub starts 2024-03-24T12:34:51.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:51 smithi190 ceph-mon[33748]: 3.11 scrub ok 2024-03-24T12:34:51.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[41500]: 8.e scrub starts 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[41500]: pgmap v353: 148 pgs: 1 peering, 2 active+clean+scrubbing, 145 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 704 op/s; 8.1 MiB/s, 128 objects/s recovering 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[41500]: 3.11 scrub starts 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[41500]: 3.11 scrub ok 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[29919]: 8.e scrub starts 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:34:51.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:34:51.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:34:51.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[29919]: pgmap v353: 148 pgs: 1 peering, 2 active+clean+scrubbing, 145 active+clean; 5.8 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 704 op/s; 8.1 MiB/s, 128 objects/s recovering 2024-03-24T12:34:51.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[29919]: 3.11 scrub starts 2024-03-24T12:34:51.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:51 smithi029 ceph-mon[29919]: 3.11 scrub ok 2024-03-24T12:34:51.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:51.701 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:51.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:51.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:51.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:51.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:51.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:51.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:51.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:51.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:51.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:52.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:52.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:52.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:52.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:52.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:52.227 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:52.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:52.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:52.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:52.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:52.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:52.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:52.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:52.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:52.631 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:52.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:52.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:52.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:52 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:34:52.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:52 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:34:52.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:52 smithi190 ceph-mon[33748]: 6.1f scrub starts 2024-03-24T12:34:52.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:52 smithi190 ceph-mon[33748]: 6.1f scrub ok 2024-03-24T12:34:52.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:52 smithi190 ceph-mon[33748]: 5.7 scrub starts 2024-03-24T12:34:52.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:52 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:34:52.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:52 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:34:52.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:52 smithi029 ceph-mon[41500]: 6.1f scrub starts 2024-03-24T12:34:52.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:52 smithi029 ceph-mon[41500]: 6.1f scrub ok 2024-03-24T12:34:52.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:52 smithi029 ceph-mon[41500]: 5.7 scrub starts 2024-03-24T12:34:52.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:52 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:34:52.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:52 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:34:52.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:52 smithi029 ceph-mon[29919]: 6.1f scrub starts 2024-03-24T12:34:52.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:52 smithi029 ceph-mon[29919]: 6.1f scrub ok 2024-03-24T12:34:52.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:52 smithi029 ceph-mon[29919]: 5.7 scrub starts 2024-03-24T12:34:52.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:52.772 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:52.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:52.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:52.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:52.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:52.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:52.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:53.005 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:53.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:53.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:53.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:53.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:53.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:53.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:53.300 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:53.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:53.401 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:53.425 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:53.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:53.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:53.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:53.560 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:53.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:53 smithi190 ceph-mon[33748]: 8.e scrub ok 2024-03-24T12:34:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:53 smithi190 ceph-mon[33748]: 5.7 scrub ok 2024-03-24T12:34:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:53 smithi190 ceph-mon[33748]: pgmap v354: 148 pgs: 1 peering, 2 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 701 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-03-24T12:34:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:53 smithi190 ceph-mon[33748]: 8.a scrub ok 2024-03-24T12:34:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:53 smithi190 ceph-mon[33748]: 4.14 scrub starts 2024-03-24T12:34:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:53 smithi190 ceph-mon[33748]: 4.14 scrub ok 2024-03-24T12:34:53.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[29919]: 8.e scrub ok 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[29919]: 5.7 scrub ok 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[29919]: pgmap v354: 148 pgs: 1 peering, 2 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 701 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[29919]: 8.a scrub ok 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[29919]: 4.14 scrub starts 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[29919]: 4.14 scrub ok 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[41500]: 8.e scrub ok 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[41500]: 5.7 scrub ok 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[41500]: pgmap v354: 148 pgs: 1 peering, 2 active+clean+scrubbing, 145 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 701 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[41500]: 8.a scrub ok 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[41500]: 4.14 scrub starts 2024-03-24T12:34:53.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:53 smithi029 ceph-mon[41500]: 4.14 scrub ok 2024-03-24T12:34:53.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:53.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:53.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:53.808 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:53.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:53.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:53.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:53.944 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:53.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:53.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:54.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:54.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:54.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:54.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:54.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:54.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:54.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:54.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:54.361 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:54.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:54.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:54.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:54.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:54.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:54.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:54.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:54.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:54.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:54.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:54.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:54.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:54.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:54.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:54.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:54 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:34:54.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:54 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:34:54.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:54 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:34:54.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:54 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:34:54.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:54 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:34:54.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:54 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:34:54.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:54 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:34:54.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:54 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:34:54.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:34:54.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:34:54.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:34:54.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:34:54.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:34:54.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:54 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:34:54.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:54.931 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:54.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:55.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:55.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:55.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:55.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:55.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:55.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:55.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:55.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:55.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:55.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:55.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:55.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:55.483 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:55.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:55.504 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:34:55.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:55.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:55.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:55.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:55.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:55.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:55.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:55.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: 6.10 scrub starts 2024-03-24T12:34:55.847 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: 6.10 scrub ok 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: pgmap v355: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 792 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: 3.13 scrub ok 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: 4.11 scrub ok 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: 3.4 scrub starts 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[29919]: 3.4 scrub ok 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: 6.10 scrub starts 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: 6.10 scrub ok 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:34:55.848 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:34:55.849 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: pgmap v355: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 792 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-03-24T12:34:55.849 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:34:55.849 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: 3.13 scrub ok 2024-03-24T12:34:55.849 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:34:55.849 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: 4.11 scrub ok 2024-03-24T12:34:55.849 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: 3.4 scrub starts 2024-03-24T12:34:55.849 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:55 smithi029 ceph-mon[41500]: 3.4 scrub ok 2024-03-24T12:34:55.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: 6.10 scrub starts 2024-03-24T12:34:55.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: 6.10 scrub ok 2024-03-24T12:34:55.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:34:55.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:34:55.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: pgmap v355: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 792 op/s; 2.6 MiB/s, 41 objects/s recovering 2024-03-24T12:34:55.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:34:55.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: 3.13 scrub ok 2024-03-24T12:34:55.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:34:55.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: 4.11 scrub ok 2024-03-24T12:34:55.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: 3.4 scrub starts 2024-03-24T12:34:55.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:55 smithi190 ceph-mon[33748]: 3.4 scrub ok 2024-03-24T12:34:55.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:55.938 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:55.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:55.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:56.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:56.076 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:56.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:56.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:56.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:56.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:56.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:56.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:56.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:56.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:56.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:56.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:56.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:56.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:56.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:56.591 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:56.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:56.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:56.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:56.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:56.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:56.759 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:56.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:56.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:56.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:56 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:34:56.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:56 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:34:56.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:56 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:34:56.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:56 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:34:56.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:56 smithi190 ceph-mon[33748]: 5.3 scrub starts 2024-03-24T12:34:56.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:56 smithi190 ceph-mon[33748]: 5.3 scrub ok 2024-03-24T12:34:56.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:56 smithi190 ceph-mon[33748]: 6.12 scrub starts 2024-03-24T12:34:56.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:56 smithi190 ceph-mon[33748]: 6.12 scrub ok 2024-03-24T12:34:56.898 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:56.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:56.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:56.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:34:56.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:34:56.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:34:56.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:34:56.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[29919]: 5.3 scrub starts 2024-03-24T12:34:56.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[29919]: 5.3 scrub ok 2024-03-24T12:34:56.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[29919]: 6.12 scrub starts 2024-03-24T12:34:56.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[29919]: 6.12 scrub ok 2024-03-24T12:34:56.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:34:56.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:34:56.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:34:56.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:34:56.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[41500]: 5.3 scrub starts 2024-03-24T12:34:56.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[41500]: 5.3 scrub ok 2024-03-24T12:34:56.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[41500]: 6.12 scrub starts 2024-03-24T12:34:56.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:56 smithi029 ceph-mon[41500]: 6.12 scrub ok 2024-03-24T12:34:57.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:57.025 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:57.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:57.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:57.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:57.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:57.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:57.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:57.298 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:57.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:57.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:57.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:57.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:57.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:57.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:57.556 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:57.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:57.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:57.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:57.696 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:57.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:57.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:57.837 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:57.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:57.868 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:57 smithi190 ceph-mon[33748]: pgmap v356: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 622 B/s rd, 0 B/s wr, 695 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-03-24T12:34:57.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:57 smithi029 ceph-mon[41500]: pgmap v356: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 622 B/s rd, 0 B/s wr, 695 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-03-24T12:34:57.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:57 smithi029 ceph-mon[29919]: pgmap v356: 148 pgs: 148 active+clean; 5.5 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 622 B/s rd, 0 B/s wr, 695 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-03-24T12:34:57.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:34:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:34:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:34:57.939 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:57.980 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:57.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:58.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:58.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:58.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:58.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:34:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:34:58.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:34:58.263 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:58.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:58.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:34:58.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:58.405 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:58.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:58.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:58.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:58.530 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:58.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:58.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:58.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:58.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:34:58.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:34:58.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[41500]: 8.2 scrub starts 2024-03-24T12:34:58.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:58.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[41500]: 6.13 scrub starts 2024-03-24T12:34:58.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[41500]: 6.13 scrub ok 2024-03-24T12:34:58.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:34:58.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:34:58.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[29919]: 8.2 scrub starts 2024-03-24T12:34:58.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:58.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[29919]: 6.13 scrub starts 2024-03-24T12:34:58.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:58 smithi029 ceph-mon[29919]: 6.13 scrub ok 2024-03-24T12:34:58.673 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:58.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:58.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:58.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:58 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:34:58.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:58 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:34:58.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:58 smithi190 ceph-mon[33748]: 8.2 scrub starts 2024-03-24T12:34:58.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:34:58.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:58 smithi190 ceph-mon[33748]: 6.13 scrub starts 2024-03-24T12:34:58.709 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:58 smithi190 ceph-mon[33748]: 6.13 scrub ok 2024-03-24T12:34:58.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:34:58.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:58.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:34:58.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:58.950 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:58.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:58.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:59.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:34:59.085 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:59.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:34:59.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:59.220 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:59.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:34:59.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:34:59.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:59.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:59.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:34:59.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:34:59.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:34:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:59.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:34:59.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:34:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:59.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:59.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:34:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:59 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:34:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:59 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:34:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:59 smithi029 ceph-mon[41500]: pgmap v357: 148 pgs: 148 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 686 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-03-24T12:34:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:34:59 smithi029 ceph-mon[41500]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T12:34:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:59 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:34:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:59 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:34:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:59 smithi029 ceph-mon[29919]: pgmap v357: 148 pgs: 148 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 686 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-03-24T12:34:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:34:59 smithi029 ceph-mon[29919]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T12:34:59.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:34:59.786 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:59.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:59.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:34:59.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:59 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:34:59.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:59 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:34:59.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:59 smithi190 ceph-mon[33748]: pgmap v357: 148 pgs: 148 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 686 op/s; 2.1 MiB/s, 33 objects/s recovering 2024-03-24T12:34:59.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:34:59 smithi190 ceph-mon[33748]: osdmap e192: 8 total, 8 up, 8 in 2024-03-24T12:34:59.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:34:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:34:59.930 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:34:59.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:00.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:00.068 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:00.069 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:00.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:00.208 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:00.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:00.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:00.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:00.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:00.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:00.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:00.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:00.480 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:35:00.575 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:00.610 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:00.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:00.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:00.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:00.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:00.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:00.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:00.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:00.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:00.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:00.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:00 smithi190 ceph-mon[33748]: 6.18 scrub starts 2024-03-24T12:35:00.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:00 smithi190 ceph-mon[33748]: 6.18 scrub ok 2024-03-24T12:35:00.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:00 smithi190 ceph-mon[33748]: 8.2 scrub ok 2024-03-24T12:35:00.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:00 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2484913290' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:35:00.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:00 smithi190 ceph-mon[33748]: 6.4 scrub starts 2024-03-24T12:35:00.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:00 smithi190 ceph-mon[33748]: 6.4 scrub ok 2024-03-24T12:35:00.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:00 smithi190 ceph-mon[33748]: 8.0 scrub starts 2024-03-24T12:35:00.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[29919]: 6.18 scrub starts 2024-03-24T12:35:00.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[29919]: 6.18 scrub ok 2024-03-24T12:35:00.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[29919]: 8.2 scrub ok 2024-03-24T12:35:00.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2484913290' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:35:00.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[29919]: 6.4 scrub starts 2024-03-24T12:35:00.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[29919]: 6.4 scrub ok 2024-03-24T12:35:00.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[29919]: 8.0 scrub starts 2024-03-24T12:35:00.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[41500]: 6.18 scrub starts 2024-03-24T12:35:00.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[41500]: 6.18 scrub ok 2024-03-24T12:35:00.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[41500]: 8.2 scrub ok 2024-03-24T12:35:00.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2484913290' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:35:00.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[41500]: 6.4 scrub starts 2024-03-24T12:35:00.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[41500]: 6.4 scrub ok 2024-03-24T12:35:00.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:00 smithi029 ceph-mon[41500]: 8.0 scrub starts 2024-03-24T12:35:00.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:01.003 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:01.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:01.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:01.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:01.117 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:01.177 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:01.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:01.177 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:01.278 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:01.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:01.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:01.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:01.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:01.449 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:01.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:01.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:01.584 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:01.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:01.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:01.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:01.726 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:01.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:01.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:01.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:01.839 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[29919]: Cluster is now healthy 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2484913290' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[29919]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[29919]: pgmap v360: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 888 op/s 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[29919]: 3.5 scrub starts 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[29919]: 3.5 scrub ok 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[41500]: Cluster is now healthy 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2484913290' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[41500]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[41500]: pgmap v360: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 888 op/s 2024-03-24T12:35:01.840 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[41500]: 3.5 scrub starts 2024-03-24T12:35:01.841 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:01 smithi029 ceph-mon[41500]: 3.5 scrub ok 2024-03-24T12:35:01.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:35:01.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:01.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:01.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:01.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:01 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:35:01.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:01 smithi190 ceph-mon[33748]: Cluster is now healthy 2024-03-24T12:35:01.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:01 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2484913290' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:35:01.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:01 smithi190 ceph-mon[33748]: osdmap e193: 8 total, 8 up, 8 in 2024-03-24T12:35:01.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:01 smithi190 ceph-mon[33748]: pgmap v360: 148 pgs: 148 active+clean; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 888 op/s 2024-03-24T12:35:01.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:01 smithi190 ceph-mon[33748]: 3.5 scrub starts 2024-03-24T12:35:01.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:01 smithi190 ceph-mon[33748]: 3.5 scrub ok 2024-03-24T12:35:01.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:02.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:02.006 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:02.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:02.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:02.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:02.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:02.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:02.300 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:02.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:02.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:02.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:02.450 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:02.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:02.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:02.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:02.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:02.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:02.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:02.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:02.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:02.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:02.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:02.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:02.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:02.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:02.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:02.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:02 smithi190 ceph-mon[33748]: 5.d scrub starts 2024-03-24T12:35:02.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:02 smithi190 ceph-mon[33748]: 5.d scrub ok 2024-03-24T12:35:02.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:02 smithi190 ceph-mon[33748]: 8.b scrub starts 2024-03-24T12:35:02.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:02 smithi190 ceph-mon[33748]: 6.17 scrub starts 2024-03-24T12:35:02.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:02 smithi190 ceph-mon[33748]: 8.0 scrub ok 2024-03-24T12:35:02.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:02 smithi190 ceph-mon[33748]: 6.17 scrub ok 2024-03-24T12:35:02.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:02 smithi190 ceph-mon[33748]: 8.f scrub starts 2024-03-24T12:35:02.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:02 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:35:02.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:02 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:02.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[41500]: 5.d scrub starts 2024-03-24T12:35:02.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[41500]: 5.d scrub ok 2024-03-24T12:35:02.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[41500]: 8.b scrub starts 2024-03-24T12:35:02.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[41500]: 6.17 scrub starts 2024-03-24T12:35:02.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[41500]: 8.0 scrub ok 2024-03-24T12:35:02.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[41500]: 6.17 scrub ok 2024-03-24T12:35:02.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[41500]: 8.f scrub starts 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[29919]: 5.d scrub starts 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[29919]: 5.d scrub ok 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[29919]: 8.b scrub starts 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[29919]: 6.17 scrub starts 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[29919]: 8.0 scrub ok 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[29919]: 6.17 scrub ok 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[29919]: 8.f scrub starts 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:35:02.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:02 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:02.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:02.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:03.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:03.116 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:03.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:03.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:03.251 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:03.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:03.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:03.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:03.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:03.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:03.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:03.531 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:03.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:03.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: pgmap v361: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 935 op/s 2024-03-24T12:35:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: 3.e scrub starts 2024-03-24T12:35:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: 3.e scrub ok 2024-03-24T12:35:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: 5.1c scrub starts 2024-03-24T12:35:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: 5.1c scrub ok 2024-03-24T12:35:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: 4.12 scrub starts 2024-03-24T12:35:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: 4.12 scrub ok 2024-03-24T12:35:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: 3.1e scrub starts 2024-03-24T12:35:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: 3.1e scrub ok 2024-03-24T12:35:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:35:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: pgmap v361: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 935 op/s 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: 3.e scrub starts 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: 3.e scrub ok 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: 5.1c scrub starts 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: 5.1c scrub ok 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: 4.12 scrub starts 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: 4.12 scrub ok 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: 3.1e scrub starts 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: 3.1e scrub ok 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:35:03.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:03 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:35:03.681 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:03.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:03.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:03.827 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:03.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:03.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: pgmap v361: 148 pgs: 148 active+clean; 5.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 935 op/s 2024-03-24T12:35:03.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: 3.e scrub starts 2024-03-24T12:35:03.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: 3.e scrub ok 2024-03-24T12:35:03.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: 5.1c scrub starts 2024-03-24T12:35:03.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: 5.1c scrub ok 2024-03-24T12:35:03.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: 4.12 scrub starts 2024-03-24T12:35:03.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: 4.12 scrub ok 2024-03-24T12:35:03.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: 3.1e scrub starts 2024-03-24T12:35:03.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: 3.1e scrub ok 2024-03-24T12:35:03.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:35:03.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:03 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:35:03.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:35:03.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:03.958 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:03.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:04.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:04.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:04.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:04.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:04.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:04.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:04.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:04.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:04.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:04.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:04.387 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:04.389 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:04.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:04.517 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:04.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:04.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:04.549 INFO:teuthology.orchestra.run.smithi029.stderr:set 8.b pg_upmap_items mapping to [7->3,6->5,1->4] 2024-03-24T12:35:04.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:04.673 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:04.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:04.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:04.811 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:04.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:04.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:04 smithi190 ceph-mon[33748]: 8.b scrub ok 2024-03-24T12:35:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:04 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2407064886' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [7, 3, 6, 5, 1, 4]}]: dispatch 2024-03-24T12:35:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:04 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:35:04.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:04 smithi029 ceph-mon[29919]: 8.b scrub ok 2024-03-24T12:35:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:04 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2407064886' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [7, 3, 6, 5, 1, 4]}]: dispatch 2024-03-24T12:35:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:04 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:35:04.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:04 smithi029 ceph-mon[41500]: 8.b scrub ok 2024-03-24T12:35:04.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:04 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2407064886' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [7, 3, 6, 5, 1, 4]}]: dispatch 2024-03-24T12:35:04.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:04 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:35:04.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:04.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:04.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:05.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:05.086 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:05.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:05.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:05.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:05.224 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:05.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:05.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:05.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:05.362 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:05.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:05.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:05.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:05.500 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:05.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:05.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:05.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:05.639 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:05.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:05.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:35:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: 8.f scrub ok 2024-03-24T12:35:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:35:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:35:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2407064886' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [7, 3, 6, 5, 1, 4]}]': finished 2024-03-24T12:35:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T12:35:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: pgmap v363: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-24T12:35:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: 6.15 scrub starts 2024-03-24T12:35:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: 6.15 scrub ok 2024-03-24T12:35:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: 8.f scrub ok 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2407064886' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [7, 3, 6, 5, 1, 4]}]': finished 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: pgmap v363: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: 6.15 scrub starts 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: 6.15 scrub ok 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:35:05.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:05 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:35:05.741 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:05.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:05.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:05.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:35:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: 8.f scrub ok 2024-03-24T12:35:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:35:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:35:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2407064886' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "8.b", "id": [7, 3, 6, 5, 1, 4]}]': finished 2024-03-24T12:35:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: osdmap e194: 8 total, 8 up, 8 in 2024-03-24T12:35:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: pgmap v363: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.26k op/s 2024-03-24T12:35:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: 6.15 scrub starts 2024-03-24T12:35:05.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: 6.15 scrub ok 2024-03-24T12:35:05.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:35:05.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:05 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:35:05.911 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:05.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:05.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:06.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:06.047 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:06.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:06.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:06.149 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:06.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:06.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:06.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:06.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:06.292 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:35:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:35:06.292 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-03-24T12:35:06.292 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:35:06.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:06.323 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:06.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:06.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:06.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:06.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:06.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:06.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:06.594 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:06.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:06.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:06.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:06.730 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:06.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:06.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:06.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:06.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:06.875 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:06.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:06.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:06 smithi190 ceph-mon[33748]: 3.17 scrub starts 2024-03-24T12:35:06.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:06 smithi190 ceph-mon[33748]: 3.17 scrub ok 2024-03-24T12:35:06.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:06 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:35:06.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:06 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:35:06.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:06 smithi029 ceph-mon[41500]: 3.17 scrub starts 2024-03-24T12:35:06.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:06 smithi029 ceph-mon[41500]: 3.17 scrub ok 2024-03-24T12:35:06.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:06 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:35:06.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:06 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:35:06.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:06 smithi029 ceph-mon[29919]: 3.17 scrub starts 2024-03-24T12:35:06.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:06 smithi029 ceph-mon[29919]: 3.17 scrub ok 2024-03-24T12:35:06.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:06 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:35:06.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:06 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:35:06.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:07.009 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:07.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:07.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:07.146 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:07.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:07.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:07.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:07.286 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:07.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:07.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:07.435 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:07.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:07.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:07.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:07.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:07.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:07.594 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:07 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:35:07.594 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:07 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:35:07.594 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:07 smithi029 ceph-mon[41500]: pgmap v364: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 642 B/s rd, 0 B/s wr, 895 op/s 2024-03-24T12:35:07.594 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:07 smithi029 ceph-mon[41500]: 6.1b scrub starts 2024-03-24T12:35:07.594 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:07 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:35:07.594 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:07 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:35:07.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:07 smithi029 ceph-mon[29919]: pgmap v364: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 642 B/s rd, 0 B/s wr, 895 op/s 2024-03-24T12:35:07.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:07 smithi029 ceph-mon[29919]: 6.1b scrub starts 2024-03-24T12:35:07.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:07.723 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:07.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:07.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:07.865 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:07.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:07.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:07.879 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:07 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:35:07.879 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:07 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:35:07.879 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:07 smithi190 ceph-mon[33748]: pgmap v364: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 642 B/s rd, 0 B/s wr, 895 op/s 2024-03-24T12:35:07.879 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:07 smithi190 ceph-mon[33748]: 6.1b scrub starts 2024-03-24T12:35:07.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:35:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:35:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:35:07.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:08.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:08.002 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:08.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:08.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:08.133 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:35:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:35:08.142 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:08.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:08.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:08.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:08.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:08.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:08.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:08.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:08.422 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:08.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:08.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:08.560 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:08.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:08.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:08.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:08.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:08.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:08.819 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[41500]: 6.1b scrub ok 2024-03-24T12:35:08.819 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:08.820 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[41500]: 6.6 scrub starts 2024-03-24T12:35:08.820 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[41500]: 6.6 scrub ok 2024-03-24T12:35:08.820 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[41500]: 4.15 scrub starts 2024-03-24T12:35:08.820 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[41500]: 4.15 scrub ok 2024-03-24T12:35:08.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[29919]: 6.1b scrub ok 2024-03-24T12:35:08.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:08.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[29919]: 6.6 scrub starts 2024-03-24T12:35:08.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[29919]: 6.6 scrub ok 2024-03-24T12:35:08.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[29919]: 4.15 scrub starts 2024-03-24T12:35:08.820 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:08 smithi029 ceph-mon[29919]: 4.15 scrub ok 2024-03-24T12:35:08.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:08.821 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:08.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:08 smithi190 ceph-mon[33748]: 6.1b scrub ok 2024-03-24T12:35:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:08 smithi190 ceph-mon[33748]: 6.6 scrub starts 2024-03-24T12:35:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:08 smithi190 ceph-mon[33748]: 6.6 scrub ok 2024-03-24T12:35:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:08 smithi190 ceph-mon[33748]: 4.15 scrub starts 2024-03-24T12:35:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:08 smithi190 ceph-mon[33748]: 4.15 scrub ok 2024-03-24T12:35:08.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:08.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:08.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:08.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:09.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:09.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:09.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:09.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:09.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:09.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:09.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:09.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:09.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:09.350 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:09.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:09.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:09.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:09.491 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:09.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:09.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:09.629 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:09.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:09.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:09.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:09.761 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:09.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:09.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:09 smithi190 ceph-mon[33748]: pgmap v365: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T12:35:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:09 smithi190 ceph-mon[33748]: 6.1a scrub starts 2024-03-24T12:35:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:09 smithi190 ceph-mon[33748]: 6.1a scrub ok 2024-03-24T12:35:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:09 smithi190 ceph-mon[33748]: 5.15 scrub starts 2024-03-24T12:35:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:09 smithi190 ceph-mon[33748]: 5.15 scrub ok 2024-03-24T12:35:09.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:09.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:09.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:09 smithi029 ceph-mon[29919]: pgmap v365: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T12:35:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:09 smithi029 ceph-mon[29919]: 6.1a scrub starts 2024-03-24T12:35:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:09 smithi029 ceph-mon[29919]: 6.1a scrub ok 2024-03-24T12:35:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:09 smithi029 ceph-mon[29919]: 5.15 scrub starts 2024-03-24T12:35:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:09 smithi029 ceph-mon[29919]: 5.15 scrub ok 2024-03-24T12:35:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:09 smithi029 ceph-mon[41500]: pgmap v365: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.7 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T12:35:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:09 smithi029 ceph-mon[41500]: 6.1a scrub starts 2024-03-24T12:35:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:09 smithi029 ceph-mon[41500]: 6.1a scrub ok 2024-03-24T12:35:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:09 smithi029 ceph-mon[41500]: 5.15 scrub starts 2024-03-24T12:35:09.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:09 smithi029 ceph-mon[41500]: 5.15 scrub ok 2024-03-24T12:35:09.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:10.032 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:10.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:10.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:10.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:10.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:10.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:10.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:10.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:10.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:10.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:10.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:10.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:10.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:10.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:35:10.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:35:10.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:35:10.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:10.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:10.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:10.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:10.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:10.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:10.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:10.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:10.839 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:10.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:10.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:10 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:35:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:10 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:35:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:10 smithi190 ceph-mon[33748]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T12:35:10.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:10 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:35:10.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:10 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:35:10.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:10 smithi029 ceph-mon[41500]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T12:35:10.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:10 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:35:10.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:10 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:35:10.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:10 smithi029 ceph-mon[29919]: osdmap e195: 8 total, 8 up, 8 in 2024-03-24T12:35:10.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:10.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:10.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:10.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:11.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:11.108 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:11.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:11.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:11.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:11.246 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:11.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:11.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:11.382 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:11.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:11.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:11.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:11.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:11.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:11.663 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:11.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:11.664 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:11.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:11.789 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:11.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:11.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:11.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:11 smithi190 ceph-mon[33748]: 6.9 scrub starts 2024-03-24T12:35:11.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:11 smithi190 ceph-mon[33748]: 6.9 scrub ok 2024-03-24T12:35:11.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:11 smithi190 ceph-mon[33748]: pgmap v367: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T12:35:11.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:11 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:35:11.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:11 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:35:11.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:11 smithi190 ceph-mon[33748]: 5.16 scrub starts 2024-03-24T12:35:11.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:11 smithi190 ceph-mon[33748]: 5.16 scrub ok 2024-03-24T12:35:11.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:11.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:11.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[29919]: 6.9 scrub starts 2024-03-24T12:35:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[29919]: 6.9 scrub ok 2024-03-24T12:35:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[29919]: pgmap v367: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T12:35:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:35:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:35:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[29919]: 5.16 scrub starts 2024-03-24T12:35:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[29919]: 5.16 scrub ok 2024-03-24T12:35:11.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[41500]: 6.9 scrub starts 2024-03-24T12:35:11.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[41500]: 6.9 scrub ok 2024-03-24T12:35:11.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[41500]: pgmap v367: 148 pgs: 148 active+clean; 4.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T12:35:11.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:35:11.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:35:11.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[41500]: 5.16 scrub starts 2024-03-24T12:35:11.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:11 smithi029 ceph-mon[41500]: 5.16 scrub ok 2024-03-24T12:35:12.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:12.054 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:12.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:12.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:12.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:12.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:12.287 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:12.323 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:12.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:12.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:12.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:12.454 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:12.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:12.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:12.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:12.579 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:12.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:12.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:12.717 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:12.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:12.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:12.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:12.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:12.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:12.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:12 smithi190 ceph-mon[33748]: 5.a scrub starts 2024-03-24T12:35:12.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:12 smithi190 ceph-mon[33748]: 5.a scrub ok 2024-03-24T12:35:12.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:12 smithi029 ceph-mon[29919]: 5.a scrub starts 2024-03-24T12:35:12.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:12 smithi029 ceph-mon[29919]: 5.a scrub ok 2024-03-24T12:35:12.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:12 smithi029 ceph-mon[41500]: 5.a scrub starts 2024-03-24T12:35:12.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:12 smithi029 ceph-mon[41500]: 5.a scrub ok 2024-03-24T12:35:12.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:12.973 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:12.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:13.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:13.111 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:13.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:13.213 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:13.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:13.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:13.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:13.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:13.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:13.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:13.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:13.505 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:13.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:13.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:13.630 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:13.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:13.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:13.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:13.765 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:13.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:13.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:13.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:35:13.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:35:13.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[29919]: pgmap v368: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 963 op/s 2024-03-24T12:35:13.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:35:13.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:35:13.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[29919]: 5.18 scrub starts 2024-03-24T12:35:13.855 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[29919]: 5.18 scrub ok 2024-03-24T12:35:13.855 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:35:13.856 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:35:13.856 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[41500]: pgmap v368: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 963 op/s 2024-03-24T12:35:13.856 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:35:13.856 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:35:13.856 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[41500]: 5.18 scrub starts 2024-03-24T12:35:13.856 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:13 smithi029 ceph-mon[41500]: 5.18 scrub ok 2024-03-24T12:35:13.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T12:35:13.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:13 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:35:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:13 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:35:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:13 smithi190 ceph-mon[33748]: pgmap v368: 148 pgs: 148 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 963 op/s 2024-03-24T12:35:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:13 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:35:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:13 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:35:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:13 smithi190 ceph-mon[33748]: 5.18 scrub starts 2024-03-24T12:35:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:13 smithi190 ceph-mon[33748]: 5.18 scrub ok 2024-03-24T12:35:13.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:13.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:13.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:13.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:14.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:14.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:14.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:14.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:14.162 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:14.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:14.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:14.308 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:14.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:14.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:14.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:14.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:14.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:14.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:14.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:14.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:14.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:14.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:14.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:14.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:14.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:14.845 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:14.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:14.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:14 smithi190 ceph-mon[33748]: 5.6 scrub starts 2024-03-24T12:35:14.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:14 smithi190 ceph-mon[33748]: 5.6 scrub ok 2024-03-24T12:35:14.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:14 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/163838511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:35:14.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:14 smithi029 ceph-mon[41500]: 5.6 scrub starts 2024-03-24T12:35:14.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:14 smithi029 ceph-mon[41500]: 5.6 scrub ok 2024-03-24T12:35:14.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:14 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/163838511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:35:14.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:14 smithi029 ceph-mon[29919]: 5.6 scrub starts 2024-03-24T12:35:14.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:14 smithi029 ceph-mon[29919]: 5.6 scrub ok 2024-03-24T12:35:14.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:14 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/163838511' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:35:14.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:14.962 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:35:14.962 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:35:14.962 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:35:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:35:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:35:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:35:14.982 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:14.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:15.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:15.121 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:15.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:15.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:15.255 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:15.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:15.356 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:15.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:15.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:15.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:15.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:15.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:15.541 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:15.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:15.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:15.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:15.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:15.824 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:15.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:15.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:15.872 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:15 smithi029 ceph-mon[41500]: pgmap v369: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T12:35:15.872 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:15 smithi029 ceph-mon[41500]: 4.1 scrub starts 2024-03-24T12:35:15.872 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:15 smithi029 ceph-mon[41500]: 4.1 scrub ok 2024-03-24T12:35:15.872 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:15 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:35:15.872 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:15 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:35:15.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:15 smithi029 ceph-mon[29919]: pgmap v369: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T12:35:15.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:15 smithi029 ceph-mon[29919]: 4.1 scrub starts 2024-03-24T12:35:15.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:15 smithi029 ceph-mon[29919]: 4.1 scrub ok 2024-03-24T12:35:15.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:15 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:35:15.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:15 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:35:15.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:15 smithi190 ceph-mon[33748]: pgmap v369: 148 pgs: 148 active+clean; 4.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T12:35:15.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:15 smithi190 ceph-mon[33748]: 4.1 scrub starts 2024-03-24T12:35:15.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:15 smithi190 ceph-mon[33748]: 4.1 scrub ok 2024-03-24T12:35:15.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:15 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:35:15.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:15 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:35:15.926 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:15.961 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:15.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:15.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:16.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:16.100 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:16.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:16.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:16.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:16.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:16.284 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:16.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:16.388 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:16.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:16.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:16.528 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:16.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:16.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:16.664 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:16.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:16.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:16.720 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[41500]: 3.1f scrub starts 2024-03-24T12:35:16.720 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[41500]: 3.1f scrub ok 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[41500]: 4.17 scrub starts 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[41500]: 4.17 scrub ok 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[41500]: 3.c scrub starts 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[41500]: 3.c scrub ok 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[29919]: 3.1f scrub starts 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[29919]: 3.1f scrub ok 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:35:16.721 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[29919]: 4.17 scrub starts 2024-03-24T12:35:16.722 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[29919]: 4.17 scrub ok 2024-03-24T12:35:16.722 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[29919]: 3.c scrub starts 2024-03-24T12:35:16.722 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:16 smithi029 ceph-mon[29919]: 3.c scrub ok 2024-03-24T12:35:16.766 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:16.796 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:16.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:16.797 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:16.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:16 smithi190 ceph-mon[33748]: 3.1f scrub starts 2024-03-24T12:35:16.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:16 smithi190 ceph-mon[33748]: 3.1f scrub ok 2024-03-24T12:35:16.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:16 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:35:16.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:16 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:35:16.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:16 smithi190 ceph-mon[33748]: 4.17 scrub starts 2024-03-24T12:35:16.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:16 smithi190 ceph-mon[33748]: 4.17 scrub ok 2024-03-24T12:35:16.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:16 smithi190 ceph-mon[33748]: 3.c scrub starts 2024-03-24T12:35:16.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:16 smithi190 ceph-mon[33748]: 3.c scrub ok 2024-03-24T12:35:16.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:16.942 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:16.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:17.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:17.075 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:17.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:17.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:17.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:17.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:17.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:17.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:17.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:17.501 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:17.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:17.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:17.632 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:17.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:17.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:17.702 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:35:17.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:17.769 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:17.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:17.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:17 smithi190 ceph-mon[33748]: 5.b scrub starts 2024-03-24T12:35:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:17 smithi190 ceph-mon[33748]: 5.b scrub ok 2024-03-24T12:35:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:17 smithi190 ceph-mon[33748]: pgmap v370: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T12:35:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:17 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:35:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:17 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:35:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:17.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:35:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:35:17.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[41500]: 5.b scrub starts 2024-03-24T12:35:17.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[41500]: 5.b scrub ok 2024-03-24T12:35:17.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[41500]: pgmap v370: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T12:35:17.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:35:17.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:35:17.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:17.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[29919]: 5.b scrub starts 2024-03-24T12:35:17.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[29919]: 5.b scrub ok 2024-03-24T12:35:17.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[29919]: pgmap v370: 148 pgs: 148 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T12:35:17.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:35:17.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:35:17.921 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:17.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:17.923 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:17.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:18.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:18.060 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:18.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:18.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:18.161 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:18.192 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:18.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:18.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:18.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:18.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:18.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:35:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:35:18.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:18.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:18.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:18.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:18.590 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:18.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:18.639 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:35:18.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:18.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:18.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:18.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:18.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:18.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:18.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:18 smithi190 ceph-mon[33748]: 4.3 deep-scrub starts 2024-03-24T12:35:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:18 smithi190 ceph-mon[33748]: 4.3 deep-scrub ok 2024-03-24T12:35:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:18 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1287613389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:35:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:18 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:35:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:18 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:35:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:18 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:35:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:18 smithi190 ceph-mon[33748]: 6.7 scrub starts 2024-03-24T12:35:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:18 smithi190 ceph-mon[33748]: 6.7 scrub ok 2024-03-24T12:35:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[29919]: 4.3 deep-scrub starts 2024-03-24T12:35:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[29919]: 4.3 deep-scrub ok 2024-03-24T12:35:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1287613389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:35:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:35:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:35:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:35:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[29919]: 6.7 scrub starts 2024-03-24T12:35:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[29919]: 6.7 scrub ok 2024-03-24T12:35:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[41500]: 4.3 deep-scrub starts 2024-03-24T12:35:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[41500]: 4.3 deep-scrub ok 2024-03-24T12:35:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1287613389' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:35:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:35:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:35:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:35:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[41500]: 6.7 scrub starts 2024-03-24T12:35:18.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:18 smithi029 ceph-mon[41500]: 6.7 scrub ok 2024-03-24T12:35:18.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:19.001 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:19.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:19.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:19.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:19.137 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:19.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:19.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:19.275 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:19.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:19.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:19.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:19.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:19.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:19.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:19.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:19.546 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:19.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:19.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:19.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:19.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:19.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:19.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:19.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:19.827 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:19.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:19.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:19 smithi190 ceph-mon[33748]: pgmap v371: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:35:19.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:19 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:35:19.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:19 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:35:19.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:19 smithi190 ceph-mon[33748]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T12:35:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:19 smithi029 ceph-mon[41500]: pgmap v371: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:35:19.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:19 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:35:19.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:19 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:35:19.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:19 smithi029 ceph-mon[41500]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T12:35:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:19 smithi029 ceph-mon[29919]: pgmap v371: 148 pgs: 148 active+clean; 4.1 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:35:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:19 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:35:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:19 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:35:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:19 smithi029 ceph-mon[29919]: osdmap e196: 8 total, 8 up, 8 in 2024-03-24T12:35:19.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:19.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:19.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:20.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:20.085 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:20.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:20.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:20.221 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:20.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:20.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:20.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:20.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:20.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:20.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:20.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:20.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:20.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:20.619 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:20.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:20.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:20.756 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:20.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:20.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:20.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:20.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:20.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:20.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:21.032 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:21.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:21.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:21.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:21.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:21.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:21.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:21.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:21.306 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:21.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:21 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:35:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:21 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:35:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:21 smithi190 ceph-mon[33748]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T12:35:21.400 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:21 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:35:21.400 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:21 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:35:21.401 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:21 smithi029 ceph-mon[41500]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T12:35:21.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:21 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:35:21.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:21 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:35:21.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:21 smithi029 ceph-mon[29919]: osdmap e197: 8 total, 8 up, 8 in 2024-03-24T12:35:21.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:21.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:21.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:21.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:21.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:21.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:21.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:21.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:21.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:21.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:21.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:21.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:21.831 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:21.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:21.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:21.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:21.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:21.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:22.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:22.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:22.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:22.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:22.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:22.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:22.243 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:22.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:22.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:22.378 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:22.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:22.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:22 smithi190 ceph-mon[33748]: pgmap v373: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T12:35:22.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:22 smithi029 ceph-mon[29919]: pgmap v373: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T12:35:22.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:22 smithi029 ceph-mon[41500]: pgmap v373: 148 pgs: 148 active+clean; 4.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-24T12:35:22.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:22.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:22.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:22.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:22.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:22.654 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:22.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:22.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:22.790 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:22.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:22.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:22.931 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:22.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:22.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:23.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:23.071 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:23.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:23.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:23.233 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:23.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:23.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:23.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:23.384 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:23.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:23.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:23.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:23.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:23.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:23.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:23.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:23.787 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:23.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:23.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:23.925 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:23.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:23.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:24.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:24.074 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:24.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:24.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:24.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:24.222 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:24.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:24.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:24.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:24.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:24.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:24 smithi190 ceph-mon[33748]: pgmap v375: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s 2024-03-24T12:35:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:24 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:35:24.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:24 smithi029 ceph-mon[41500]: pgmap v375: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s 2024-03-24T12:35:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:24 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:35:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:24 smithi029 ceph-mon[29919]: pgmap v375: 148 pgs: 148 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 925 op/s 2024-03-24T12:35:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:24 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:35:24.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:24.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:24.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:24.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:24.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:24.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:24.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:24.843 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:24.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:24.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:25.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:25.005 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:25.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:25.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:25.150 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:25.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:25.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:25 smithi029 ceph-mon[41500]: pgmap v376: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T12:35:25.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:25 smithi029 ceph-mon[29919]: pgmap v376: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T12:35:25.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:25.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:25.295 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:25.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:25 smithi190 ceph-mon[33748]: pgmap v376: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-24T12:35:25.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:25.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:25.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:25.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:25.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:25.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:25.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:25.709 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:25.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:25.793 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:35:25.793 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:35:25.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:25.846 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:25.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:25.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:25.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:25.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:25.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:26.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:26.115 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:26.135 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:26.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:26.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:26 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:35:26.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:26 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:35:26.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:26 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:35:26.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:26 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:35:26.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:26 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:35:26.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:26 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:35:26.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:26.271 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:26.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:26.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:26 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:35:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:26 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:35:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:26 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:35:26.406 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:26.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:26.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:26.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:26.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:35:26.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:26.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:26.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:26.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:26.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:26.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:26.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:26.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:26.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:26.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:26.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:26.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:26.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:27.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:27.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:27.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:27.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:27.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:27 smithi029 ceph-mon[41500]: pgmap v377: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T12:35:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:27 smithi029 ceph-mon[29919]: pgmap v377: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T12:35:27.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:27.261 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:27.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:27.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:27.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:27 smithi190 ceph-mon[33748]: pgmap v377: 148 pgs: 148 active+clean; 3.6 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T12:35:27.403 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:27.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:27.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:27.543 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:27.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:27.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:27.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:27.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:27.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:27.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:27.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:27.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:27.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:27.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:27.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:27.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:35:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:35:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:35:27.941 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:27.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:28.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:28.079 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:28.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:28.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:28.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:28.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:28.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:28.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:28.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:35:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:35:28.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:28.342 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:28.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:28.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:28.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:28.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:28.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:28.484 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:28.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:28.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:28.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T12:35:28.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:28.641 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:28.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:28.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:28.789 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:28.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:28.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:28.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:28.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:28.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:28.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:29.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:29.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:29.068 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:29.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:29.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:29.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:29.201 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:29.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:29.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:29.336 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:29 smithi029 ceph-mon[41500]: pgmap v378: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 960 op/s 2024-03-24T12:35:29.336 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:29 smithi029 ceph-mon[29919]: pgmap v378: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 960 op/s 2024-03-24T12:35:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:29.349 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:29.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:29.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:29 smithi190 ceph-mon[33748]: pgmap v378: 148 pgs: 148 active+clean; 3.4 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 960 op/s 2024-03-24T12:35:29.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:29.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:29.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:29.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:29.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:29.626 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:29.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:29.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:29.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:29.765 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:29.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:29.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:29.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:29.913 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:29.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:30.014 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:30.063 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:30.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:30.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:30.179 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:30.240 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:30.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:30.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:30.263 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:30.293 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:35:30.341 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:35:30.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:30.372 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:30.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:30.384 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:35:30.402 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:35:30.410 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:35:30.426 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:35:30.469 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:35:30.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:30.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:30.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:30.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:30.512 INFO:teuthology.orchestra.run.smithi029.stdout:osd.5: {} 2024-03-24T12:35:30.559 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:35:30.604 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:35:30.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:30.612 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:35:30.612 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:35:30.612 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:35:30.612 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:35:30.612 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:35:30.612 INFO:teuthology.orchestra.run.smithi029.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-24T12:35:30.613 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:35:30.613 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:35:30.656 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:30.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:30.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:30.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:30.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:30.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:30.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:30.941 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:30.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:31.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:31.075 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:31.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:31.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:31.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:31.214 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:31.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:31.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:31.281 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:31 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1333513668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:35:31.281 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:31 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1333513668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:35:31.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:31.328 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":378,"stamp":"2024-03-24T12:35:28.558632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3688438134,"num_objects":56471,"num_object_clones":0,"num_object_copies":169413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56471,"num_whiteouts":0,"num_read":1492,"num_read_kb":1338,"num_write":172058,"num_write_kb":7289310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6323,"num_bytes_recovered":414779306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86217,"ondisk_log_size":86217,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13245740,"kb_used_data":11042408,"kb_used_omap":12,"kb_used_meta":2202675,"kb_avail":736715476,"statfs":{"total":767960285184,"available":754396647424,"internally_reserved":0,"allocated":11307425792,"data_stored":11304241490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2255539761},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-624492544,"num_objects":-9529,"num_object_clones":0,"num_object_copies":-28587,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9529,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9529,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.927291"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":294,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.626951+0000","last_change":"2024-03-24T12:35:08.860005+0000","last_active":"2024-03-24T12:35:20.626951+0000","last_peered":"2024-03-24T12:35:20.626951+0000","last_clean":"2024-03-24T12:35:20.626951+0000","last_became_active":"2024-03-24T12:26:09.350521+0000","last_became_peered":"2024-03-24T12:26:09.350521+0000","last_unstale":"2024-03-24T12:35:20.626951+0000","last_undegraded":"2024-03-24T12:35:20.626951+0000","last_fullsized":"2024-03-24T12:35:20.626951+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:08.859869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.859869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.651960+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.636833+0000","last_change":"2024-03-24T12:34:48.745018+0000","last_active":"2024-03-24T12:35:20.636833+0000","last_peered":"2024-03-24T12:35:20.636833+0000","last_clean":"2024-03-24T12:35:20.636833+0000","last_became_active":"2024-03-24T12:26:07.335517+0000","last_became_peered":"2024-03-24T12:26:07.335517+0000","last_unstale":"2024-03-24T12:35:20.636833+0000","last_undegraded":"2024-03-24T12:35:20.636833+0000","last_fullsized":"2024-03-24T12:35:20.636833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:48.744937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.744937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.920329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.638387+0000","last_change":"2024-03-24T12:35:14.967127+0000","last_active":"2024-03-24T12:35:20.638387+0000","last_peered":"2024-03-24T12:35:20.638387+0000","last_clean":"2024-03-24T12:35:20.638387+0000","last_became_active":"2024-03-24T12:26:03.170822+0000","last_became_peered":"2024-03-24T12:26:03.170822+0000","last_unstale":"2024-03-24T12:35:20.638387+0000","last_undegraded":"2024-03-24T12:35:20.638387+0000","last_fullsized":"2024-03-24T12:35:20.638387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:14.966328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:14.966328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.789519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":310,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.626401+0000","last_change":"2024-03-24T12:33:12.325849+0000","last_active":"2024-03-24T12:35:20.626401+0000","last_peered":"2024-03-24T12:35:20.626401+0000","last_clean":"2024-03-24T12:35:20.626401+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:35:20.626401+0000","last_undegraded":"2024-03-24T12:35:20.626401+0000","last_fullsized":"2024-03-24T12:35:20.626401+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:12.325694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.325694+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":286,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641702+0000","last_change":"2024-03-24T12:35:07.031107+0000","last_active":"2024-03-24T12:35:20.641702+0000","last_peered":"2024-03-24T12:35:20.641702+0000","last_clean":"2024-03-24T12:35:20.641702+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:35:20.641702+0000","last_undegraded":"2024-03-24T12:35:20.641702+0000","last_fullsized":"2024-03-24T12:35:20.641702+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:07.030987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:07.030987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:21.246152+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.626307+0000","last_change":"2024-03-24T12:35:12.883693+0000","last_active":"2024-03-24T12:35:20.626307+0000","last_peered":"2024-03-24T12:35:20.626307+0000","last_clean":"2024-03-24T12:35:20.626307+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:35:20.626307+0000","last_undegraded":"2024-03-24T12:35:20.626307+0000","last_fullsized":"2024-03-24T12:35:20.626307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:12.883613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:12.883613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:38.551474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.640841+0000","last_change":"2024-03-24T12:35:02.993072+0000","last_active":"2024-03-24T12:35:20.640841+0000","last_peered":"2024-03-24T12:35:20.640841+0000","last_clean":"2024-03-24T12:35:20.640841+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:35:20.640841+0000","last_undegraded":"2024-03-24T12:35:20.640841+0000","last_fullsized":"2024-03-24T12:35:20.640841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:02.992900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.992900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:06.907402+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"90'22","reported_seq":326,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.640819+0000","last_change":"2024-03-24T12:33:36.263856+0000","last_active":"2024-03-24T12:35:20.640819+0000","last_peered":"2024-03-24T12:35:20.640819+0000","last_clean":"2024-03-24T12:35:20.640819+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:35:20.640819+0000","last_undegraded":"2024-03-24T12:35:20.640819+0000","last_fullsized":"2024-03-24T12:35:20.640819+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:33:36.263707+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.263707+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":294,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.637451+0000","last_change":"2024-03-24T12:34:58.789197+0000","last_active":"2024-03-24T12:35:20.637451+0000","last_peered":"2024-03-24T12:35:20.637451+0000","last_clean":"2024-03-24T12:35:20.637451+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:35:20.637451+0000","last_undegraded":"2024-03-24T12:35:20.637451+0000","last_fullsized":"2024-03-24T12:35:20.637451+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:58.789088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:34:58.789088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:11.317903+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":290,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634659+0000","last_change":"2024-03-24T12:34:44.161766+0000","last_active":"2024-03-24T12:35:20.634659+0000","last_peered":"2024-03-24T12:35:20.634659+0000","last_clean":"2024-03-24T12:35:20.634659+0000","last_became_active":"2024-03-24T12:26:07.331562+0000","last_became_peered":"2024-03-24T12:26:07.331562+0000","last_unstale":"2024-03-24T12:35:20.634659+0000","last_undegraded":"2024-03-24T12:35:20.634659+0000","last_fullsized":"2024-03-24T12:35:20.634659+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:44.161505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.161505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:11.680955+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634311+0000","last_change":"2024-03-24T12:33:33.178127+0000","last_active":"2024-03-24T12:35:20.634311+0000","last_peered":"2024-03-24T12:35:20.634311+0000","last_clean":"2024-03-24T12:35:20.634311+0000","last_became_active":"2024-03-24T12:26:03.161544+0000","last_became_peered":"2024-03-24T12:26:03.161544+0000","last_unstale":"2024-03-24T12:35:20.634311+0000","last_undegraded":"2024-03-24T12:35:20.634311+0000","last_fullsized":"2024-03-24T12:35:20.634311+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:33.177938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:33.177938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.036638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":310,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627224+0000","last_change":"2024-03-24T12:33:34.398616+0000","last_active":"2024-03-24T12:35:20.627224+0000","last_peered":"2024-03-24T12:35:20.627224+0000","last_clean":"2024-03-24T12:35:20.627224+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:35:20.627224+0000","last_undegraded":"2024-03-24T12:35:20.627224+0000","last_fullsized":"2024-03-24T12:35:20.627224+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:34.398401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.398401+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":286,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634103+0000","last_change":"2024-03-24T12:33:21.155477+0000","last_active":"2024-03-24T12:35:20.634103+0000","last_peered":"2024-03-24T12:35:20.634103+0000","last_clean":"2024-03-24T12:35:20.634103+0000","last_became_active":"2024-03-24T12:26:09.350721+0000","last_became_peered":"2024-03-24T12:26:09.350721+0000","last_unstale":"2024-03-24T12:35:20.634103+0000","last_undegraded":"2024-03-24T12:35:20.634103+0000","last_fullsized":"2024-03-24T12:35:20.634103+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.155345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.155345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.036654+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":290,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641634+0000","last_change":"2024-03-24T12:33:37.143601+0000","last_active":"2024-03-24T12:35:20.641634+0000","last_peered":"2024-03-24T12:35:20.641634+0000","last_clean":"2024-03-24T12:35:20.641634+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:35:20.641634+0000","last_undegraded":"2024-03-24T12:35:20.641634+0000","last_fullsized":"2024-03-24T12:35:20.641634+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:37.143487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:33:37.143487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634825+0000","last_change":"2024-03-24T12:33:23.095826+0000","last_active":"2024-03-24T12:35:20.634825+0000","last_peered":"2024-03-24T12:35:20.634825+0000","last_clean":"2024-03-24T12:35:20.634825+0000","last_became_active":"2024-03-24T12:26:03.159997+0000","last_became_peered":"2024-03-24T12:26:03.159997+0000","last_unstale":"2024-03-24T12:35:20.634825+0000","last_undegraded":"2024-03-24T12:35:20.634825+0000","last_fullsized":"2024-03-24T12:35:20.634825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:23.095706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:33:23.095706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.036687+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":312,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627065+0000","last_change":"2024-03-24T12:34:32.424665+0000","last_active":"2024-03-24T12:35:20.627065+0000","last_peered":"2024-03-24T12:35:20.627065+0000","last_clean":"2024-03-24T12:35:20.627065+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:35:20.627065+0000","last_undegraded":"2024-03-24T12:35:20.627065+0000","last_fullsized":"2024-03-24T12:35:20.627065+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T12:34:32.424537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:32.424537+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:00.556829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":295,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627984+0000","last_change":"2024-03-24T12:34:49.652294+0000","last_active":"2024-03-24T12:35:20.627984+0000","last_peered":"2024-03-24T12:35:20.627984+0000","last_clean":"2024-03-24T12:35:20.627984+0000","last_became_active":"2024-03-24T12:26:09.643671+0000","last_became_peered":"2024-03-24T12:26:09.643671+0000","last_unstale":"2024-03-24T12:35:20.627984+0000","last_undegraded":"2024-03-24T12:35:20.627984+0000","last_fullsized":"2024-03-24T12:35:20.627984+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:49.652214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:49.652214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:10.671588+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635850+0000","last_change":"2024-03-24T12:35:03.087600+0000","last_active":"2024-03-24T12:35:20.635850+0000","last_peered":"2024-03-24T12:35:20.635850+0000","last_clean":"2024-03-24T12:35:20.635850+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:35:20.635850+0000","last_undegraded":"2024-03-24T12:35:20.635850+0000","last_fullsized":"2024-03-24T12:35:20.635850+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:03.087476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:03.087476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:07.618666+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.639134+0000","last_change":"2024-03-24T12:34:46.648871+0000","last_active":"2024-03-24T12:35:20.639134+0000","last_peered":"2024-03-24T12:35:20.639134+0000","last_clean":"2024-03-24T12:35:20.639134+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:35:20.639134+0000","last_undegraded":"2024-03-24T12:35:20.639134+0000","last_fullsized":"2024-03-24T12:35:20.639134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:46.648735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:34:46.648735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:13.075407+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":327,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.632869+0000","last_change":"2024-03-24T12:35:04.954711+0000","last_active":"2024-03-24T12:35:20.632869+0000","last_peered":"2024-03-24T12:35:20.632869+0000","last_clean":"2024-03-24T12:35:20.632869+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:35:20.632869+0000","last_undegraded":"2024-03-24T12:35:20.632869+0000","last_fullsized":"2024-03-24T12:35:20.632869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:35:04.954560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.954560+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:26.969948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.643573+0000","last_change":"2024-03-24T12:34:51.927758+0000","last_active":"2024-03-24T12:35:20.643573+0000","last_peered":"2024-03-24T12:35:20.643573+0000","last_clean":"2024-03-24T12:35:20.643573+0000","last_became_active":"2024-03-24T12:26:09.643738+0000","last_became_peered":"2024-03-24T12:26:09.643738+0000","last_unstale":"2024-03-24T12:35:20.643573+0000","last_undegraded":"2024-03-24T12:35:20.643573+0000","last_fullsized":"2024-03-24T12:35:20.643573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:51.927644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:51.927644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:03.636261+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627883+0000","last_change":"2024-03-24T12:35:02.853101+0000","last_active":"2024-03-24T12:35:20.627883+0000","last_peered":"2024-03-24T12:35:20.627883+0000","last_clean":"2024-03-24T12:35:20.627883+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:35:20.627883+0000","last_undegraded":"2024-03-24T12:35:20.627883+0000","last_fullsized":"2024-03-24T12:35:20.627883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:02.852982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.852982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:16.016074+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627316+0000","last_change":"2024-03-24T12:33:18.258903+0000","last_active":"2024-03-24T12:35:20.627316+0000","last_peered":"2024-03-24T12:35:20.627316+0000","last_clean":"2024-03-24T12:35:20.627316+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:35:20.627316+0000","last_undegraded":"2024-03-24T12:35:20.627316+0000","last_fullsized":"2024-03-24T12:35:20.627316+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:18.258782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.258782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":309,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.640563+0000","last_change":"2024-03-24T12:33:07.191584+0000","last_active":"2024-03-24T12:35:20.640563+0000","last_peered":"2024-03-24T12:35:20.640563+0000","last_clean":"2024-03-24T12:35:20.640563+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:35:20.640563+0000","last_undegraded":"2024-03-24T12:35:20.640563+0000","last_fullsized":"2024-03-24T12:35:20.640563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:33:07.191431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.191431+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642312+0000","last_change":"2024-03-24T12:34:42.820621+0000","last_active":"2024-03-24T12:35:20.642312+0000","last_peered":"2024-03-24T12:35:20.642312+0000","last_clean":"2024-03-24T12:35:20.642312+0000","last_became_active":"2024-03-24T12:26:09.343786+0000","last_became_peered":"2024-03-24T12:26:09.343786+0000","last_unstale":"2024-03-24T12:35:20.642312+0000","last_undegraded":"2024-03-24T12:35:20.642312+0000","last_fullsized":"2024-03-24T12:35:20.642312+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:34:42.820487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:42.820487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:56.870887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":423,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.638365+0000","last_change":"2024-03-24T12:34:49.207292+0000","last_active":"2024-03-24T12:35:20.638365+0000","last_peered":"2024-03-24T12:35:20.638365+0000","last_clean":"2024-03-24T12:35:20.638365+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:35:20.638365+0000","last_undegraded":"2024-03-24T12:35:20.638365+0000","last_fullsized":"2024-03-24T12:35:20.638365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:34:49.207206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:49.207206+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:18.220803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":307,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.636732+0000","last_change":"2024-03-24T12:33:10.675622+0000","last_active":"2024-03-24T12:35:20.636732+0000","last_peered":"2024-03-24T12:35:20.636732+0000","last_clean":"2024-03-24T12:35:20.636732+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:35:20.636732+0000","last_undegraded":"2024-03-24T12:35:20.636732+0000","last_fullsized":"2024-03-24T12:35:20.636732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:10.675383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:10.675383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.106457+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":318,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642259+0000","last_change":"2024-03-24T12:35:05.755602+0000","last_active":"2024-03-24T12:35:20.642259+0000","last_peered":"2024-03-24T12:35:20.642259+0000","last_clean":"2024-03-24T12:35:20.642259+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:35:20.642259+0000","last_undegraded":"2024-03-24T12:35:20.642259+0000","last_fullsized":"2024-03-24T12:35:20.642259+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:35:05.755456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:05.755456+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:18.121034+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":294,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635989+0000","last_change":"2024-03-24T12:33:03.556099+0000","last_active":"2024-03-24T12:35:20.635989+0000","last_peered":"2024-03-24T12:35:20.635989+0000","last_clean":"2024-03-24T12:35:20.635989+0000","last_became_active":"2024-03-24T12:26:09.348355+0000","last_became_peered":"2024-03-24T12:26:09.348355+0000","last_unstale":"2024-03-24T12:35:20.635989+0000","last_undegraded":"2024-03-24T12:35:20.635989+0000","last_fullsized":"2024-03-24T12:35:20.635989+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:03.555975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.555975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.106483+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":283,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642486+0000","last_change":"2024-03-24T12:33:15.244131+0000","last_active":"2024-03-24T12:35:20.642486+0000","last_peered":"2024-03-24T12:35:20.642486+0000","last_clean":"2024-03-24T12:35:20.642486+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:35:20.642486+0000","last_undegraded":"2024-03-24T12:35:20.642486+0000","last_fullsized":"2024-03-24T12:35:20.642486+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:15.244006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:15.244006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.645881+0000","last_change":"2024-03-24T12:33:28.354892+0000","last_active":"2024-03-24T12:35:20.645881+0000","last_peered":"2024-03-24T12:35:20.645881+0000","last_clean":"2024-03-24T12:35:20.645881+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:35:20.645881+0000","last_undegraded":"2024-03-24T12:35:20.645881+0000","last_fullsized":"2024-03-24T12:35:20.645881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:28.354367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.354367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":322,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.638294+0000","last_change":"2024-03-24T12:34:43.138957+0000","last_active":"2024-03-24T12:35:20.638294+0000","last_peered":"2024-03-24T12:35:20.638294+0000","last_clean":"2024-03-24T12:35:20.638294+0000","last_became_active":"2024-03-24T12:26:05.317587+0000","last_became_peered":"2024-03-24T12:26:05.317587+0000","last_unstale":"2024-03-24T12:35:20.638294+0000","last_undegraded":"2024-03-24T12:35:20.638294+0000","last_fullsized":"2024-03-24T12:35:20.638294+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:34:43.138814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:43.138814+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.385797+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641595+0000","last_change":"2024-03-24T12:34:55.791634+0000","last_active":"2024-03-24T12:35:20.641595+0000","last_peered":"2024-03-24T12:35:20.641595+0000","last_clean":"2024-03-24T12:35:20.641595+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:35:20.641595+0000","last_undegraded":"2024-03-24T12:35:20.641595+0000","last_fullsized":"2024-03-24T12:35:20.641595+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:55.791517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:55.791517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:16.978653+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.636818+0000","last_change":"2024-03-24T12:34:34.270238+0000","last_active":"2024-03-24T12:35:20.636818+0000","last_peered":"2024-03-24T12:35:20.636818+0000","last_clean":"2024-03-24T12:35:20.636818+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:35:20.636818+0000","last_undegraded":"2024-03-24T12:35:20.636818+0000","last_fullsized":"2024-03-24T12:35:20.636818+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:34.270066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:34.270066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:47.337748+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.638536+0000","last_change":"2024-03-24T12:35:04.720760+0000","last_active":"2024-03-24T12:35:20.638536+0000","last_peered":"2024-03-24T12:35:20.638536+0000","last_clean":"2024-03-24T12:35:20.638536+0000","last_became_active":"2024-03-24T12:26:03.170395+0000","last_became_peered":"2024-03-24T12:26:03.170395+0000","last_unstale":"2024-03-24T12:35:20.638536+0000","last_undegraded":"2024-03-24T12:35:20.638536+0000","last_fullsized":"2024-03-24T12:35:20.638536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.720633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.720633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:28.770573+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":301,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.646011+0000","last_change":"2024-03-24T12:33:27.357494+0000","last_active":"2024-03-24T12:35:20.646011+0000","last_peered":"2024-03-24T12:35:20.646011+0000","last_clean":"2024-03-24T12:35:20.646011+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:35:20.646011+0000","last_undegraded":"2024-03-24T12:35:20.646011+0000","last_fullsized":"2024-03-24T12:35:20.646011+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:33:27.355578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:27.355578+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:28.942213+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":286,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641892+0000","last_change":"2024-03-24T12:34:57.998315+0000","last_active":"2024-03-24T12:35:20.641892+0000","last_peered":"2024-03-24T12:35:20.641892+0000","last_clean":"2024-03-24T12:35:20.641892+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:35:20.641892+0000","last_undegraded":"2024-03-24T12:35:20.641892+0000","last_fullsized":"2024-03-24T12:35:20.641892+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:57.998162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:34:57.998162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.432696+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641860+0000","last_change":"2024-03-24T12:35:12.719879+0000","last_active":"2024-03-24T12:35:20.641860+0000","last_peered":"2024-03-24T12:35:20.641860+0000","last_clean":"2024-03-24T12:35:20.641860+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:35:20.641860+0000","last_undegraded":"2024-03-24T12:35:20.641860+0000","last_fullsized":"2024-03-24T12:35:20.641860+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:12.719663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:12.719663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.187535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":304,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635153+0000","last_change":"2024-03-24T12:33:28.059411+0000","last_active":"2024-03-24T12:35:20.635153+0000","last_peered":"2024-03-24T12:35:20.635153+0000","last_clean":"2024-03-24T12:35:20.635153+0000","last_became_active":"2024-03-24T12:26:03.165083+0000","last_became_peered":"2024-03-24T12:26:03.165083+0000","last_unstale":"2024-03-24T12:35:20.635153+0000","last_undegraded":"2024-03-24T12:35:20.635153+0000","last_fullsized":"2024-03-24T12:35:20.635153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:28.059288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.059288+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.036705+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":313,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.640724+0000","last_change":"2024-03-24T12:34:54.934073+0000","last_active":"2024-03-24T12:35:20.640724+0000","last_peered":"2024-03-24T12:35:20.640724+0000","last_clean":"2024-03-24T12:35:20.640724+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:35:20.640724+0000","last_undegraded":"2024-03-24T12:35:20.640724+0000","last_fullsized":"2024-03-24T12:35:20.640724+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:34:54.933925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.933925+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:09.321569+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.637248+0000","last_change":"2024-03-24T12:34:53.896024+0000","last_active":"2024-03-24T12:35:20.637248+0000","last_peered":"2024-03-24T12:35:20.637248+0000","last_clean":"2024-03-24T12:35:20.637248+0000","last_became_active":"2024-03-24T12:26:09.350656+0000","last_became_peered":"2024-03-24T12:26:09.350656+0000","last_unstale":"2024-03-24T12:35:20.637248+0000","last_undegraded":"2024-03-24T12:35:20.637248+0000","last_fullsized":"2024-03-24T12:35:20.637248+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:53.895902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.895902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:21.860444+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":290,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.640972+0000","last_change":"2024-03-24T12:33:31.383451+0000","last_active":"2024-03-24T12:35:20.640972+0000","last_peered":"2024-03-24T12:35:20.640972+0000","last_clean":"2024-03-24T12:35:20.640972+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:35:20.640972+0000","last_undegraded":"2024-03-24T12:35:20.640972+0000","last_fullsized":"2024-03-24T12:35:20.640972+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_clean_scrub_stamp":"2024-03-24T12:33:31.383330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":306,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642433+0000","last_change":"2024-03-24T12:35:10.729253+0000","last_active":"2024-03-24T12:35:20.642433+0000","last_peered":"2024-03-24T12:35:20.642433+0000","last_clean":"2024-03-24T12:35:20.642433+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:35:20.642433+0000","last_undegraded":"2024-03-24T12:35:20.642433+0000","last_fullsized":"2024-03-24T12:35:20.642433+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.729138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.729138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:30.421888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":288,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634079+0000","last_change":"2024-03-24T12:35:02.977053+0000","last_active":"2024-03-24T12:35:20.634079+0000","last_peered":"2024-03-24T12:35:20.634079+0000","last_clean":"2024-03-24T12:35:20.634079+0000","last_became_active":"2024-03-24T12:26:16.874170+0000","last_became_peered":"2024-03-24T12:26:16.874170+0000","last_unstale":"2024-03-24T12:35:20.634079+0000","last_undegraded":"2024-03-24T12:35:20.634079+0000","last_fullsized":"2024-03-24T12:35:20.634079+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:35:02.976915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.976915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.099485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":286,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.640404+0000","last_change":"2024-03-24T12:34:47.899102+0000","last_active":"2024-03-24T12:35:20.640404+0000","last_peered":"2024-03-24T12:35:20.640404+0000","last_clean":"2024-03-24T12:35:20.640404+0000","last_became_active":"2024-03-24T12:26:09.343296+0000","last_became_peered":"2024-03-24T12:26:09.343296+0000","last_unstale":"2024-03-24T12:35:20.640404+0000","last_undegraded":"2024-03-24T12:35:20.640404+0000","last_fullsized":"2024-03-24T12:35:20.640404+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.898988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.898988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:03.389230+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635465+0000","last_change":"2024-03-24T12:33:21.145032+0000","last_active":"2024-03-24T12:35:20.635465+0000","last_peered":"2024-03-24T12:35:20.635465+0000","last_clean":"2024-03-24T12:35:20.635465+0000","last_became_active":"2024-03-24T12:26:07.330811+0000","last_became_peered":"2024-03-24T12:26:07.330811+0000","last_unstale":"2024-03-24T12:35:20.635465+0000","last_undegraded":"2024-03-24T12:35:20.635465+0000","last_fullsized":"2024-03-24T12:35:20.635465+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.144883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.144883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.106509+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":307,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627525+0000","last_change":"2024-03-24T12:33:17.214730+0000","last_active":"2024-03-24T12:35:20.627525+0000","last_peered":"2024-03-24T12:35:20.627525+0000","last_clean":"2024-03-24T12:35:20.627525+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:35:20.627525+0000","last_undegraded":"2024-03-24T12:35:20.627525+0000","last_fullsized":"2024-03-24T12:35:20.627525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:17.214655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:17.214655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"88'16","reported_seq":335,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627456+0000","last_change":"2024-03-24T12:35:04.793820+0000","last_active":"2024-03-24T12:35:20.627456+0000","last_peered":"2024-03-24T12:35:20.627456+0000","last_clean":"2024-03-24T12:35:20.627456+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:35:20.627456+0000","last_undegraded":"2024-03-24T12:35:20.627456+0000","last_fullsized":"2024-03-24T12:35:20.627456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:35:04.793689+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.793689+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:15.328478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641867+0000","last_change":"2024-03-24T12:34:57.796652+0000","last_active":"2024-03-24T12:35:20.641867+0000","last_peered":"2024-03-24T12:35:20.641867+0000","last_clean":"2024-03-24T12:35:20.641867+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:35:20.641867+0000","last_undegraded":"2024-03-24T12:35:20.641867+0000","last_fullsized":"2024-03-24T12:35:20.641867+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:34:57.796520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:57.796520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:23.345910+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":411,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634505+0000","last_change":"2024-03-24T12:35:09.007633+0000","last_active":"2024-03-24T12:35:20.634505+0000","last_peered":"2024-03-24T12:35:20.634505+0000","last_clean":"2024-03-24T12:35:20.634505+0000","last_became_active":"2024-03-24T12:26:07.329487+0000","last_became_peered":"2024-03-24T12:26:07.329487+0000","last_unstale":"2024-03-24T12:35:20.634505+0000","last_undegraded":"2024-03-24T12:35:20.634505+0000","last_fullsized":"2024-03-24T12:35:20.634505+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:35:09.007492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:09.007492+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.347607+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642054+0000","last_change":"2024-03-24T12:34:54.776782+0000","last_active":"2024-03-24T12:35:20.642054+0000","last_peered":"2024-03-24T12:35:20.642054+0000","last_clean":"2024-03-24T12:35:20.642054+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:35:20.642054+0000","last_undegraded":"2024-03-24T12:35:20.642054+0000","last_fullsized":"2024-03-24T12:35:20.642054+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:54.776673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.776673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:06.865253+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":309,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641445+0000","last_change":"2024-03-24T12:34:52.887121+0000","last_active":"2024-03-24T12:35:20.641445+0000","last_peered":"2024-03-24T12:35:20.641445+0000","last_clean":"2024-03-24T12:35:20.641445+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:35:20.641445+0000","last_undegraded":"2024-03-24T12:35:20.641445+0000","last_fullsized":"2024-03-24T12:35:20.641445+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:34:52.886768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:52.886768+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:55.685280+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":294,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627564+0000","last_change":"2024-03-24T12:35:01.913414+0000","last_active":"2024-03-24T12:35:20.627564+0000","last_peered":"2024-03-24T12:35:20.627564+0000","last_clean":"2024-03-24T12:35:20.627564+0000","last_became_active":"2024-03-24T12:26:09.346062+0000","last_became_peered":"2024-03-24T12:26:09.346062+0000","last_unstale":"2024-03-24T12:35:20.627564+0000","last_undegraded":"2024-03-24T12:35:20.627564+0000","last_fullsized":"2024-03-24T12:35:20.627564+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:01.913287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:01.913287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.142232+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"93'9","reported_seq":404,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641995+0000","last_change":"2024-03-24T12:33:06.165605+0000","last_active":"2024-03-24T12:35:20.641995+0000","last_peered":"2024-03-24T12:35:20.641995+0000","last_clean":"2024-03-24T12:35:20.641995+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:35:20.641995+0000","last_undegraded":"2024-03-24T12:35:20.641995+0000","last_fullsized":"2024-03-24T12:35:20.641995+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'9","last_scrub_stamp":"2024-03-24T12:33:06.165466+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:33:06.165466+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":298,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.637827+0000","last_change":"2024-03-24T12:33:26.284194+0000","last_active":"2024-03-24T12:35:20.637827+0000","last_peered":"2024-03-24T12:35:20.637827+0000","last_clean":"2024-03-24T12:35:20.637827+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:35:20.637827+0000","last_undegraded":"2024-03-24T12:35:20.637827+0000","last_fullsized":"2024-03-24T12:35:20.637827+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:26.283885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:26.283885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'13","reported_seq":322,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634715+0000","last_change":"2024-03-24T12:35:08.002157+0000","last_active":"2024-03-24T12:35:20.634715+0000","last_peered":"2024-03-24T12:35:20.634715+0000","last_clean":"2024-03-24T12:35:20.634715+0000","last_became_active":"2024-03-24T12:26:05.358213+0000","last_became_peered":"2024-03-24T12:26:05.358213+0000","last_unstale":"2024-03-24T12:35:20.634715+0000","last_undegraded":"2024-03-24T12:35:20.634715+0000","last_fullsized":"2024-03-24T12:35:20.634715+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:35:08.002032+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.002032+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:17.551062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":294,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.633018+0000","last_change":"2024-03-24T12:34:54.995904+0000","last_active":"2024-03-24T12:35:20.633018+0000","last_peered":"2024-03-24T12:35:20.633018+0000","last_clean":"2024-03-24T12:35:20.633018+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:35:20.633018+0000","last_undegraded":"2024-03-24T12:35:20.633018+0000","last_fullsized":"2024-03-24T12:35:20.633018+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:54.995668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.995668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:21.864194+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":282,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.643573+0000","last_change":"2024-03-24T12:33:18.355672+0000","last_active":"2024-03-24T12:35:20.643573+0000","last_peered":"2024-03-24T12:35:20.643573+0000","last_clean":"2024-03-24T12:35:20.643573+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:35:20.643573+0000","last_undegraded":"2024-03-24T12:35:20.643573+0000","last_fullsized":"2024-03-24T12:35:20.643573+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:18.355449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.355449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":313,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641728+0000","last_change":"2024-03-24T12:34:50.800957+0000","last_active":"2024-03-24T12:35:20.641728+0000","last_peered":"2024-03-24T12:35:20.641728+0000","last_clean":"2024-03-24T12:35:20.641728+0000","last_became_active":"2024-03-24T12:32:43.256833+0000","last_became_peered":"2024-03-24T12:32:43.256833+0000","last_unstale":"2024-03-24T12:35:20.641728+0000","last_undegraded":"2024-03-24T12:35:20.641728+0000","last_fullsized":"2024-03-24T12:35:20.641728+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:50.800852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:50.800852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:20.220743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":310,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642346+0000","last_change":"2024-03-24T12:33:18.443409+0000","last_active":"2024-03-24T12:35:20.642346+0000","last_peered":"2024-03-24T12:35:20.642346+0000","last_clean":"2024-03-24T12:35:20.642346+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:35:20.642346+0000","last_undegraded":"2024-03-24T12:35:20.642346+0000","last_fullsized":"2024-03-24T12:35:20.642346+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:18.443276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.443276+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":286,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642212+0000","last_change":"2024-03-24T12:35:04.719690+0000","last_active":"2024-03-24T12:35:20.642212+0000","last_peered":"2024-03-24T12:35:20.642212+0000","last_clean":"2024-03-24T12:35:20.642212+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:35:20.642212+0000","last_undegraded":"2024-03-24T12:35:20.642212+0000","last_fullsized":"2024-03-24T12:35:20.642212+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.719553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.719553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:30.298796+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":290,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635135+0000","last_change":"2024-03-24T12:35:10.995737+0000","last_active":"2024-03-24T12:35:20.635135+0000","last_peered":"2024-03-24T12:35:20.635135+0000","last_clean":"2024-03-24T12:35:20.635135+0000","last_became_active":"2024-03-24T12:26:07.329117+0000","last_became_peered":"2024-03-24T12:26:07.329117+0000","last_unstale":"2024-03-24T12:35:20.635135+0000","last_undegraded":"2024-03-24T12:35:20.635135+0000","last_fullsized":"2024-03-24T12:35:20.635135+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.995618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.995618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:39.944517+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":307,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.636475+0000","last_change":"2024-03-24T12:34:47.189025+0000","last_active":"2024-03-24T12:35:20.636475+0000","last_peered":"2024-03-24T12:35:20.636475+0000","last_clean":"2024-03-24T12:35:20.636475+0000","last_became_active":"2024-03-24T12:26:03.161414+0000","last_became_peered":"2024-03-24T12:26:03.161414+0000","last_unstale":"2024-03-24T12:35:20.636475+0000","last_undegraded":"2024-03-24T12:35:20.636475+0000","last_fullsized":"2024-03-24T12:35:20.636475+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.188916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.188916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:49.981138+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":303,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.643663+0000","last_change":"2024-03-24T12:35:15.907723+0000","last_active":"2024-03-24T12:35:20.643663+0000","last_peered":"2024-03-24T12:35:20.643663+0000","last_clean":"2024-03-24T12:35:20.643663+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:35:20.643663+0000","last_undegraded":"2024-03-24T12:35:20.643663+0000","last_fullsized":"2024-03-24T12:35:20.643663+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T12:35:15.907580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.907580+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.415309+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"197'11046","reported_seq":11210,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:27.578517+0000","last_change":"2024-03-24T12:34:34.953934+0000","last_active":"2024-03-24T12:35:27.578517+0000","last_peered":"2024-03-24T12:35:27.578517+0000","last_clean":"2024-03-24T12:35:27.578517+0000","last_became_active":"2024-03-24T12:32:41.565218+0000","last_became_peered":"2024-03-24T12:32:41.565218+0000","last_unstale":"2024-03-24T12:35:27.578517+0000","last_undegraded":"2024-03-24T12:35:27.578517+0000","last_fullsized":"2024-03-24T12:35:27.578517+0000","mapping_epoch":158,"log_start":"183'5700","ondisk_log_start":"183'5700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'7823","last_scrub_stamp":"2024-03-24T12:34:34.953754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:34.953754+0000","objects_scrubbed":6714,"log_size":5346,"log_dups_size":0,"ondisk_log_size":5346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:35.466489+0000","scrub_duration":1748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224395264,"num_objects":3424,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11046,"num_write_kb":463040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634604+0000","last_change":"2024-03-24T12:33:34.170601+0000","last_active":"2024-03-24T12:35:20.634604+0000","last_peered":"2024-03-24T12:35:20.634604+0000","last_clean":"2024-03-24T12:35:20.634604+0000","last_became_active":"2024-03-24T12:26:09.643602+0000","last_became_peered":"2024-03-24T12:26:09.643602+0000","last_unstale":"2024-03-24T12:35:20.634604+0000","last_undegraded":"2024-03-24T12:35:20.634604+0000","last_fullsized":"2024-03-24T12:35:20.634604+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.170473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.170473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.036720+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":311,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642338+0000","last_change":"2024-03-24T12:34:40.851414+0000","last_active":"2024-03-24T12:35:20.642338+0000","last_peered":"2024-03-24T12:35:20.642338+0000","last_clean":"2024-03-24T12:35:20.642338+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:35:20.642338+0000","last_undegraded":"2024-03-24T12:35:20.642338+0000","last_fullsized":"2024-03-24T12:35:20.642338+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:34:40.851280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:40.851280+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:07.220024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":320,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634103+0000","last_change":"2024-03-24T12:33:26.095777+0000","last_active":"2024-03-24T12:35:20.634103+0000","last_peered":"2024-03-24T12:35:20.634103+0000","last_clean":"2024-03-24T12:35:20.634103+0000","last_became_active":"2024-03-24T12:26:05.358217+0000","last_became_peered":"2024-03-24T12:26:05.358217+0000","last_unstale":"2024-03-24T12:35:20.634103+0000","last_undegraded":"2024-03-24T12:35:20.634103+0000","last_fullsized":"2024-03-24T12:35:20.634103+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:33:26.094785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:26.094785+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.036672+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":423,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641882+0000","last_change":"2024-03-24T12:34:53.783253+0000","last_active":"2024-03-24T12:35:20.641882+0000","last_peered":"2024-03-24T12:35:20.641882+0000","last_clean":"2024-03-24T12:35:20.641882+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:35:20.641882+0000","last_undegraded":"2024-03-24T12:35:20.641882+0000","last_fullsized":"2024-03-24T12:35:20.641882+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:34:53.783165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.783165+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:15.706424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"197'10689","reported_seq":10847,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:27.503985+0000","last_change":"2024-03-24T12:34:42.447980+0000","last_active":"2024-03-24T12:35:27.503985+0000","last_peered":"2024-03-24T12:35:27.503985+0000","last_clean":"2024-03-24T12:35:27.503985+0000","last_became_active":"2024-03-24T12:32:41.045244+0000","last_became_peered":"2024-03-24T12:32:41.045244+0000","last_unstale":"2024-03-24T12:35:27.503985+0000","last_undegraded":"2024-03-24T12:35:27.503985+0000","last_fullsized":"2024-03-24T12:35:27.503985+0000","mapping_epoch":158,"log_start":"175'4300","ondisk_log_start":"175'4300","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'7895","last_scrub_stamp":"2024-03-24T12:34:42.447811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:42.447811+0000","objects_scrubbed":6138,"log_size":6389,"log_dups_size":0,"ondisk_log_size":6389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:59.127848+0000","scrub_duration":1249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217251840,"num_objects":3315,"num_object_clones":0,"num_object_copies":9945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10689,"num_write_kb":448128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":279,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641802+0000","last_change":"2024-03-24T12:33:20.675885+0000","last_active":"2024-03-24T12:35:20.641802+0000","last_peered":"2024-03-24T12:35:20.641802+0000","last_clean":"2024-03-24T12:35:20.641802+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:35:20.641802+0000","last_undegraded":"2024-03-24T12:35:20.641802+0000","last_fullsized":"2024-03-24T12:35:20.641802+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:20.675649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:20.675649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.643835+0000","last_change":"2024-03-24T12:35:02.723283+0000","last_active":"2024-03-24T12:35:20.643835+0000","last_peered":"2024-03-24T12:35:20.643835+0000","last_clean":"2024-03-24T12:35:20.643835+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:35:20.643835+0000","last_undegraded":"2024-03-24T12:35:20.643835+0000","last_fullsized":"2024-03-24T12:35:20.643835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:02.723170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.723170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:26.611016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"140'17","reported_seq":331,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.626745+0000","last_change":"2024-03-24T12:33:30.333141+0000","last_active":"2024-03-24T12:35:20.626745+0000","last_peered":"2024-03-24T12:35:20.626745+0000","last_clean":"2024-03-24T12:35:20.626745+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:35:20.626745+0000","last_undegraded":"2024-03-24T12:35:20.626745+0000","last_fullsized":"2024-03-24T12:35:20.626745+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'17","last_scrub_stamp":"2024-03-24T12:33:30.332974+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:33:30.332974+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.631875+0000","last_change":"2024-03-24T12:35:11.997564+0000","last_active":"2024-03-24T12:35:20.631875+0000","last_peered":"2024-03-24T12:35:20.631875+0000","last_clean":"2024-03-24T12:35:20.631875+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:35:20.631875+0000","last_undegraded":"2024-03-24T12:35:20.631875+0000","last_fullsized":"2024-03-24T12:35:20.631875+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:11.997339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:11.997339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.100592+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"197'10833","reported_seq":10993,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:25.926484+0000","last_change":"2024-03-24T12:34:41.649606+0000","last_active":"2024-03-24T12:35:25.926484+0000","last_peered":"2024-03-24T12:35:25.926484+0000","last_clean":"2024-03-24T12:35:25.926484+0000","last_became_active":"2024-03-24T12:32:41.041303+0000","last_became_peered":"2024-03-24T12:32:41.041303+0000","last_unstale":"2024-03-24T12:35:25.926484+0000","last_undegraded":"2024-03-24T12:35:25.926484+0000","last_fullsized":"2024-03-24T12:35:25.926484+0000","mapping_epoch":158,"log_start":"179'5000","ondisk_log_start":"179'5000","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'8059","last_scrub_stamp":"2024-03-24T12:34:41.649439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:41.649439+0000","objects_scrubbed":6312,"log_size":5833,"log_dups_size":0,"ondisk_log_size":5833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:00.198020+0000","scrub_duration":3527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224854016,"num_objects":3431,"num_object_clones":0,"num_object_copies":10293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10833,"num_write_kb":456448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641509+0000","last_change":"2024-03-24T12:34:45.838849+0000","last_active":"2024-03-24T12:35:20.641509+0000","last_peered":"2024-03-24T12:35:20.641509+0000","last_clean":"2024-03-24T12:35:20.641509+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:35:20.641509+0000","last_undegraded":"2024-03-24T12:35:20.641509+0000","last_fullsized":"2024-03-24T12:35:20.641509+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:45.838705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:45.838705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:02.368329+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.630251+0000","last_change":"2024-03-24T12:35:16.019453+0000","last_active":"2024-03-24T12:35:20.630251+0000","last_peered":"2024-03-24T12:35:20.630251+0000","last_clean":"2024-03-24T12:35:20.630251+0000","last_became_active":"2024-03-24T12:26:07.331416+0000","last_became_peered":"2024-03-24T12:26:07.331416+0000","last_unstale":"2024-03-24T12:35:20.630251+0000","last_undegraded":"2024-03-24T12:35:20.630251+0000","last_fullsized":"2024-03-24T12:35:20.630251+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:16.019255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.019255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:22.005776+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642617+0000","last_change":"2024-03-24T12:33:34.204979+0000","last_active":"2024-03-24T12:35:20.642617+0000","last_peered":"2024-03-24T12:35:20.642617+0000","last_clean":"2024-03-24T12:35:20.642617+0000","last_became_active":"2024-03-24T12:26:03.165214+0000","last_became_peered":"2024-03-24T12:26:03.165214+0000","last_unstale":"2024-03-24T12:35:20.642617+0000","last_undegraded":"2024-03-24T12:35:20.642617+0000","last_fullsized":"2024-03-24T12:35:20.642617+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.204829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":334,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.636842+0000","last_change":"2024-03-24T12:34:51.459238+0000","last_active":"2024-03-24T12:35:20.636842+0000","last_peered":"2024-03-24T12:35:20.636842+0000","last_clean":"2024-03-24T12:35:20.636842+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:35:20.636842+0000","last_undegraded":"2024-03-24T12:35:20.636842+0000","last_fullsized":"2024-03-24T12:35:20.636842+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:34:51.459093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:51.459093+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:54.535337+0000","scrub_duration":248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"197'10667","reported_seq":10827,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:23.551351+0000","last_change":"2024-03-24T12:34:40.602767+0000","last_active":"2024-03-24T12:35:23.551351+0000","last_peered":"2024-03-24T12:35:23.551351+0000","last_clean":"2024-03-24T12:35:23.551351+0000","last_became_active":"2024-03-24T12:32:41.042042+0000","last_became_peered":"2024-03-24T12:32:41.042042+0000","last_unstale":"2024-03-24T12:35:23.551351+0000","last_undegraded":"2024-03-24T12:35:23.551351+0000","last_fullsized":"2024-03-24T12:35:23.551351+0000","mapping_epoch":158,"log_start":"183'5700","ondisk_log_start":"183'5700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'8067","last_scrub_stamp":"2024-03-24T12:34:40.602560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:40.602560+0000","objects_scrubbed":6331,"log_size":4967,"log_dups_size":0,"ondisk_log_size":4967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:09.923583+0000","scrub_duration":3051,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240844800,"num_objects":3675,"num_object_clones":0,"num_object_copies":11025,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3675,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10667,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.639660+0000","last_change":"2024-03-24T12:35:09.948925+0000","last_active":"2024-03-24T12:35:20.639660+0000","last_peered":"2024-03-24T12:35:20.639660+0000","last_clean":"2024-03-24T12:35:20.639660+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:35:20.639660+0000","last_undegraded":"2024-03-24T12:35:20.639660+0000","last_fullsized":"2024-03-24T12:35:20.639660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:09.948402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:09.948402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.433285+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.631613+0000","last_change":"2024-03-24T12:35:10.999806+0000","last_active":"2024-03-24T12:35:20.631613+0000","last_peered":"2024-03-24T12:35:20.631613+0000","last_clean":"2024-03-24T12:35:20.631613+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:35:20.631613+0000","last_undegraded":"2024-03-24T12:35:20.631613+0000","last_fullsized":"2024-03-24T12:35:20.631613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.999487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.999487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:39.057856+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":314,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634921+0000","last_change":"2024-03-24T12:35:15.994912+0000","last_active":"2024-03-24T12:35:20.634921+0000","last_peered":"2024-03-24T12:35:20.634921+0000","last_clean":"2024-03-24T12:35:20.634921+0000","last_became_active":"2024-03-24T12:26:03.162562+0000","last_became_peered":"2024-03-24T12:26:03.162562+0000","last_unstale":"2024-03-24T12:35:20.634921+0000","last_undegraded":"2024-03-24T12:35:20.634921+0000","last_fullsized":"2024-03-24T12:35:20.634921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:35:15.994784+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.994784+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:16.011588+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":319,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.638919+0000","last_change":"2024-03-24T12:33:17.435894+0000","last_active":"2024-03-24T12:35:20.638919+0000","last_peered":"2024-03-24T12:35:20.638919+0000","last_clean":"2024-03-24T12:35:20.638919+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:35:20.638919+0000","last_undegraded":"2024-03-24T12:35:20.638919+0000","last_fullsized":"2024-03-24T12:35:20.638919+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:17.435756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:17.435756+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"197'10812","reported_seq":10972,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:27.515692+0000","last_change":"2024-03-24T12:34:38.651157+0000","last_active":"2024-03-24T12:35:27.515692+0000","last_peered":"2024-03-24T12:35:27.515692+0000","last_clean":"2024-03-24T12:35:27.515692+0000","last_became_active":"2024-03-24T12:32:41.043709+0000","last_became_peered":"2024-03-24T12:32:41.043709+0000","last_unstale":"2024-03-24T12:35:27.515692+0000","last_undegraded":"2024-03-24T12:35:27.515692+0000","last_fullsized":"2024-03-24T12:35:27.515692+0000","mapping_epoch":158,"log_start":"182'5500","ondisk_log_start":"182'5500","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'7870","last_scrub_stamp":"2024-03-24T12:34:38.651008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:38.651008+0000","objects_scrubbed":6332,"log_size":5312,"log_dups_size":0,"ondisk_log_size":5312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:56.743064+0000","scrub_duration":2447,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":217710592,"num_objects":3322,"num_object_clones":0,"num_object_copies":9966,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3322,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10812,"num_write_kb":452288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641035+0000","last_change":"2024-03-24T12:35:07.993235+0000","last_active":"2024-03-24T12:35:20.641035+0000","last_peered":"2024-03-24T12:35:20.641035+0000","last_clean":"2024-03-24T12:35:20.641035+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:35:20.641035+0000","last_undegraded":"2024-03-24T12:35:20.641035+0000","last_fullsized":"2024-03-24T12:35:20.641035+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:07.993108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:35:07.993108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:18.965706+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":308,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634382+0000","last_change":"2024-03-24T12:34:53.972192+0000","last_active":"2024-03-24T12:35:20.634382+0000","last_peered":"2024-03-24T12:35:20.634382+0000","last_clean":"2024-03-24T12:35:20.634382+0000","last_became_active":"2024-03-24T12:26:03.165771+0000","last_became_peered":"2024-03-24T12:26:03.165771+0000","last_unstale":"2024-03-24T12:35:20.634382+0000","last_undegraded":"2024-03-24T12:35:20.634382+0000","last_fullsized":"2024-03-24T12:35:20.634382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:34:53.972073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.972073+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:55.404283+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":307,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627159+0000","last_change":"2024-03-24T12:34:55.729278+0000","last_active":"2024-03-24T12:35:20.627159+0000","last_peered":"2024-03-24T12:35:20.627159+0000","last_clean":"2024-03-24T12:35:20.627159+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:35:20.627159+0000","last_undegraded":"2024-03-24T12:35:20.627159+0000","last_fullsized":"2024-03-24T12:35:20.627159+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:55.729129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:55.729129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:22.046705+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"90'38","reported_seq":466,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635265+0000","last_change":"2024-03-24T12:32:55.403983+0000","last_active":"2024-03-24T12:35:20.635265+0000","last_peered":"2024-03-24T12:35:20.635265+0000","last_clean":"2024-03-24T12:35:20.635265+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:35:20.635265+0000","last_undegraded":"2024-03-24T12:35:20.635265+0000","last_fullsized":"2024-03-24T12:35:20.635265+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'38","last_scrub_stamp":"2024-03-24T12:32:55.403873+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.403873+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.106424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":291,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.638055+0000","last_change":"2024-03-24T12:34:43.416737+0000","last_active":"2024-03-24T12:35:20.638055+0000","last_peered":"2024-03-24T12:35:20.638055+0000","last_clean":"2024-03-24T12:35:20.638055+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:35:20.638055+0000","last_undegraded":"2024-03-24T12:35:20.638055+0000","last_fullsized":"2024-03-24T12:35:20.638055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:43.416627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:43.416627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:07.477996+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"197'10553","reported_seq":10704,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:24.254703+0000","last_change":"2024-03-24T12:35:03.249200+0000","last_active":"2024-03-24T12:35:24.254703+0000","last_peered":"2024-03-24T12:35:24.254703+0000","last_clean":"2024-03-24T12:35:24.254703+0000","last_became_active":"2024-03-24T12:32:41.045113+0000","last_became_peered":"2024-03-24T12:32:41.045113+0000","last_unstale":"2024-03-24T12:35:24.254703+0000","last_undegraded":"2024-03-24T12:35:24.254703+0000","last_fullsized":"2024-03-24T12:35:24.254703+0000","mapping_epoch":158,"log_start":"183'5700","ondisk_log_start":"183'5700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'9142","last_scrub_stamp":"2024-03-24T12:35:03.248772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:35:03.248772+0000","objects_scrubbed":5152,"log_size":4853,"log_dups_size":0,"ondisk_log_size":4853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:29.461790+0000","scrub_duration":2292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240975872,"num_objects":3677,"num_object_clones":0,"num_object_copies":11031,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3677,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10553,"num_write_kb":455360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":295,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642687+0000","last_change":"2024-03-24T12:35:17.750703+0000","last_active":"2024-03-24T12:35:20.642687+0000","last_peered":"2024-03-24T12:35:20.642687+0000","last_clean":"2024-03-24T12:35:20.642687+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:35:20.642687+0000","last_undegraded":"2024-03-24T12:35:20.642687+0000","last_fullsized":"2024-03-24T12:35:20.642687+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:17.750601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:17.750601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:23.838416+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":307,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634562+0000","last_change":"2024-03-24T12:33:19.170350+0000","last_active":"2024-03-24T12:35:20.634562+0000","last_peered":"2024-03-24T12:35:20.634562+0000","last_clean":"2024-03-24T12:35:20.634562+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:35:20.634562+0000","last_undegraded":"2024-03-24T12:35:20.634562+0000","last_fullsized":"2024-03-24T12:35:20.634562+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:19.170147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.170147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.106595+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":314,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.630345+0000","last_change":"2024-03-24T12:34:53.064043+0000","last_active":"2024-03-24T12:35:20.630345+0000","last_peered":"2024-03-24T12:35:20.630345+0000","last_clean":"2024-03-24T12:35:20.630345+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:35:20.630345+0000","last_undegraded":"2024-03-24T12:35:20.630345+0000","last_fullsized":"2024-03-24T12:35:20.630345+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:53.063885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.063885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:00.946389+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":335,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634853+0000","last_change":"2024-03-24T12:33:02.509385+0000","last_active":"2024-03-24T12:35:20.634853+0000","last_peered":"2024-03-24T12:35:20.634853+0000","last_clean":"2024-03-24T12:35:20.634853+0000","last_became_active":"2024-03-24T12:26:05.315639+0000","last_became_peered":"2024-03-24T12:26:05.315639+0000","last_unstale":"2024-03-24T12:35:20.634853+0000","last_undegraded":"2024-03-24T12:35:20.634853+0000","last_fullsized":"2024-03-24T12:35:20.634853+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:33:02.509218+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:31:08.237387+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.509218+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.106538+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.631329+0000","last_change":"2024-03-24T12:35:13.031821+0000","last_active":"2024-03-24T12:35:20.631329+0000","last_peered":"2024-03-24T12:35:20.631329+0000","last_clean":"2024-03-24T12:35:20.631329+0000","last_became_active":"2024-03-24T12:26:07.343010+0000","last_became_peered":"2024-03-24T12:26:07.343010+0000","last_unstale":"2024-03-24T12:35:20.631329+0000","last_undegraded":"2024-03-24T12:35:20.631329+0000","last_fullsized":"2024-03-24T12:35:20.631329+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:13.031712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:13.031712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:24.248838+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"197'10811","reported_seq":10969,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:27.511112+0000","last_change":"2024-03-24T12:34:42.660114+0000","last_active":"2024-03-24T12:35:27.511112+0000","last_peered":"2024-03-24T12:35:27.511112+0000","last_clean":"2024-03-24T12:35:27.511112+0000","last_became_active":"2024-03-24T12:32:41.042370+0000","last_became_peered":"2024-03-24T12:32:41.042370+0000","last_unstale":"2024-03-24T12:35:27.511112+0000","last_undegraded":"2024-03-24T12:35:27.511112+0000","last_fullsized":"2024-03-24T12:35:27.511112+0000","mapping_epoch":158,"log_start":"175'4400","ondisk_log_start":"175'4400","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'8097","last_scrub_stamp":"2024-03-24T12:34:42.659934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:42.659934+0000","objects_scrubbed":6161,"log_size":6411,"log_dups_size":0,"ondisk_log_size":6411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:46.218491+0000","scrub_duration":2450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220790784,"num_objects":3369,"num_object_clones":0,"num_object_copies":10107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10811,"num_write_kb":453760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":295,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634719+0000","last_change":"2024-03-24T12:35:18.057402+0000","last_active":"2024-03-24T12:35:20.634719+0000","last_peered":"2024-03-24T12:35:20.634719+0000","last_clean":"2024-03-24T12:35:20.634719+0000","last_became_active":"2024-03-24T12:26:09.347821+0000","last_became_peered":"2024-03-24T12:26:09.347821+0000","last_unstale":"2024-03-24T12:35:20.634719+0000","last_undegraded":"2024-03-24T12:35:20.634719+0000","last_fullsized":"2024-03-24T12:35:20.634719+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:18.057286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:18.057286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:46.147860+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":560,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641622+0000","last_change":"2024-03-24T12:35:16.741056+0000","last_active":"2024-03-24T12:35:20.641622+0000","last_peered":"2024-03-24T12:35:20.641622+0000","last_clean":"2024-03-24T12:35:20.641622+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:35:20.641622+0000","last_undegraded":"2024-03-24T12:35:20.641622+0000","last_fullsized":"2024-03-24T12:35:20.641622+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:35:16.740813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.740813+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.220003+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.640517+0000","last_change":"2024-03-24T12:33:24.345396+0000","last_active":"2024-03-24T12:35:20.640517+0000","last_peered":"2024-03-24T12:35:20.640517+0000","last_clean":"2024-03-24T12:35:20.640517+0000","last_became_active":"2024-03-24T12:26:03.166467+0000","last_became_peered":"2024-03-24T12:26:03.166467+0000","last_unstale":"2024-03-24T12:35:20.640517+0000","last_undegraded":"2024-03-24T12:35:20.640517+0000","last_fullsized":"2024-03-24T12:35:20.640517+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:24.345279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:24.345279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":438,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.637023+0000","last_change":"2024-03-24T12:34:54.253870+0000","last_active":"2024-03-24T12:35:20.637023+0000","last_peered":"2024-03-24T12:35:20.637023+0000","last_clean":"2024-03-24T12:35:20.637023+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:35:20.637023+0000","last_undegraded":"2024-03-24T12:35:20.637023+0000","last_fullsized":"2024-03-24T12:35:20.637023+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:34:54.253723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.253723+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.669804+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":291,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642420+0000","last_change":"2024-03-24T12:34:47.802764+0000","last_active":"2024-03-24T12:35:20.642420+0000","last_peered":"2024-03-24T12:35:20.642420+0000","last_clean":"2024-03-24T12:35:20.642420+0000","last_became_active":"2024-03-24T12:26:07.337548+0000","last_became_peered":"2024-03-24T12:26:07.337548+0000","last_unstale":"2024-03-24T12:35:20.642420+0000","last_undegraded":"2024-03-24T12:35:20.642420+0000","last_fullsized":"2024-03-24T12:35:20.642420+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.802609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.802609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.819650+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"197'10288","reported_seq":10442,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:23.581949+0000","last_change":"2024-03-24T12:34:52.725759+0000","last_active":"2024-03-24T12:35:23.581949+0000","last_peered":"2024-03-24T12:35:23.581949+0000","last_clean":"2024-03-24T12:35:23.581949+0000","last_became_active":"2024-03-24T12:32:41.565379+0000","last_became_peered":"2024-03-24T12:32:41.565379+0000","last_unstale":"2024-03-24T12:35:23.581949+0000","last_undegraded":"2024-03-24T12:35:23.581949+0000","last_fullsized":"2024-03-24T12:35:23.581949+0000","mapping_epoch":158,"log_start":"182'5300","ondisk_log_start":"182'5300","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'8328","last_scrub_stamp":"2024-03-24T12:34:52.725350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:52.725350+0000","objects_scrubbed":5683,"log_size":4988,"log_dups_size":0,"ondisk_log_size":4988,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:58.480014+0000","scrub_duration":2941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240123904,"num_objects":3664,"num_object_clones":0,"num_object_copies":10992,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3664,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":446464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":303,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634905+0000","last_change":"2024-03-24T12:34:59.974404+0000","last_active":"2024-03-24T12:35:20.634905+0000","last_peered":"2024-03-24T12:35:20.634905+0000","last_clean":"2024-03-24T12:35:20.634905+0000","last_became_active":"2024-03-24T12:26:09.349435+0000","last_became_peered":"2024-03-24T12:26:09.349435+0000","last_unstale":"2024-03-24T12:35:20.634905+0000","last_undegraded":"2024-03-24T12:35:20.634905+0000","last_fullsized":"2024-03-24T12:35:20.634905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:59.974279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:59.974279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:22.996463+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.639026+0000","last_change":"2024-03-24T12:33:34.251764+0000","last_active":"2024-03-24T12:35:20.639026+0000","last_peered":"2024-03-24T12:35:20.639026+0000","last_clean":"2024-03-24T12:35:20.639026+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:35:20.639026+0000","last_undegraded":"2024-03-24T12:35:20.639026+0000","last_fullsized":"2024-03-24T12:35:20.639026+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.251636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.251636+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.251636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":874,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:26.126160+0000","last_change":"2024-03-24T12:35:15.719490+0000","last_active":"2024-03-24T12:35:26.126160+0000","last_peered":"2024-03-24T12:35:26.126160+0000","last_clean":"2024-03-24T12:35:26.126160+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:35:26.126160+0000","last_undegraded":"2024-03-24T12:35:26.126160+0000","last_fullsized":"2024-03-24T12:35:26.126160+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:35:15.719354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.719354+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:18.484367+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":557,"num_read_kb":552,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":313,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642076+0000","last_change":"2024-03-24T12:34:35.069650+0000","last_active":"2024-03-24T12:35:20.642076+0000","last_peered":"2024-03-24T12:35:20.642076+0000","last_clean":"2024-03-24T12:35:20.642076+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:35:20.642076+0000","last_undegraded":"2024-03-24T12:35:20.642076+0000","last_fullsized":"2024-03-24T12:35:20.642076+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:34:35.069369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:35.069369+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:04.642714+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635312+0000","last_change":"2024-03-24T12:34:52.137065+0000","last_active":"2024-03-24T12:35:20.635312+0000","last_peered":"2024-03-24T12:35:20.635312+0000","last_clean":"2024-03-24T12:35:20.635312+0000","last_became_active":"2024-03-24T12:26:07.326457+0000","last_became_peered":"2024-03-24T12:26:07.326457+0000","last_unstale":"2024-03-24T12:35:20.635312+0000","last_undegraded":"2024-03-24T12:35:20.635312+0000","last_fullsized":"2024-03-24T12:35:20.635312+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:52.136918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:52.136918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:04.242917+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"197'10692","reported_seq":10841,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:24.223467+0000","last_change":"2024-03-24T12:35:04.336817+0000","last_active":"2024-03-24T12:35:24.223467+0000","last_peered":"2024-03-24T12:35:24.223467+0000","last_clean":"2024-03-24T12:35:24.223467+0000","last_became_active":"2024-03-24T12:32:41.563612+0000","last_became_peered":"2024-03-24T12:32:41.563612+0000","last_unstale":"2024-03-24T12:35:24.223467+0000","last_undegraded":"2024-03-24T12:35:24.223467+0000","last_fullsized":"2024-03-24T12:35:24.223467+0000","mapping_epoch":158,"log_start":"183'5800","ondisk_log_start":"183'5800","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'9353","last_scrub_stamp":"2024-03-24T12:35:04.336536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.336536+0000","objects_scrubbed":5058,"log_size":4892,"log_dups_size":0,"ondisk_log_size":4892,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:16.783132+0000","scrub_duration":2389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":238288896,"num_objects":3636,"num_object_clones":0,"num_object_copies":10908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10692,"num_write_kb":458496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":295,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.636922+0000","last_change":"2024-03-24T12:34:47.787323+0000","last_active":"2024-03-24T12:35:20.636922+0000","last_peered":"2024-03-24T12:35:20.636922+0000","last_clean":"2024-03-24T12:35:20.636922+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:35:20.636922+0000","last_undegraded":"2024-03-24T12:35:20.636922+0000","last_fullsized":"2024-03-24T12:35:20.636922+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.785509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.785509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:02.500786+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":307,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635116+0000","last_change":"2024-03-24T12:34:54.953694+0000","last_active":"2024-03-24T12:35:20.635116+0000","last_peered":"2024-03-24T12:35:20.635116+0000","last_clean":"2024-03-24T12:35:20.635116+0000","last_became_active":"2024-03-24T12:26:03.167244+0000","last_became_peered":"2024-03-24T12:26:03.167244+0000","last_unstale":"2024-03-24T12:35:20.635116+0000","last_undegraded":"2024-03-24T12:35:20.635116+0000","last_fullsized":"2024-03-24T12:35:20.635116+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:54.953570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.953570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:05.662761+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"195'27","reported_seq":339,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.639027+0000","last_change":"2024-03-24T12:35:16.981335+0000","last_active":"2024-03-24T12:35:20.639027+0000","last_peered":"2024-03-24T12:35:20.639027+0000","last_clean":"2024-03-24T12:35:20.639027+0000","last_became_active":"2024-03-24T12:26:05.356745+0000","last_became_peered":"2024-03-24T12:26:05.356745+0000","last_unstale":"2024-03-24T12:35:20.639027+0000","last_undegraded":"2024-03-24T12:35:20.639027+0000","last_fullsized":"2024-03-24T12:35:20.639027+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'27","last_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_deep_scrub":"195'27","last_deep_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.981148+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:23.598941+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.636794+0000","last_change":"2024-03-24T12:34:44.164236+0000","last_active":"2024-03-24T12:35:20.636794+0000","last_peered":"2024-03-24T12:35:20.636794+0000","last_clean":"2024-03-24T12:35:20.636794+0000","last_became_active":"2024-03-24T12:26:07.337303+0000","last_became_peered":"2024-03-24T12:26:07.337303+0000","last_unstale":"2024-03-24T12:35:20.636794+0000","last_undegraded":"2024-03-24T12:35:20.636794+0000","last_fullsized":"2024-03-24T12:35:20.636794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:44.164114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.164114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:46.833106+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"197'11002","reported_seq":11158,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:25.915852+0000","last_change":"2024-03-24T12:34:51.920874+0000","last_active":"2024-03-24T12:35:25.915852+0000","last_peered":"2024-03-24T12:35:25.915852+0000","last_clean":"2024-03-24T12:35:25.915852+0000","last_became_active":"2024-03-24T12:32:41.565563+0000","last_became_peered":"2024-03-24T12:32:41.565563+0000","last_unstale":"2024-03-24T12:35:25.915852+0000","last_undegraded":"2024-03-24T12:35:25.915852+0000","last_fullsized":"2024-03-24T12:35:25.915852+0000","mapping_epoch":158,"log_start":"179'5200","ondisk_log_start":"179'5200","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'8699","last_scrub_stamp":"2024-03-24T12:34:51.920651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:51.920651+0000","objects_scrubbed":5891,"log_size":5802,"log_dups_size":0,"ondisk_log_size":5802,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:17.483495+0000","scrub_duration":2177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231211008,"num_objects":3528,"num_object_clones":0,"num_object_copies":10584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11002,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":279,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.637302+0000","last_change":"2024-03-24T12:33:12.332284+0000","last_active":"2024-03-24T12:35:20.637302+0000","last_peered":"2024-03-24T12:35:20.637302+0000","last_clean":"2024-03-24T12:35:20.637302+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:35:20.637302+0000","last_undegraded":"2024-03-24T12:35:20.637302+0000","last_fullsized":"2024-03-24T12:35:20.637302+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:12.332162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.332162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":307,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634570+0000","last_change":"2024-03-24T12:35:00.924220+0000","last_active":"2024-03-24T12:35:20.634570+0000","last_peered":"2024-03-24T12:35:20.634570+0000","last_clean":"2024-03-24T12:35:20.634570+0000","last_became_active":"2024-03-24T12:26:03.168184+0000","last_became_peered":"2024-03-24T12:26:03.168184+0000","last_unstale":"2024-03-24T12:35:20.634570+0000","last_undegraded":"2024-03-24T12:35:20.634570+0000","last_fullsized":"2024-03-24T12:35:20.634570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:00.924106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.924106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:20.399991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"138'15","reported_seq":323,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635611+0000","last_change":"2024-03-24T12:33:07.518493+0000","last_active":"2024-03-24T12:35:20.635611+0000","last_peered":"2024-03-24T12:35:20.635611+0000","last_clean":"2024-03-24T12:35:20.635611+0000","last_became_active":"2024-03-24T12:26:05.316739+0000","last_became_peered":"2024-03-24T12:26:05.316739+0000","last_unstale":"2024-03-24T12:35:20.635611+0000","last_undegraded":"2024-03-24T12:35:20.635611+0000","last_fullsized":"2024-03-24T12:35:20.635611+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:33:07.518368+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.518368+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.106568+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":412,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.638742+0000","last_change":"2024-03-24T12:34:55.749516+0000","last_active":"2024-03-24T12:35:20.638742+0000","last_peered":"2024-03-24T12:35:20.638742+0000","last_clean":"2024-03-24T12:35:20.638742+0000","last_became_active":"2024-03-24T12:26:07.336080+0000","last_became_peered":"2024-03-24T12:26:07.336080+0000","last_unstale":"2024-03-24T12:35:20.638742+0000","last_undegraded":"2024-03-24T12:35:20.638742+0000","last_fullsized":"2024-03-24T12:35:20.638742+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:34:55.749381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:55.749381+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:16.630589+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"197'10810","reported_seq":10967,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:27.551219+0000","last_change":"2024-03-24T12:34:48.325053+0000","last_active":"2024-03-24T12:35:27.551219+0000","last_peered":"2024-03-24T12:35:27.551219+0000","last_clean":"2024-03-24T12:35:27.551219+0000","last_became_active":"2024-03-24T12:32:41.563736+0000","last_became_peered":"2024-03-24T12:32:41.563736+0000","last_unstale":"2024-03-24T12:35:27.551219+0000","last_undegraded":"2024-03-24T12:35:27.551219+0000","last_fullsized":"2024-03-24T12:35:27.551219+0000","mapping_epoch":158,"log_start":"182'5500","ondisk_log_start":"182'5500","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'8351","last_scrub_stamp":"2024-03-24T12:34:48.324926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.324926+0000","objects_scrubbed":5851,"log_size":5310,"log_dups_size":0,"ondisk_log_size":5310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:09.073002+0000","scrub_duration":2162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218431512,"num_objects":3334,"num_object_clones":0,"num_object_copies":10002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3334,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10810,"num_write_kb":452545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642150+0000","last_change":"2024-03-24T12:33:38.172365+0000","last_active":"2024-03-24T12:35:20.642150+0000","last_peered":"2024-03-24T12:35:20.642150+0000","last_clean":"2024-03-24T12:35:20.642150+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:35:20.642150+0000","last_undegraded":"2024-03-24T12:35:20.642150+0000","last_fullsized":"2024-03-24T12:35:20.642150+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:38.172239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:38.172239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":300,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.642172+0000","last_change":"2024-03-24T12:33:29.273583+0000","last_active":"2024-03-24T12:35:20.642172+0000","last_peered":"2024-03-24T12:35:20.642172+0000","last_clean":"2024-03-24T12:35:20.642172+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:35:20.642172+0000","last_undegraded":"2024-03-24T12:35:20.642172+0000","last_fullsized":"2024-03-24T12:35:20.642172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:29.273325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:29.273325+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"93'21","reported_seq":331,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.626575+0000","last_change":"2024-03-24T12:35:14.829619+0000","last_active":"2024-03-24T12:35:20.626575+0000","last_peered":"2024-03-24T12:35:20.626575+0000","last_clean":"2024-03-24T12:35:20.626575+0000","last_became_active":"2024-03-24T12:26:05.315454+0000","last_became_peered":"2024-03-24T12:26:05.315454+0000","last_unstale":"2024-03-24T12:35:20.626575+0000","last_undegraded":"2024-03-24T12:35:20.626575+0000","last_fullsized":"2024-03-24T12:35:20.626575+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'21","last_scrub_stamp":"2024-03-24T12:35:14.829466+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:35:14.829466+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:35.823180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":412,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.646278+0000","last_change":"2024-03-24T12:33:35.289329+0000","last_active":"2024-03-24T12:35:20.646278+0000","last_peered":"2024-03-24T12:35:20.646278+0000","last_clean":"2024-03-24T12:35:20.646278+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:35:20.646278+0000","last_undegraded":"2024-03-24T12:35:20.646278+0000","last_fullsized":"2024-03-24T12:35:20.646278+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:33:35.288932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:35.288932+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"197'10527","reported_seq":10690,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:25.927706+0000","last_change":"2024-03-24T12:34:34.361654+0000","last_active":"2024-03-24T12:35:25.927706+0000","last_peered":"2024-03-24T12:35:25.927706+0000","last_clean":"2024-03-24T12:35:25.927706+0000","last_became_active":"2024-03-24T12:32:41.036596+0000","last_became_peered":"2024-03-24T12:32:41.036596+0000","last_unstale":"2024-03-24T12:35:25.927706+0000","last_undegraded":"2024-03-24T12:35:25.927706+0000","last_fullsized":"2024-03-24T12:35:25.927706+0000","mapping_epoch":158,"log_start":"178'4700","ondisk_log_start":"178'4700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'7551","last_scrub_stamp":"2024-03-24T12:34:34.361066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:34.361066+0000","objects_scrubbed":6609,"log_size":5827,"log_dups_size":0,"ondisk_log_size":5827,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:52.114579+0000","scrub_duration":2368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":233111552,"num_objects":3557,"num_object_clones":0,"num_object_copies":10671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10527,"num_write_kb":450688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627062+0000","last_change":"2024-03-24T12:33:35.381730+0000","last_active":"2024-03-24T12:35:20.627062+0000","last_peered":"2024-03-24T12:35:20.627062+0000","last_clean":"2024-03-24T12:35:20.627062+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:35:20.627062+0000","last_undegraded":"2024-03-24T12:35:20.627062+0000","last_fullsized":"2024-03-24T12:35:20.627062+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:35.381614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:35.381614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":291,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.643435+0000","last_change":"2024-03-24T12:33:15.388058+0000","last_active":"2024-03-24T12:35:20.643435+0000","last_peered":"2024-03-24T12:35:20.643435+0000","last_clean":"2024-03-24T12:35:20.643435+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:35:20.643435+0000","last_undegraded":"2024-03-24T12:35:20.643435+0000","last_fullsized":"2024-03-24T12:35:20.643435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:15.387796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:15.387796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":327,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.646153+0000","last_change":"2024-03-24T12:33:33.420555+0000","last_active":"2024-03-24T12:35:20.646153+0000","last_peered":"2024-03-24T12:35:20.646153+0000","last_clean":"2024-03-24T12:35:20.646153+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:35:20.646153+0000","last_undegraded":"2024-03-24T12:35:20.646153+0000","last_fullsized":"2024-03-24T12:35:20.646153+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T12:33:33.420394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:33.420394+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.626362+0000","last_change":"2024-03-24T12:34:34.454336+0000","last_active":"2024-03-24T12:35:20.626362+0000","last_peered":"2024-03-24T12:35:20.626362+0000","last_clean":"2024-03-24T12:35:20.626362+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:35:20.626362+0000","last_undegraded":"2024-03-24T12:35:20.626362+0000","last_fullsized":"2024-03-24T12:35:20.626362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:34.454215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:34.454215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:56.494180+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"197'10625","reported_seq":10782,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:23.577846+0000","last_change":"2024-03-24T12:34:48.526866+0000","last_active":"2024-03-24T12:35:23.577846+0000","last_peered":"2024-03-24T12:35:23.577846+0000","last_clean":"2024-03-24T12:35:23.577846+0000","last_became_active":"2024-03-24T12:32:41.042732+0000","last_became_peered":"2024-03-24T12:32:41.042732+0000","last_unstale":"2024-03-24T12:35:23.577846+0000","last_undegraded":"2024-03-24T12:35:23.577846+0000","last_fullsized":"2024-03-24T12:35:23.577846+0000","mapping_epoch":158,"log_start":"183'5700","ondisk_log_start":"183'5700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'8368","last_scrub_stamp":"2024-03-24T12:34:48.526733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.526733+0000","objects_scrubbed":6006,"log_size":4925,"log_dups_size":0,"ondisk_log_size":4925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:01.870203+0000","scrub_duration":2757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":242548736,"num_objects":3701,"num_object_clones":0,"num_object_copies":11103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10625,"num_write_kb":458432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":287,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.634254+0000","last_change":"2024-03-24T12:33:36.210705+0000","last_active":"2024-03-24T12:35:20.634254+0000","last_peered":"2024-03-24T12:35:20.634254+0000","last_clean":"2024-03-24T12:35:20.634254+0000","last_became_active":"2024-03-24T12:26:09.351061+0000","last_became_peered":"2024-03-24T12:26:09.351061+0000","last_unstale":"2024-03-24T12:35:20.634254+0000","last_undegraded":"2024-03-24T12:35:20.634254+0000","last_fullsized":"2024-03-24T12:35:20.634254+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:36.210440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.210440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:24.036620+0000","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"140'9","reported_seq":421,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.628063+0000","last_change":"2024-03-24T12:34:44.703519+0000","last_active":"2024-03-24T12:35:20.628063+0000","last_peered":"2024-03-24T12:35:20.628063+0000","last_clean":"2024-03-24T12:35:20.628063+0000","last_became_active":"2024-03-24T12:26:07.328595+0000","last_became_peered":"2024-03-24T12:26:07.328595+0000","last_unstale":"2024-03-24T12:35:20.628063+0000","last_undegraded":"2024-03-24T12:35:20.628063+0000","last_fullsized":"2024-03-24T12:35:20.628063+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'9","last_scrub_stamp":"2024-03-24T12:34:44.703379+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.703379+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:57.223693+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":291,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.646055+0000","last_change":"2024-03-24T12:33:08.247372+0000","last_active":"2024-03-24T12:35:20.646055+0000","last_peered":"2024-03-24T12:35:20.646055+0000","last_clean":"2024-03-24T12:35:20.646055+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:35:20.646055+0000","last_undegraded":"2024-03-24T12:35:20.646055+0000","last_fullsized":"2024-03-24T12:35:20.646055+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:08.247253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.247253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"92'22","reported_seq":310,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.630418+0000","last_change":"2024-03-24T12:34:50.250429+0000","last_active":"2024-03-24T12:35:20.630418+0000","last_peered":"2024-03-24T12:35:20.630418+0000","last_clean":"2024-03-24T12:35:20.630418+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:35:20.630418+0000","last_undegraded":"2024-03-24T12:35:20.630418+0000","last_fullsized":"2024-03-24T12:35:20.630418+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-24T12:34:50.250291+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:34:50.250291+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:51.281964+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"197'10800","reported_seq":10950,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:27.474359+0000","last_change":"2024-03-24T12:34:59.406398+0000","last_active":"2024-03-24T12:35:27.474359+0000","last_peered":"2024-03-24T12:35:27.474359+0000","last_clean":"2024-03-24T12:35:27.474359+0000","last_became_active":"2024-03-24T12:32:41.044207+0000","last_became_peered":"2024-03-24T12:32:41.044207+0000","last_unstale":"2024-03-24T12:35:27.474359+0000","last_undegraded":"2024-03-24T12:35:27.474359+0000","last_fullsized":"2024-03-24T12:35:27.474359+0000","mapping_epoch":158,"log_start":"184'6000","ondisk_log_start":"184'6000","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'8943","last_scrub_stamp":"2024-03-24T12:34:59.405885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:59.405885+0000","objects_scrubbed":5256,"log_size":4800,"log_dups_size":0,"ondisk_log_size":4800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:18.542470+0000","scrub_duration":1672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219545600,"num_objects":3350,"num_object_clones":0,"num_object_copies":10050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10800,"num_write_kb":452800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":279,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.641365+0000","last_change":"2024-03-24T12:33:19.377017+0000","last_active":"2024-03-24T12:35:20.641365+0000","last_peered":"2024-03-24T12:35:20.641365+0000","last_clean":"2024-03-24T12:35:20.641365+0000","last_became_active":"2024-03-24T12:26:09.643739+0000","last_became_peered":"2024-03-24T12:26:09.643739+0000","last_unstale":"2024-03-24T12:35:20.641365+0000","last_undegraded":"2024-03-24T12:35:20.641365+0000","last_fullsized":"2024-03-24T12:35:20.641365+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:19.376894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.376894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635247+0000","last_change":"2024-03-24T12:35:15.004983+0000","last_active":"2024-03-24T12:35:20.635247+0000","last_peered":"2024-03-24T12:35:20.635247+0000","last_clean":"2024-03-24T12:35:20.635247+0000","last_became_active":"2024-03-24T12:26:07.332766+0000","last_became_peered":"2024-03-24T12:26:07.332766+0000","last_unstale":"2024-03-24T12:35:20.635247+0000","last_undegraded":"2024-03-24T12:35:20.635247+0000","last_fullsized":"2024-03-24T12:35:20.635247+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:15.004831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.004831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.429614+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":299,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.626446+0000","last_change":"2024-03-24T12:33:08.165239+0000","last_active":"2024-03-24T12:35:20.626446+0000","last_peered":"2024-03-24T12:35:20.626446+0000","last_clean":"2024-03-24T12:35:20.626446+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:35:20.626446+0000","last_undegraded":"2024-03-24T12:35:20.626446+0000","last_fullsized":"2024-03-24T12:35:20.626446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:08.165107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.165107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":322,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627897+0000","last_change":"2024-03-24T12:34:53.872190+0000","last_active":"2024-03-24T12:35:20.627897+0000","last_peered":"2024-03-24T12:35:20.627897+0000","last_clean":"2024-03-24T12:35:20.627897+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:35:20.627897+0000","last_undegraded":"2024-03-24T12:35:20.627897+0000","last_fullsized":"2024-03-24T12:35:20.627897+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:34:53.872101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.872101+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:23.514337+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"197'10614","reported_seq":10778,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:23.573668+0000","last_change":"2024-03-24T12:34:33.907750+0000","last_active":"2024-03-24T12:35:23.573668+0000","last_peered":"2024-03-24T12:35:23.573668+0000","last_clean":"2024-03-24T12:35:23.573668+0000","last_became_active":"2024-03-24T12:32:41.563886+0000","last_became_peered":"2024-03-24T12:32:41.563886+0000","last_unstale":"2024-03-24T12:35:23.573668+0000","last_undegraded":"2024-03-24T12:35:23.573668+0000","last_fullsized":"2024-03-24T12:35:23.573668+0000","mapping_epoch":158,"log_start":"183'5600","ondisk_log_start":"183'5600","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'7610","last_scrub_stamp":"2024-03-24T12:34:33.907525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:33.907525+0000","objects_scrubbed":6738,"log_size":5014,"log_dups_size":0,"ondisk_log_size":5014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:50.675694+0000","scrub_duration":2380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":240648192,"num_objects":3672,"num_object_clones":0,"num_object_copies":11016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10614,"num_write_kb":457152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":295,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.630537+0000","last_change":"2024-03-24T12:34:48.341578+0000","last_active":"2024-03-24T12:35:20.630537+0000","last_peered":"2024-03-24T12:35:20.630537+0000","last_clean":"2024-03-24T12:35:20.630537+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:35:20.630537+0000","last_undegraded":"2024-03-24T12:35:20.630537+0000","last_fullsized":"2024-03-24T12:35:20.630537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:48.341457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.341457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:14.005235+0000","scrub_duration":236,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":307,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.635358+0000","last_change":"2024-03-24T12:35:04.121930+0000","last_active":"2024-03-24T12:35:20.635358+0000","last_peered":"2024-03-24T12:35:20.635358+0000","last_clean":"2024-03-24T12:35:20.635358+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:35:20.635358+0000","last_undegraded":"2024-03-24T12:35:20.635358+0000","last_fullsized":"2024-03-24T12:35:20.635358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.121802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.121802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:28.728829+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":307,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.636632+0000","last_change":"2024-03-24T12:34:50.252743+0000","last_active":"2024-03-24T12:35:20.636632+0000","last_peered":"2024-03-24T12:35:20.636632+0000","last_clean":"2024-03-24T12:35:20.636632+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:35:20.636632+0000","last_undegraded":"2024-03-24T12:35:20.636632+0000","last_fullsized":"2024-03-24T12:35:20.636632+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:50.252634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:50.252634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:59.548446+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"94'25","reported_seq":331,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627472+0000","last_change":"2024-03-24T12:34:43.556976+0000","last_active":"2024-03-24T12:35:20.627472+0000","last_peered":"2024-03-24T12:35:20.627472+0000","last_clean":"2024-03-24T12:35:20.627472+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:35:20.627472+0000","last_undegraded":"2024-03-24T12:35:20.627472+0000","last_fullsized":"2024-03-24T12:35:20.627472+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'25","last_scrub_stamp":"2024-03-24T12:34:43.556836+0000","last_deep_scrub":"94'25","last_deep_scrub_stamp":"2024-03-24T12:28:02.436198+0000","last_clean_scrub_stamp":"2024-03-24T12:34:43.556836+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:45.444607+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"197'10674","reported_seq":23442,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:24.254562+0000","last_change":"2024-03-24T12:35:01.894367+0000","last_active":"2024-03-24T12:35:24.254562+0000","last_peered":"2024-03-24T12:35:24.254562+0000","last_clean":"2024-03-24T12:35:24.254562+0000","last_became_active":"2024-03-24T12:34:49.326034+0000","last_became_peered":"2024-03-24T12:34:49.326034+0000","last_unstale":"2024-03-24T12:35:24.254562+0000","last_undegraded":"2024-03-24T12:35:24.254562+0000","last_fullsized":"2024-03-24T12:35:24.254562+0000","mapping_epoch":190,"log_start":"183'5800","ondisk_log_start":"183'5800","created":158,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'9180","last_scrub_stamp":"2024-03-24T12:35:01.892660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:35:01.892660+0000","objects_scrubbed":5168,"log_size":4874,"log_dups_size":0,"ondisk_log_size":4874,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:06.389031+0000","scrub_duration":1912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":237109248,"num_objects":3618,"num_object_clones":0,"num_object_copies":10854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10674,"num_write_kb":457344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6302,"num_bytes_recovered":413007872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":295,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.628065+0000","last_change":"2024-03-24T12:33:10.193697+0000","last_active":"2024-03-24T12:35:20.628065+0000","last_peered":"2024-03-24T12:35:20.628065+0000","last_clean":"2024-03-24T12:35:20.628065+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:35:20.628065+0000","last_undegraded":"2024-03-24T12:35:20.628065+0000","last_fullsized":"2024-03-24T12:35:20.628065+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:10.193581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:10.193581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":421,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:23.323180+0000","last_change":"2024-03-24T12:35:00.998847+0000","last_active":"2024-03-24T12:35:23.323180+0000","last_peered":"2024-03-24T12:35:23.323180+0000","last_clean":"2024-03-24T12:35:23.323180+0000","last_became_active":"2024-03-24T12:26:07.334798+0000","last_became_peered":"2024-03-24T12:26:07.334798+0000","last_unstale":"2024-03-24T12:35:23.323180+0000","last_undegraded":"2024-03-24T12:35:23.323180+0000","last_fullsized":"2024-03-24T12:35:23.323180+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:35:00.998713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.998713+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:06.501313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":291,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.640873+0000","last_change":"2024-03-24T12:33:21.383831+0000","last_active":"2024-03-24T12:35:20.640873+0000","last_peered":"2024-03-24T12:35:20.640873+0000","last_clean":"2024-03-24T12:35:20.640873+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:35:20.640873+0000","last_undegraded":"2024-03-24T12:35:20.640873+0000","last_fullsized":"2024-03-24T12:35:20.640873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.383703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.383703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":318,"reported_epoch":197,"state":"active+clean","last_fresh":"2024-03-24T12:35:20.627760+0000","last_change":"2024-03-24T12:34:48.616552+0000","last_active":"2024-03-24T12:35:20.627760+0000","last_peered":"2024-03-24T12:35:20.627760+0000","last_clean":"2024-03-24T12:35:20.627760+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:35:20.627760+0000","last_undegraded":"2024-03-24T12:35:20.627760+0000","last_fullsized":"2024-03-24T12:35:20.627760+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:34:48.616461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.616461+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:58.876656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":3687841816,"num_objects":56273,"num_object_clones":0,"num_object_copies":168819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56273,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":171443,"num_write_kb":7286849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6302,"num_bytes_recovered":413007872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11173310464,"data_stored":11173298248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85543,"ondisk_log_size":85543,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":557,"num_read_kb":552,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037877,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1314660,"kb_used_data":1089116,"kb_used_omap":1,"kb_used_meta":225534,"kb_avail":92430492,"statfs":{"total":95995035648,"available":94648823808,"internally_reserved":0,"allocated":1115254784,"data_stored":1114850218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":230947274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332219,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1311560,"kb_used_data":1081792,"kb_used_omap":1,"kb_used_meta":229694,"kb_avail":92433592,"statfs":{"total":95995035648,"available":94651998208,"internally_reserved":0,"allocated":1107755008,"data_stored":1107378502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":235207111},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574997,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2012028,"kb_used_data":1746224,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":91733124,"statfs":{"total":95995035648,"available":93934718976,"internally_reserved":0,"allocated":1788133376,"data_stored":1787708152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738522,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1710176,"kb_used_data":1372056,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":92034976,"statfs":{"total":95995035648,"available":94243815424,"internally_reserved":0,"allocated":1404985344,"data_stored":1404584194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":346094021},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411472,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1855112,"kb_used_data":1503104,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":91890040,"statfs":{"total":95995035648,"available":94095400960,"internally_reserved":0,"allocated":1539178496,"data_stored":1538800930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117128,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1730284,"kb_used_data":1406052,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92014868,"statfs":{"total":95995035648,"available":94223224832,"internally_reserved":0,"allocated":1439797248,"data_stored":1439384238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":331872711},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920901,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1292828,"kb_used_data":1099092,"kb_used_omap":1,"kb_used_meta":193598,"kb_avail":92452324,"statfs":{"total":95995035648,"available":94671179776,"internally_reserved":0,"allocated":1125470208,"data_stored":1125072408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":198244805},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626560,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2019092,"kb_used_data":1744972,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":91726060,"statfs":{"total":95995035648,"available":93927485440,"internally_reserved":0,"allocated":1786851328,"data_stored":1786462848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":280689093},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1375862784,"data_stored":1375862784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1772421120,"data_stored":1772421120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1520504832,"data_stored":1520500760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1428881408,"data_stored":1428881408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1111687168,"data_stored":1111687168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1766658048,"data_stored":1766653976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1101856768,"data_stored":1101856768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1095438336,"data_stored":1095434264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:35:31.349 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:31.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:31.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:31 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1333513668' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:35:31.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:31.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:31.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:31.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:31.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:31.624 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:31.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:31.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:31.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:31.757 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:31.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:31.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:31.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:31.899 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:31.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:31.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:32.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:32.055 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:32.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:32.056 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:32.157 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:32 smithi029 ceph-mon[41500]: from='client.24784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:35:32.157 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:32 smithi029 ceph-mon[41500]: pgmap v379: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T12:35:32.157 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:32 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:35:32.157 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:32 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:35:32.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:32 smithi029 ceph-mon[29919]: from='client.24784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:35:32.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:32 smithi029 ceph-mon[29919]: pgmap v379: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T12:35:32.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:32 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:35:32.157 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:32 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:35:32.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:32.192 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:32.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:32.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:32.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:32.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:32.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:32.328 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:32 smithi190 ceph-mon[33748]: from='client.24784 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:35:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:32 smithi190 ceph-mon[33748]: pgmap v379: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-24T12:35:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:32 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:35:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:32 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:35:32.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:32.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:32.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:32.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:32.568 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:32.603 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:32.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:32.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:32.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:32.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:32.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:32.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:32.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:32.884 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:32.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:32.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:33.024 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:33.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:33.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:33.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:33.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:33.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:33.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:33.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:33.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:33 smithi190 ceph-mon[33748]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T12:35:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:35:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:33 smithi190 ceph-mon[33748]: pgmap v381: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T12:35:33.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:33.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:33 smithi029 ceph-mon[29919]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T12:35:33.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:35:33.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:33.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:33 smithi029 ceph-mon[29919]: pgmap v381: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T12:35:33.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:33 smithi029 ceph-mon[41500]: osdmap e198: 8 total, 8 up, 8 in 2024-03-24T12:35:33.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:35:33.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:33.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:33 smithi029 ceph-mon[41500]: pgmap v381: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T12:35:33.434 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:33.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:33.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:33.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:33.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:33.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:33.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:33.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:33.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:33.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:33.847 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:33.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:33.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:33.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:33.979 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:33.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:34.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:34.115 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:34.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:34.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:34.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:34.251 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:34.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:34.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:34.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:34.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:34.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:34.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:34.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:34.517 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:34.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:34.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:34.651 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:34.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:34.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:34.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:34.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:34.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:34.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:34.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:34.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:34.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:35.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:35.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:35.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:35.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:35.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:35.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:35.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:35.320 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:35.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:35.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:35.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:35.460 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:35.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:35.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:35.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:35.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:35.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:35.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:35.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:35.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:35.862 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:35.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:35.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:35.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:35.994 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:35.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:35.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:36.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:36.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:36.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:36.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:36.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:36.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:36.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:36.334 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:35:36.334 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:35:36.334 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-24T12:35:36.335 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-24T12:35:36.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:35:36.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:36.374 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:36 smithi029 ceph-mon[41500]: pgmap v382: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:35:36.374 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:36 smithi029 ceph-mon[29919]: pgmap v382: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:35:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:36 smithi190 ceph-mon[33748]: pgmap v382: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:35:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:36.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:36.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:36.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:36.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:36.554 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:36.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:36.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:36.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:36.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:36.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:36.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:36.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:36.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:36.946 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:36.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:36.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:37.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:37.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:37.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:37.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:37.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:37.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:37.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:37.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:37.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:37.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:37.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:37.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:37 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:35:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:37 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:35:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:37 smithi190 ceph-mon[33748]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T12:35:37.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:37 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:35:37.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:37 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:35:37.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:37 smithi029 ceph-mon[29919]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T12:35:37.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:37 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:35:37.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:37 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:35:37.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:37 smithi029 ceph-mon[41500]: osdmap e199: 8 total, 8 up, 8 in 2024-03-24T12:35:37.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:37.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:37.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:37.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:37.614 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:37.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:37.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:37.717 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:37.754 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:37.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:37.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:37.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:37.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:37.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:37.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:35:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:35:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:35:37.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:38.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:38.028 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:38.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:38.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:38.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:38.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:38.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:38.307 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:38.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:38.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:38 smithi190 ceph-mon[33748]: pgmap v384: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:35:38.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:35:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:35:38.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:38.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:38 smithi029 ceph-mon[29919]: pgmap v384: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:35:38.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:38 smithi029 ceph-mon[41500]: pgmap v384: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:35:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:38.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:38.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:38.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:38.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:38.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:38.716 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:38.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:38.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:38.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:38.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:38.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:38.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:38.953 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:38.982 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:38.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:38.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:39.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:39.130 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:39.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:39.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:39.270 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:39.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:39.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:39.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:39.411 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:39.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:39.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:39.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:39.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:39.555 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:39.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:39.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:39.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:39.678 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:35:39.686 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:39.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:39.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:39.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:39.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:39.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:39.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:39.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:39.975 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:40.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:40.088 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:40 smithi029 ceph-mon[41500]: pgmap v385: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T12:35:40.088 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:40 smithi029 ceph-mon[29919]: pgmap v385: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T12:35:40.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:40.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:40.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:40.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:40.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:40.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:40.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:40.381 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:40.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:40 smithi190 ceph-mon[33748]: pgmap v385: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-03-24T12:35:40.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:40.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:40.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:40.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:40.655 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:40.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:40.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:40.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:40.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:40.813 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:150960 7.18 7:1d604460:::benchmark_data_smithi029_84282_object50197:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:08.190147+0000", 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 272.31735074800002, 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.076245219, 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:35:40.814 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 150960 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190147+0000", 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190147+0000", 2024-03-24T12:35:40.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190149+0000", 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7570000000000001e-06 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190155+0000", 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1290000000000001e-06 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190167+0000", 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2293000000000001e-05 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190176+0000", 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3340000000000002e-06 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.196962+0000", 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0067861099999999997 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.196971+0000", 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9940000000000006e-06 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.258125+0000", 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0611542940000001 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for missing object", 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.258536+0000", 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041083200000000003 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.266309+0000", 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0077733589999999997 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.266392+0000", 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3116999999999997e-05 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.819 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:151002 7.18 7:1805c9dd:::benchmark_data_smithi029_84282_object50239:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:08.242717+0000", 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 272.264780589, 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.028275295, 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 151002 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.242717+0000", 2024-03-24T12:35:40.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.242717+0000", 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.242719+0000", 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.175e-06 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.242722+0000", 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4800000000000001e-06 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.242730+0000", 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2279999999999998e-06 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.242736+0000", 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1290000000000001e-06 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.822 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.242799+0000", 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2651999999999996e-05 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.242811+0000", 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1711e-05 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.258960+0000", 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0161490479999999 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.259075+0000", 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011519900000000001 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.259871+0000", 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00079627099999999998 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.265953+0000", 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060815160000000004 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.266002+0000", 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8964000000000001e-05 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.266429+0000", 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00042688999999999997 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.266480+0000", 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1307000000000002e-05 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.270993+0000", 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0045127250000000004 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:151019 7.18 7:18ab73a4:::benchmark_data_smithi029_84282_object50256:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:08.266529+0000", 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 272.240969296, 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0103486749999999, 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.826 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 151019 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.266529+0000", 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.266529+0000", 2024-03-24T12:35:40.827 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.266530+0000", 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.378e-06 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.266534+0000", 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2320000000000003e-06 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.266543+0000", 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6039999999999993e-06 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.266550+0000", 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.25e-06 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.266857+0000", 2024-03-24T12:35:40.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030691699999999998 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.266866+0000", 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3889999999999994e-06 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.263309+0000", 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.99644299400000003 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.263418+0000", 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000108519 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.263807+0000", 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00038934400000000002 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.266014+0000", 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0022065129999999998 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.276466+0000", 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010452483 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.276687+0000", 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00022033 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.276743+0000", 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6681000000000002e-05 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.276877+0000", 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000134041 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:151027 7.18 7:1a753ac8:::benchmark_data_smithi029_84282_object50264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:08.272448+0000", 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 272.23504990200001, 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0049181650000001, 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.833 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 151027 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.272448+0000", 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.834 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.272448+0000", 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.272449+0000", 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3e-07 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.272452+0000", 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0529999999999998e-06 2024-03-24T12:35:40.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.272462+0000", 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0385e-05 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.272470+0000", 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5820000000000004e-06 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.274808+0000", 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0023378470000000001 2024-03-24T12:35:40.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.274819+0000", 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1185999999999999e-05 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.271038+0000", 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.99621888000000003 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.271158+0000", 2024-03-24T12:35:40.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012023699999999999 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.271596+0000", 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00043808300000000003 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.276482+0000", 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0048862510000000003 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.277031+0000", 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00054907899999999995 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.277127+0000", 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5983999999999994e-05 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.277182+0000", 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4681e-05 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.277366+0000", 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018418699999999999 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:151043 7.18 7:1b06efeb:::benchmark_data_smithi029_84282_object50280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:08.287635+0000", 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 272.21986315700002, 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0041872270000001, 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.840 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 151043 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.287635+0000", 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.287635+0000", 2024-03-24T12:35:40.841 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.287635+0000", 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.46e-07 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.287638+0000", 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9519999999999999e-06 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.287644+0000", 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0430000000000004e-06 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.287650+0000", 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6829999999999996e-06 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.287875+0000", 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00022553199999999999 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.287886+0000", 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0131e-05 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.277408+0000", 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.98952265100000003 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.277540+0000", 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013176299999999999 2024-03-24T12:35:40.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.278059+0000", 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00051854799999999997 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.291222+0000", 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.013163352999999999 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.291271+0000", 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8946000000000002e-05 2024-03-24T12:35:40.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.291285+0000", 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4022999999999999e-05 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.291342+0000", 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7204000000000003e-05 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.291822+0000", 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00047995200000000002 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:151059 7.18 7:1db14590:::benchmark_data_smithi029_84282_object50296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:08.304179+0000", 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 272.20331914000002, 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.98821009199999998, 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.847 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 151059 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.304179+0000", 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.304179+0000", 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.304180+0000", 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4300000000000001e-07 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.304185+0000", 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9989999999999999e-06 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.304196+0000", 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.098e-05 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.304204+0000", 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1890000000000007e-06 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.304385+0000", 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000180738 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.304396+0000", 2024-03-24T12:35:40.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1195999999999999e-05 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.281940+0000", 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.97754394099999997 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.282061+0000", 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000120912 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:35:40.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.282446+0000", 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00038487999999999999 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.291830+0000", 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0093844170000000008 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.291995+0000", 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016513699999999999 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.292066+0000", 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0458999999999997e-05 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.292174+0000", 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000108436 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.292389+0000", 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021486500000000001 2024-03-24T12:35:40.853 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:151120 7.18 7:1cff0c1d:::benchmark_data_smithi029_84282_object50357:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:08.382472+0000", 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 272.12502605399999, 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.925669042, 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.854 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 151120 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.382472+0000", 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.382472+0000", 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.382473+0000", 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3659999999999999e-06 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.382480+0000", 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4389999999999997e-06 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.382490+0000", 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0292000000000001e-05 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.382498+0000", 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6090000000000003e-06 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.382683+0000", 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018568999999999999 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.382709+0000", 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.5840999999999999e-05 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.288103+0000", 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.90539393000000001 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.288213+0000", 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000109895 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.288589+0000", 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00037569200000000001 2024-03-24T12:35:40.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.292433+0000", 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0038442160000000001 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.307816+0000", 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.015383196 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.307830+0000", 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3692999999999999e-05 2024-03-24T12:35:40.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.307876+0000", 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6187999999999999e-05 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.308141+0000", 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026499500000000002 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.860 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:151136 7.18 7:1cef592c:::benchmark_data_smithi029_84282_object50373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:08.399870+0000", 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 272.10762829800001, 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.90852168200000005, 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 151136 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.861 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.399870+0000", 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.399870+0000", 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.399870+0000", 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1900000000000003e-07 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.399873+0000", 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.65e-06 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.399880+0000", 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8569999999999997e-06 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.399885+0000", 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2249999999999999e-06 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.400061+0000", 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017606499999999999 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.400072+0000", 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0720000000000001e-05 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.292464+0000", 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.89239217100000001 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.864 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.292572+0000", 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000107971 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 6,7", 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.292962+0000", 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00039030100000000002 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.308151+0000", 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.015189249 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.865 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.308202+0000", 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0516000000000002e-05 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.308244+0000", 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2012999999999999e-05 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.308292+0000", 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8223999999999998e-05 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.308391+0000", 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9201000000000001e-05 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:163980 7.13 7:cef35ea1:::benchmark_data_smithi029_84282_object63217:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.071712+0000", 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.43578562100001, 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.14239947, 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:35:40.867 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 163980 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071712+0000", 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071712+0000", 2024-03-24T12:35:40.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071714+0000", 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.669e-06 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071720+0000", 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0719999999999996e-06 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071737+0000", 2024-03-24T12:35:40.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6583999999999999e-05 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071747+0000", 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0244999999999999e-05 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071844+0000", 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6825999999999998e-05 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071853+0000", 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3409999999999992e-06 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214027+0000", 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1421742610000001 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214112+0000", 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4472e-05 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:163984 7.13 7:cd9b75c3:::benchmark_data_smithi029_84282_object63221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.071773+0000", 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.43572482600001, 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1428663079999999, 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 163984 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071773+0000", 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071773+0000", 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071774+0000", 2024-03-24T12:35:40.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0900000000000002e-07 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071781+0000", 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2450000000000001e-06 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071788+0000", 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6000000000000003e-06 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071793+0000", 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7049999999999996e-06 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071881+0000", 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7682999999999994e-05 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071887+0000", 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7699999999999998e-06 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214557+0000", 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1426706010000001 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214639+0000", 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2194999999999995e-05 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164027 7.13 7:cd84f892:::benchmark_data_smithi029_84282_object63264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.107232+0000", 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.40026582300001, 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1125308970000001, 2024-03-24T12:35:40.876 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164027 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107232+0000", 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107232+0000", 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107233+0000", 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9199999999999999e-07 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107237+0000", 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3449999999999999e-06 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107248+0000", 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0278000000000001e-05 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107255+0000", 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8080000000000005e-06 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107313+0000", 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7259999999999997e-05 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107321+0000", 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0020000000000006e-06 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214680+0000", 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.107358882 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214776+0000", 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6278000000000002e-05 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.215116+0000", 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000340012 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219615+0000", 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0044993300000000002 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219629+0000", 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3993e-05 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219666+0000", 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7017000000000002e-05 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219716+0000", 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9487000000000002e-05 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219763+0000", 2024-03-24T12:35:40.882 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7312999999999999e-05 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164078 7.13 7:ccf4d89e:::benchmark_data_smithi029_84282_object63315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.157889+0000", 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.349609152, 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0645090349999999, 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.883 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164078 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157889+0000", 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157889+0000", 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157890+0000", 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.387e-06 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157895+0000", 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4939999999999997e-06 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157908+0000", 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3327e-05 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.885 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157921+0000", 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2719e-05 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157980+0000", 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9626999999999997e-05 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157989+0000", 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5120000000000008e-06 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.217363+0000", 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0593739980000001 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.217463+0000", 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010005600000000001 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.217715+0000", 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025247600000000001 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.887 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222212+0000", 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0044967549999999998 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222265+0000", 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2377999999999999e-05 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222303+0000", 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8455999999999999e-05 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222354+0000", 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1387000000000001e-05 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222398+0000", 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3463e-05 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164090 7.13 7:cef084c6:::benchmark_data_smithi029_84282_object63327:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.163867+0000", 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.34363066, 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0635696459999999, 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164090 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.890 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.163867+0000", 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.163867+0000", 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.163869+0000", 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4920000000000001e-06 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.163874+0000", 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6030000000000004e-06 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.163886+0000", 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1148000000000001e-05 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.163894+0000", 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5299999999999996e-06 2024-03-24T12:35:40.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.163968+0000", 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4035000000000005e-05 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.163977+0000", 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0599999999999997e-06 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219792+0000", 2024-03-24T12:35:40.893 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0558151840000001 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219890+0000", 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7619000000000002e-05 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.220191+0000", 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030105699999999997 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.894 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.227263+0000", 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00707232 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.227304+0000", 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0504999999999999e-05 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.227347+0000", 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3300999999999998e-05 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.895 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.227394+0000", 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6909000000000001e-05 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.227437+0000", 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2883000000000003e-05 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164170 7.13 7:cc7f219c:::benchmark_data_smithi029_84282_object63407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.896 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.239685+0000", 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.26781251700001, 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.99330538000000002, 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164170 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239685+0000", 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239685+0000", 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239687+0000", 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3599999999999999e-06 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239691+0000", 2024-03-24T12:35:40.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2939999999999999e-06 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239703+0000", 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1457000000000001e-05 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239711+0000", 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5369999999999997e-06 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239774+0000", 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.321e-05 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239783+0000", 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6799999999999999e-06 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222425+0000", 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.98264235300000002 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222520+0000", 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4765999999999994e-05 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222797+0000", 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027647000000000001 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.227316+0000", 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0045189119999999999 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.901 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.227475+0000", 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015983699999999999 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.232889+0000", 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0054132859999999998 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.232947+0000", 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8820000000000003e-05 2024-03-24T12:35:40.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.232991+0000", 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3398e-05 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164171 7.13 7:ccb1377c:::benchmark_data_smithi029_84282_object63408:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.239800+0000", 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.26769822700001, 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.99954678399999997, 2024-03-24T12:35:40.903 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164171 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239800+0000", 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239800+0000", 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239800+0000", 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3000000000000001e-07 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239803+0000", 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.88e-06 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.905 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239809+0000", 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0950000000000001e-06 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239815+0000", 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6160000000000001e-06 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239856+0000", 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0952999999999999e-05 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.906 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.239862+0000", 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3260000000000001e-06 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.224683+0000", 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.98482061200000004 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.224801+0000", 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011827799999999999 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.907 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.225153+0000", 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00035243399999999999 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.227486+0000", 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0023322439999999998 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.232812+0000", 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.005326267 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.908 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239243+0000", 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.006430524 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239305+0000", 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2147e-05 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239347+0000", 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1878000000000001e-05 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.909 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164190 7.13 7:cbd176e7:::benchmark_data_smithi029_84282_object63427:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.446525+0000", 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.060972894, 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.79294125100000001, 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164190 2024-03-24T12:35:40.910 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.446525+0000", 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.446525+0000", 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.446526+0000", 2024-03-24T12:35:40.911 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3820000000000001e-06 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.446531+0000", 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0389999999999998e-06 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.446539+0000", 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6100000000000006e-06 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.912 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.446546+0000", 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1219999999999997e-06 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.446606+0000", 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0202999999999999e-05 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.446615+0000", 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2940000000000006e-06 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.227510+0000", 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.780894855 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.227621+0000", 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011150999999999999 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.228071+0000", 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00045017199999999999 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.914 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.232851+0000", 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.004779373 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239124+0000", 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0062731239999999997 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239385+0000", 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026086300000000001 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.915 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239430+0000", 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5241999999999998e-05 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239466+0000", 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6461999999999998e-05 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164204 7.13 7:ce8ae6fe:::benchmark_data_smithi029_84282_object63441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.461449+0000", 2024-03-24T12:35:40.916 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.04604872100001, 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.77813031700000002, 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164204 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.461449+0000", 2024-03-24T12:35:40.917 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.461449+0000", 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.461450+0000", 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.198e-06 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.461456+0000", 2024-03-24T12:35:40.918 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7289999999999997e-06 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.461465+0000", 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4500000000000004e-06 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.461472+0000", 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6320000000000008e-06 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.919 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.461547+0000", 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4488999999999998e-05 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.461558+0000", 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1725999999999999e-05 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.230333+0000", 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.76877420399999996 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.920 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.230433+0000", 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9848999999999999e-05 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.230759+0000", 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00032628399999999998 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.922 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239135+0000", 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0083766399999999994 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239177+0000", 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1563000000000002e-05 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239506+0000", 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00032860000000000002 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.923 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239547+0000", 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1458999999999997e-05 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239580+0000", 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2493999999999998e-05 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164214 7.13 7:cda718f9:::benchmark_data_smithi029_84282_object63451:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.470851+0000", 2024-03-24T12:35:40.924 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.03664710300001, 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.77504884900000004, 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164214 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.470851+0000", 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.925 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.470851+0000", 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.470853+0000", 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7740000000000001e-06 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.470858+0000", 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5470000000000004e-06 2024-03-24T12:35:40.926 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.470870+0000", 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1683e-05 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.470879+0000", 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9099999999999994e-06 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.470972+0000", 2024-03-24T12:35:40.927 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2721999999999998e-05 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.470981+0000", 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4129999999999995e-06 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.233021+0000", 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.76203971199999998 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.233150+0000", 2024-03-24T12:35:40.928 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000129765 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.233459+0000", 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000308323 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239189+0000", 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0057302259999999997 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.929 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239616+0000", 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00042693899999999999 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.245797+0000", 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0061812849999999999 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.245854+0000", 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7191000000000002e-05 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.930 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.245900+0000", 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5358999999999997e-05 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164223 7.13 7:cd96ee34:::benchmark_data_smithi029_84282_object63460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.477004+0000", 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.03049413900001, 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.76913614699999999, 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.931 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164223 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.477004+0000", 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.932 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.477004+0000", 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.477004+0000", 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4899999999999995e-07 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.477007+0000", 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.785e-06 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.933 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.477013+0000", 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2330000000000003e-06 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.477019+0000", 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5729999999999998e-06 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.477081+0000", 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2131999999999994e-05 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.934 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.477090+0000", 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5459999999999992e-06 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.236134+0000", 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.75904452099999997 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.236242+0000", 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000108113 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.935 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.236576+0000", 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00033398500000000002 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239208+0000", 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026312100000000001 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.246017+0000", 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0068091970000000003 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.936 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.246055+0000", 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8745999999999998e-05 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.246103+0000", 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7491000000000003e-05 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.246140+0000", 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7066000000000001e-05 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.937 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164231 7.13 7:cfd747a1:::benchmark_data_smithi029_84282_object63468:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.488174+0000", 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 252.01932446000001, 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.77072931099999997, 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:35:40.938 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164231 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.488174+0000", 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.488174+0000", 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:35:40.939 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.488175+0000", 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.716e-06 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.488181+0000", 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2870000000000004e-06 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.488191+0000", 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0376e-05 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.940 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.488200+0000", 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7199999999999995e-06 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.488344+0000", 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000144024 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.488353+0000", 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2820000000000002e-06 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.941 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239644+0000", 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.75129147900000004 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.239750+0000", 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010566299999999999 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.240810+0000", 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0010603870000000001 2024-03-24T12:35:40.942 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.245971+0000", 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00516082 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.251835+0000", 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0058636469999999996 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.258092+0000", 2024-03-24T12:35:40.943 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0062568299999999997 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.258859+0000", 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00076754199999999996 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.258903+0000", 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3538000000000002e-05 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:35:40.944 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:35:40.945 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:35:40.986 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:40.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:41.066 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:41 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2948609937' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:35:41.066 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:41 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:35:41.067 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:41 smithi029 ceph-mon[41500]: pgmap v386: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-03-24T12:35:41.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:41 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2948609937' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:35:41.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:41 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:35:41.067 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:41 smithi029 ceph-mon[29919]: pgmap v386: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-03-24T12:35:41.069 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:35:41.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:41.127 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:41.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:41.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:41.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:41.264 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:41.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:41.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:41.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:41 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2948609937' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:35:41.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:41 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:35:41.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:41 smithi190 ceph-mon[33748]: pgmap v386: 148 pgs: 148 active+clean; 2.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-03-24T12:35:41.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:41.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:41.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:41.538 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:41.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:41.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:41.678 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:41.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:41.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:41.809 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:41.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:41.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:41.947 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:41.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:42.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:42.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:42.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:42.078 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:42.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:42 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:35:42.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:42 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:35:42.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:42 smithi029 ceph-mon[41500]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T12:35:42.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:42 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:35:42.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:42 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:35:42.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:42 smithi029 ceph-mon[29919]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T12:35:42.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:42.209 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:42.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:42.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:42.350 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:42.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:42 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:35:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:42 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:35:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:42 smithi190 ceph-mon[33748]: osdmap e200: 8 total, 8 up, 8 in 2024-03-24T12:35:42.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:42.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:42.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:42.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:42.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:42.622 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:42.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:42.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:42.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:42.758 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:42.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:42.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:42.887 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:42.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:42.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:42.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:43.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:43.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:43.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:43.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:43.164 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:43.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:43.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:43.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:43.290 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:43.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:43.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:43 smithi190 ceph-mon[33748]: pgmap v388: 148 pgs: 148 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 911 op/s 2024-03-24T12:35:43.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:43.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:43 smithi029 ceph-mon[29919]: pgmap v388: 148 pgs: 148 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 911 op/s 2024-03-24T12:35:43.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:43 smithi029 ceph-mon[41500]: pgmap v388: 148 pgs: 148 active+clean; 2.5 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 911 op/s 2024-03-24T12:35:43.431 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:43.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:43.532 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:43.564 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:43.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:43.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:43.704 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:43.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:43.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:43.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:43.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:43.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:43.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:43.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:43.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:44.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:44.121 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:44.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:44.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:44.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:44.258 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:44.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:44.359 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:44.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:44.398 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:44.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:44.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:35:44.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:44.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:44.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:44.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:44.679 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:44.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:44.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:44.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:44.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:44.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:44.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:44.959 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:44.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:45.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:45.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:45.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:45.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:45.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:45.222 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:45.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:45.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:45.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:45.361 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:45.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:45.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:45.500 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:45.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:45.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:45.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:45.639 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:45.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:45.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:45.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:45.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:45.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:45.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:45 smithi190 ceph-mon[33748]: pgmap v389: 148 pgs: 148 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:35:45.901 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:45.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:45.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:45.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:45 smithi029 ceph-mon[29919]: pgmap v389: 148 pgs: 148 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:35:45.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:45 smithi029 ceph-mon[41500]: pgmap v389: 148 pgs: 148 active+clean; 2.3 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:35:46.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:46.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:46.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:46.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:46.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:46.153 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:46.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:46.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:46.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:46.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:46.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:46.416 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:46.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:46.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:46.541 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:46.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:46.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:46.686 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:46.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:46.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:46.823 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:46.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:46.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:46.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:46 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:35:46.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:46 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:35:46.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:46 smithi029 ceph-mon[41500]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T12:35:46.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:46 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:35:46.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:46 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:35:46.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:46 smithi029 ceph-mon[29919]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T12:35:46.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:46.958 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:46.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:46.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:47.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:47.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:47.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:47.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:47.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:46 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:35:47.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:46 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:35:47.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:46 smithi190 ceph-mon[33748]: osdmap e201: 8 total, 8 up, 8 in 2024-03-24T12:35:47.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:47.221 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:47.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:47.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:47.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:47.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:47.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:47.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:47.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:47.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:47.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:47.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:47.620 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:47.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:47.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:47.763 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:47.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:47.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:47.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:47.888 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:47.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:47.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:47.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:47 smithi029 ceph-mon[29919]: pgmap v391: 148 pgs: 148 active+clean; 2.2 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:35:47.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:47 smithi029 ceph-mon[41500]: pgmap v391: 148 pgs: 148 active+clean; 2.2 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:35:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:47.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:35:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:35:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:35:47.924 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:47 smithi190 ceph-mon[33748]: pgmap v391: 148 pgs: 148 active+clean; 2.2 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.09k op/s 2024-03-24T12:35:47.924 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:35:47.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:48.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:48.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:48.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:48.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:48.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:48.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:48.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:48.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:48.293 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:48.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:48.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:48.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:35:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:35:48.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:48.427 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:48.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:48.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:48.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:48.565 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:48.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:48.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:48.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:48.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:48.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:48.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:48.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:48.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:48.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:48.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:48 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:48.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:48 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:48.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:48.986 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:48.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:49.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:49.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:49.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:48 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:49.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:49.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:49.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:49.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T12:35:49.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:49.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:49.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:49.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:49.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:49.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:49.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:49.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:49.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:49.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:49.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:49.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:49.750 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:49.781 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:49.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:49.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:49.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:49.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:49 smithi029 ceph-mon[29919]: pgmap v392: 148 pgs: 148 active+clean; 2.1 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T12:35:49.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:49 smithi029 ceph-mon[41500]: pgmap v392: 148 pgs: 148 active+clean; 2.1 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T12:35:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:49.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:50.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:50.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:50.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:49 smithi190 ceph-mon[33748]: pgmap v392: 148 pgs: 148 active+clean; 2.1 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T12:35:50.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:50.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:50.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:50.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:50.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:50.300 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:50.327 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:35:50.333 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:35:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:50.347 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:50.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:50.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:50.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:50.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:50.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:50.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:50.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:50.615 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:50.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:50.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:50.745 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:50.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:50.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:50.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:50.893 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:50.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:50.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:51.046 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:51.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:51.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:51.148 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:51.182 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:51.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:51.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:51.315 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:51 smithi029 ceph-mon[41500]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:35:51.315 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:51 smithi029 ceph-mon[41500]: pgmap v393: 148 pgs: 148 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 991 op/s 2024-03-24T12:35:51.315 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:51 smithi029 ceph-mon[29919]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:35:51.315 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:51 smithi029 ceph-mon[29919]: pgmap v393: 148 pgs: 148 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 991 op/s 2024-03-24T12:35:51.318 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:51.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:51.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:51 smithi190 ceph-mon[33748]: from='client.15021 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:35:51.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:51 smithi190 ceph-mon[33748]: pgmap v393: 148 pgs: 148 active+clean; 2.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 991 op/s 2024-03-24T12:35:51.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:51.456 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:51.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:51.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:51.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:51.593 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:51.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:51.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:51.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:51.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:51.735 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:51.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:51.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:51.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:51.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:51.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:52.014 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:52.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:52.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:52.052 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-03-24T12:35:48.564979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2250119542,"num_objects":34524,"num_object_clones":0,"num_object_copies":103572,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34524,"num_whiteouts":0,"num_read":1512,"num_read_kb":1358,"num_write":194005,"num_write_kb":7289310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6323,"num_bytes_recovered":414779306,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86364,"ondisk_log_size":86364,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":9049808,"kb_used_data":6798148,"kb_used_omap":12,"kb_used_meta":2251251,"kb_avail":740911408,"statfs":{"total":767960285184,"available":758693281792,"internally_reserved":0,"allocated":6961303552,"data_stored":6958393883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2305281585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,3,11,4,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-537198592,"num_objects":-8197,"num_object_clones":0,"num_object_copies":-24591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8197,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8197,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002590"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636358+0000","last_change":"2024-03-24T12:35:08.860005+0000","last_active":"2024-03-24T12:35:45.636358+0000","last_peered":"2024-03-24T12:35:45.636358+0000","last_clean":"2024-03-24T12:35:45.636358+0000","last_became_active":"2024-03-24T12:26:09.350521+0000","last_became_peered":"2024-03-24T12:26:09.350521+0000","last_unstale":"2024-03-24T12:35:45.636358+0000","last_undegraded":"2024-03-24T12:35:45.636358+0000","last_fullsized":"2024-03-24T12:35:45.636358+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:08.859869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.859869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.651960+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":304,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.066548+0000","last_change":"2024-03-24T12:34:48.745018+0000","last_active":"2024-03-24T12:35:41.066548+0000","last_peered":"2024-03-24T12:35:41.066548+0000","last_clean":"2024-03-24T12:35:41.066548+0000","last_became_active":"2024-03-24T12:26:07.335517+0000","last_became_peered":"2024-03-24T12:26:07.335517+0000","last_unstale":"2024-03-24T12:35:41.066548+0000","last_undegraded":"2024-03-24T12:35:41.066548+0000","last_fullsized":"2024-03-24T12:35:41.066548+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:48.744937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.744937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.920329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647384+0000","last_change":"2024-03-24T12:35:14.967127+0000","last_active":"2024-03-24T12:35:45.647384+0000","last_peered":"2024-03-24T12:35:45.647384+0000","last_clean":"2024-03-24T12:35:45.647384+0000","last_became_active":"2024-03-24T12:26:03.170822+0000","last_became_peered":"2024-03-24T12:26:03.170822+0000","last_unstale":"2024-03-24T12:35:45.647384+0000","last_undegraded":"2024-03-24T12:35:45.647384+0000","last_fullsized":"2024-03-24T12:35:45.647384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:14.966328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:14.966328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.789519+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":319,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.635977+0000","last_change":"2024-03-24T12:33:12.325849+0000","last_active":"2024-03-24T12:35:45.635977+0000","last_peered":"2024-03-24T12:35:45.635977+0000","last_clean":"2024-03-24T12:35:45.635977+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:35:45.635977+0000","last_undegraded":"2024-03-24T12:35:45.635977+0000","last_fullsized":"2024-03-24T12:35:45.635977+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:12.325694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.325694+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.742360+0000","scrub_duration":140,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":292,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.072687+0000","last_change":"2024-03-24T12:35:07.031107+0000","last_active":"2024-03-24T12:35:41.072687+0000","last_peered":"2024-03-24T12:35:41.072687+0000","last_clean":"2024-03-24T12:35:41.072687+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:35:41.072687+0000","last_undegraded":"2024-03-24T12:35:41.072687+0000","last_fullsized":"2024-03-24T12:35:41.072687+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:07.030987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:07.030987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:21.246152+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.635966+0000","last_change":"2024-03-24T12:35:12.883693+0000","last_active":"2024-03-24T12:35:45.635966+0000","last_peered":"2024-03-24T12:35:45.635966+0000","last_clean":"2024-03-24T12:35:45.635966+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:35:45.635966+0000","last_undegraded":"2024-03-24T12:35:45.635966+0000","last_fullsized":"2024-03-24T12:35:45.635966+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:12.883613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:12.883613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:38.551474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":305,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.075606+0000","last_change":"2024-03-24T12:35:02.993072+0000","last_active":"2024-03-24T12:35:41.075606+0000","last_peered":"2024-03-24T12:35:41.075606+0000","last_clean":"2024-03-24T12:35:41.075606+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:35:41.075606+0000","last_undegraded":"2024-03-24T12:35:41.075606+0000","last_fullsized":"2024-03-24T12:35:41.075606+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:02.992900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.992900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:06.907402+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"90'22","reported_seq":332,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.072837+0000","last_change":"2024-03-24T12:33:36.263856+0000","last_active":"2024-03-24T12:35:41.072837+0000","last_peered":"2024-03-24T12:35:41.072837+0000","last_clean":"2024-03-24T12:35:41.072837+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:35:41.072837+0000","last_undegraded":"2024-03-24T12:35:41.072837+0000","last_fullsized":"2024-03-24T12:35:41.072837+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:33:36.263707+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.263707+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646478+0000","last_change":"2024-03-24T12:34:58.789197+0000","last_active":"2024-03-24T12:35:45.646478+0000","last_peered":"2024-03-24T12:35:45.646478+0000","last_clean":"2024-03-24T12:35:45.646478+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:35:45.646478+0000","last_undegraded":"2024-03-24T12:35:45.646478+0000","last_fullsized":"2024-03-24T12:35:45.646478+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:58.789088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:34:58.789088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:11.317903+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":298,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647962+0000","last_change":"2024-03-24T12:34:44.161766+0000","last_active":"2024-03-24T12:35:45.647962+0000","last_peered":"2024-03-24T12:35:45.647962+0000","last_clean":"2024-03-24T12:35:45.647962+0000","last_became_active":"2024-03-24T12:26:07.331562+0000","last_became_peered":"2024-03-24T12:26:07.331562+0000","last_unstale":"2024-03-24T12:35:45.647962+0000","last_undegraded":"2024-03-24T12:35:45.647962+0000","last_fullsized":"2024-03-24T12:35:45.647962+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:44.161505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.161505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:11.680955+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646883+0000","last_change":"2024-03-24T12:33:33.178127+0000","last_active":"2024-03-24T12:35:45.646883+0000","last_peered":"2024-03-24T12:35:45.646883+0000","last_clean":"2024-03-24T12:35:45.646883+0000","last_became_active":"2024-03-24T12:26:03.161544+0000","last_became_peered":"2024-03-24T12:26:03.161544+0000","last_unstale":"2024-03-24T12:35:45.646883+0000","last_undegraded":"2024-03-24T12:35:45.646883+0000","last_fullsized":"2024-03-24T12:35:45.646883+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:33.177938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:33.177938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":318,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636646+0000","last_change":"2024-03-24T12:33:34.398616+0000","last_active":"2024-03-24T12:35:45.636646+0000","last_peered":"2024-03-24T12:35:45.636646+0000","last_clean":"2024-03-24T12:35:45.636646+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:35:45.636646+0000","last_undegraded":"2024-03-24T12:35:45.636646+0000","last_fullsized":"2024-03-24T12:35:45.636646+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:34.398401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.398401+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.742408+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":294,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646405+0000","last_change":"2024-03-24T12:33:21.155477+0000","last_active":"2024-03-24T12:35:45.646405+0000","last_peered":"2024-03-24T12:35:45.646405+0000","last_clean":"2024-03-24T12:35:45.646405+0000","last_became_active":"2024-03-24T12:26:09.350721+0000","last_became_peered":"2024-03-24T12:26:09.350721+0000","last_unstale":"2024-03-24T12:35:45.646405+0000","last_undegraded":"2024-03-24T12:35:45.646405+0000","last_fullsized":"2024-03-24T12:35:45.646405+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.155345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.155345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":298,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647672+0000","last_change":"2024-03-24T12:33:37.143601+0000","last_active":"2024-03-24T12:35:45.647672+0000","last_peered":"2024-03-24T12:35:45.647672+0000","last_clean":"2024-03-24T12:35:45.647672+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:35:45.647672+0000","last_undegraded":"2024-03-24T12:35:45.647672+0000","last_fullsized":"2024-03-24T12:35:45.647672+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:37.143487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:33:37.143487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646272+0000","last_change":"2024-03-24T12:33:23.095826+0000","last_active":"2024-03-24T12:35:45.646272+0000","last_peered":"2024-03-24T12:35:45.646272+0000","last_clean":"2024-03-24T12:35:45.646272+0000","last_became_active":"2024-03-24T12:26:03.159997+0000","last_became_peered":"2024-03-24T12:26:03.159997+0000","last_unstale":"2024-03-24T12:35:45.646272+0000","last_undegraded":"2024-03-24T12:35:45.646272+0000","last_fullsized":"2024-03-24T12:35:45.646272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:23.095706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:33:23.095706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":320,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636098+0000","last_change":"2024-03-24T12:34:32.424665+0000","last_active":"2024-03-24T12:35:45.636098+0000","last_peered":"2024-03-24T12:35:45.636098+0000","last_clean":"2024-03-24T12:35:45.636098+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:35:45.636098+0000","last_undegraded":"2024-03-24T12:35:45.636098+0000","last_fullsized":"2024-03-24T12:35:45.636098+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T12:34:32.424537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:32.424537+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:00.556829+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636672+0000","last_change":"2024-03-24T12:34:49.652294+0000","last_active":"2024-03-24T12:35:45.636672+0000","last_peered":"2024-03-24T12:35:45.636672+0000","last_clean":"2024-03-24T12:35:45.636672+0000","last_became_active":"2024-03-24T12:26:09.643671+0000","last_became_peered":"2024-03-24T12:26:09.643671+0000","last_unstale":"2024-03-24T12:35:45.636672+0000","last_undegraded":"2024-03-24T12:35:45.636672+0000","last_fullsized":"2024-03-24T12:35:45.636672+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:49.652214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:49.652214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:10.671588+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":312,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.065362+0000","last_change":"2024-03-24T12:35:03.087600+0000","last_active":"2024-03-24T12:35:41.065362+0000","last_peered":"2024-03-24T12:35:41.065362+0000","last_clean":"2024-03-24T12:35:41.065362+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:35:41.065362+0000","last_undegraded":"2024-03-24T12:35:41.065362+0000","last_fullsized":"2024-03-24T12:35:41.065362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:03.087476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:03.087476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:07.618666+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647063+0000","last_change":"2024-03-24T12:34:46.648871+0000","last_active":"2024-03-24T12:35:45.647063+0000","last_peered":"2024-03-24T12:35:45.647063+0000","last_clean":"2024-03-24T12:35:45.647063+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:35:45.647063+0000","last_undegraded":"2024-03-24T12:35:45.647063+0000","last_fullsized":"2024-03-24T12:35:45.647063+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:46.648735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:34:46.648735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:13.075407+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":335,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.643982+0000","last_change":"2024-03-24T12:35:04.954711+0000","last_active":"2024-03-24T12:35:45.643982+0000","last_peered":"2024-03-24T12:35:45.643982+0000","last_clean":"2024-03-24T12:35:45.643982+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:35:45.643982+0000","last_undegraded":"2024-03-24T12:35:45.643982+0000","last_fullsized":"2024-03-24T12:35:45.643982+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:35:04.954560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.954560+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:26.969948+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":293,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.079247+0000","last_change":"2024-03-24T12:34:51.927758+0000","last_active":"2024-03-24T12:35:41.079247+0000","last_peered":"2024-03-24T12:35:41.079247+0000","last_clean":"2024-03-24T12:35:41.079247+0000","last_became_active":"2024-03-24T12:26:09.643738+0000","last_became_peered":"2024-03-24T12:26:09.643738+0000","last_unstale":"2024-03-24T12:35:41.079247+0000","last_undegraded":"2024-03-24T12:35:41.079247+0000","last_fullsized":"2024-03-24T12:35:41.079247+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:51.927644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:51.927644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:03.636261+0000","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636472+0000","last_change":"2024-03-24T12:35:02.853101+0000","last_active":"2024-03-24T12:35:45.636472+0000","last_peered":"2024-03-24T12:35:45.636472+0000","last_clean":"2024-03-24T12:35:45.636472+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:35:45.636472+0000","last_undegraded":"2024-03-24T12:35:45.636472+0000","last_fullsized":"2024-03-24T12:35:45.636472+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:02.852982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.852982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:16.016074+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636740+0000","last_change":"2024-03-24T12:33:18.258903+0000","last_active":"2024-03-24T12:35:45.636740+0000","last_peered":"2024-03-24T12:35:45.636740+0000","last_clean":"2024-03-24T12:35:45.636740+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:35:45.636740+0000","last_undegraded":"2024-03-24T12:35:45.636740+0000","last_fullsized":"2024-03-24T12:35:45.636740+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:18.258782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.258782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.742342+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":315,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.073336+0000","last_change":"2024-03-24T12:33:07.191584+0000","last_active":"2024-03-24T12:35:41.073336+0000","last_peered":"2024-03-24T12:35:41.073336+0000","last_clean":"2024-03-24T12:35:41.073336+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:35:41.073336+0000","last_undegraded":"2024-03-24T12:35:41.073336+0000","last_fullsized":"2024-03-24T12:35:41.073336+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:33:07.191431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.191431+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647807+0000","last_change":"2024-03-24T12:34:42.820621+0000","last_active":"2024-03-24T12:35:45.647807+0000","last_peered":"2024-03-24T12:35:45.647807+0000","last_clean":"2024-03-24T12:35:45.647807+0000","last_became_active":"2024-03-24T12:26:09.343786+0000","last_became_peered":"2024-03-24T12:26:09.343786+0000","last_unstale":"2024-03-24T12:35:45.647807+0000","last_undegraded":"2024-03-24T12:35:45.647807+0000","last_fullsized":"2024-03-24T12:35:45.647807+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:34:42.820487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:42.820487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:56.870887+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":435,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.645545+0000","last_change":"2024-03-24T12:34:49.207292+0000","last_active":"2024-03-24T12:35:45.645545+0000","last_peered":"2024-03-24T12:35:45.645545+0000","last_clean":"2024-03-24T12:35:45.645545+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:35:45.645545+0000","last_undegraded":"2024-03-24T12:35:45.645545+0000","last_fullsized":"2024-03-24T12:35:45.645545+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:34:49.207206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:49.207206+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:18.220803+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.065514+0000","last_change":"2024-03-24T12:33:10.675622+0000","last_active":"2024-03-24T12:35:41.065514+0000","last_peered":"2024-03-24T12:35:41.065514+0000","last_clean":"2024-03-24T12:35:41.065514+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:35:41.065514+0000","last_undegraded":"2024-03-24T12:35:41.065514+0000","last_fullsized":"2024-03-24T12:35:41.065514+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:10.675383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:10.675383+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:42.268767+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":326,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.649022+0000","last_change":"2024-03-24T12:35:05.755602+0000","last_active":"2024-03-24T12:35:45.649022+0000","last_peered":"2024-03-24T12:35:45.649022+0000","last_clean":"2024-03-24T12:35:45.649022+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:35:45.649022+0000","last_undegraded":"2024-03-24T12:35:45.649022+0000","last_fullsized":"2024-03-24T12:35:45.649022+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:35:05.755456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:05.755456+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:18.121034+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":300,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.066250+0000","last_change":"2024-03-24T12:33:03.556099+0000","last_active":"2024-03-24T12:35:41.066250+0000","last_peered":"2024-03-24T12:35:41.066250+0000","last_clean":"2024-03-24T12:35:41.066250+0000","last_became_active":"2024-03-24T12:26:09.348355+0000","last_became_peered":"2024-03-24T12:26:09.348355+0000","last_unstale":"2024-03-24T12:35:41.066250+0000","last_undegraded":"2024-03-24T12:35:41.066250+0000","last_fullsized":"2024-03-24T12:35:41.066250+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:03.555975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.555975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:42.268793+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":291,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.649266+0000","last_change":"2024-03-24T12:33:15.244131+0000","last_active":"2024-03-24T12:35:45.649266+0000","last_peered":"2024-03-24T12:35:45.649266+0000","last_clean":"2024-03-24T12:35:45.649266+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:35:45.649266+0000","last_undegraded":"2024-03-24T12:35:45.649266+0000","last_fullsized":"2024-03-24T12:35:45.649266+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:15.244006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:15.244006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:50.284606+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":305,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.080229+0000","last_change":"2024-03-24T12:33:28.354892+0000","last_active":"2024-03-24T12:35:41.080229+0000","last_peered":"2024-03-24T12:35:41.080229+0000","last_clean":"2024-03-24T12:35:41.080229+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:35:41.080229+0000","last_undegraded":"2024-03-24T12:35:41.080229+0000","last_fullsized":"2024-03-24T12:35:41.080229+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:28.354367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.354367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:46.971125+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":330,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.645521+0000","last_change":"2024-03-24T12:34:43.138957+0000","last_active":"2024-03-24T12:35:45.645521+0000","last_peered":"2024-03-24T12:35:45.645521+0000","last_clean":"2024-03-24T12:35:45.645521+0000","last_became_active":"2024-03-24T12:26:05.317587+0000","last_became_peered":"2024-03-24T12:26:05.317587+0000","last_unstale":"2024-03-24T12:35:45.645521+0000","last_undegraded":"2024-03-24T12:35:45.645521+0000","last_fullsized":"2024-03-24T12:35:45.645521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:34:43.138814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:43.138814+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.385797+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647578+0000","last_change":"2024-03-24T12:34:55.791634+0000","last_active":"2024-03-24T12:35:45.647578+0000","last_peered":"2024-03-24T12:35:45.647578+0000","last_clean":"2024-03-24T12:35:45.647578+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:35:45.647578+0000","last_undegraded":"2024-03-24T12:35:45.647578+0000","last_fullsized":"2024-03-24T12:35:45.647578+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:55.791517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:55.791517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:16.978653+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.644955+0000","last_change":"2024-03-24T12:34:34.270238+0000","last_active":"2024-03-24T12:35:45.644955+0000","last_peered":"2024-03-24T12:35:45.644955+0000","last_clean":"2024-03-24T12:35:45.644955+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:35:45.644955+0000","last_undegraded":"2024-03-24T12:35:45.644955+0000","last_fullsized":"2024-03-24T12:35:45.644955+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:34.270066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:34.270066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:47.337748+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":306,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647605+0000","last_change":"2024-03-24T12:35:04.720760+0000","last_active":"2024-03-24T12:35:45.647605+0000","last_peered":"2024-03-24T12:35:45.647605+0000","last_clean":"2024-03-24T12:35:45.647605+0000","last_became_active":"2024-03-24T12:26:03.170395+0000","last_became_peered":"2024-03-24T12:26:03.170395+0000","last_unstale":"2024-03-24T12:35:45.647605+0000","last_undegraded":"2024-03-24T12:35:45.647605+0000","last_fullsized":"2024-03-24T12:35:45.647605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.720633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.720633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:28.770573+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":307,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.080277+0000","last_change":"2024-03-24T12:33:27.357494+0000","last_active":"2024-03-24T12:35:41.080277+0000","last_peered":"2024-03-24T12:35:41.080277+0000","last_clean":"2024-03-24T12:35:41.080277+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:35:41.080277+0000","last_undegraded":"2024-03-24T12:35:41.080277+0000","last_fullsized":"2024-03-24T12:35:41.080277+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:33:27.355578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:27.355578+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":293,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.072867+0000","last_change":"2024-03-24T12:34:57.998315+0000","last_active":"2024-03-24T12:35:41.072867+0000","last_peered":"2024-03-24T12:35:41.072867+0000","last_clean":"2024-03-24T12:35:41.072867+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:35:41.072867+0000","last_undegraded":"2024-03-24T12:35:41.072867+0000","last_fullsized":"2024-03-24T12:35:41.072867+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:57.998162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:34:57.998162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.432696+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647625+0000","last_change":"2024-03-24T12:35:12.719879+0000","last_active":"2024-03-24T12:35:45.647625+0000","last_peered":"2024-03-24T12:35:45.647625+0000","last_clean":"2024-03-24T12:35:45.647625+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:35:45.647625+0000","last_undegraded":"2024-03-24T12:35:45.647625+0000","last_fullsized":"2024-03-24T12:35:45.647625+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:12.719663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:12.719663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.187535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":313,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646382+0000","last_change":"2024-03-24T12:33:28.059411+0000","last_active":"2024-03-24T12:35:45.646382+0000","last_peered":"2024-03-24T12:35:45.646382+0000","last_clean":"2024-03-24T12:35:45.646382+0000","last_became_active":"2024-03-24T12:26:03.165083+0000","last_became_peered":"2024-03-24T12:26:03.165083+0000","last_unstale":"2024-03-24T12:35:45.646382+0000","last_undegraded":"2024-03-24T12:35:45.646382+0000","last_fullsized":"2024-03-24T12:35:45.646382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:28.059288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.059288+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":320,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.072743+0000","last_change":"2024-03-24T12:34:54.934073+0000","last_active":"2024-03-24T12:35:41.072743+0000","last_peered":"2024-03-24T12:35:41.072743+0000","last_clean":"2024-03-24T12:35:41.072743+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:35:41.072743+0000","last_undegraded":"2024-03-24T12:35:41.072743+0000","last_fullsized":"2024-03-24T12:35:41.072743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:34:54.933925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.933925+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:09.321569+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646250+0000","last_change":"2024-03-24T12:34:53.896024+0000","last_active":"2024-03-24T12:35:45.646250+0000","last_peered":"2024-03-24T12:35:45.646250+0000","last_clean":"2024-03-24T12:35:45.646250+0000","last_became_active":"2024-03-24T12:26:09.350656+0000","last_became_peered":"2024-03-24T12:26:09.350656+0000","last_unstale":"2024-03-24T12:35:45.646250+0000","last_undegraded":"2024-03-24T12:35:45.646250+0000","last_fullsized":"2024-03-24T12:35:45.646250+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:53.895902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.895902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:21.860444+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":297,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.072606+0000","last_change":"2024-03-24T12:33:31.383451+0000","last_active":"2024-03-24T12:35:41.072606+0000","last_peered":"2024-03-24T12:35:41.072606+0000","last_clean":"2024-03-24T12:35:41.072606+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:35:41.072606+0000","last_undegraded":"2024-03-24T12:35:41.072606+0000","last_fullsized":"2024-03-24T12:35:41.072606+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_clean_scrub_stamp":"2024-03-24T12:33:31.383330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":315,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647938+0000","last_change":"2024-03-24T12:35:10.729253+0000","last_active":"2024-03-24T12:35:45.647938+0000","last_peered":"2024-03-24T12:35:45.647938+0000","last_clean":"2024-03-24T12:35:45.647938+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:35:45.647938+0000","last_undegraded":"2024-03-24T12:35:45.647938+0000","last_fullsized":"2024-03-24T12:35:45.647938+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.729138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.729138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:30.421888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":297,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646517+0000","last_change":"2024-03-24T12:35:02.977053+0000","last_active":"2024-03-24T12:35:45.646517+0000","last_peered":"2024-03-24T12:35:45.646517+0000","last_clean":"2024-03-24T12:35:45.646517+0000","last_became_active":"2024-03-24T12:26:16.874170+0000","last_became_peered":"2024-03-24T12:26:16.874170+0000","last_unstale":"2024-03-24T12:35:45.646517+0000","last_undegraded":"2024-03-24T12:35:45.646517+0000","last_fullsized":"2024-03-24T12:35:45.646517+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:35:02.976915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.976915+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.099485+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":293,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.072440+0000","last_change":"2024-03-24T12:34:47.899102+0000","last_active":"2024-03-24T12:35:41.072440+0000","last_peered":"2024-03-24T12:35:41.072440+0000","last_clean":"2024-03-24T12:35:41.072440+0000","last_became_active":"2024-03-24T12:26:09.343296+0000","last_became_peered":"2024-03-24T12:26:09.343296+0000","last_unstale":"2024-03-24T12:35:41.072440+0000","last_undegraded":"2024-03-24T12:35:41.072440+0000","last_fullsized":"2024-03-24T12:35:41.072440+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.898988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.898988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:03.389230+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":305,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.068759+0000","last_change":"2024-03-24T12:33:21.145032+0000","last_active":"2024-03-24T12:35:41.068759+0000","last_peered":"2024-03-24T12:35:41.068759+0000","last_clean":"2024-03-24T12:35:41.068759+0000","last_became_active":"2024-03-24T12:26:07.330811+0000","last_became_peered":"2024-03-24T12:26:07.330811+0000","last_unstale":"2024-03-24T12:35:41.068759+0000","last_undegraded":"2024-03-24T12:35:41.068759+0000","last_fullsized":"2024-03-24T12:35:41.068759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.144883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.144883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:42.268819+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":315,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636163+0000","last_change":"2024-03-24T12:33:17.214730+0000","last_active":"2024-03-24T12:35:45.636163+0000","last_peered":"2024-03-24T12:35:45.636163+0000","last_clean":"2024-03-24T12:35:45.636163+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:35:45.636163+0000","last_undegraded":"2024-03-24T12:35:45.636163+0000","last_fullsized":"2024-03-24T12:35:45.636163+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:17.214655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:17.214655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.742392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"88'16","reported_seq":343,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636257+0000","last_change":"2024-03-24T12:35:04.793820+0000","last_active":"2024-03-24T12:35:45.636257+0000","last_peered":"2024-03-24T12:35:45.636257+0000","last_clean":"2024-03-24T12:35:45.636257+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:35:45.636257+0000","last_undegraded":"2024-03-24T12:35:45.636257+0000","last_fullsized":"2024-03-24T12:35:45.636257+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:35:04.793689+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.793689+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:15.328478+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.648742+0000","last_change":"2024-03-24T12:34:57.796652+0000","last_active":"2024-03-24T12:35:45.648742+0000","last_peered":"2024-03-24T12:35:45.648742+0000","last_clean":"2024-03-24T12:35:45.648742+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:35:45.648742+0000","last_undegraded":"2024-03-24T12:35:45.648742+0000","last_fullsized":"2024-03-24T12:35:45.648742+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:34:57.796520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:57.796520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:23.345910+0000","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":424,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646982+0000","last_change":"2024-03-24T12:35:09.007633+0000","last_active":"2024-03-24T12:35:45.646982+0000","last_peered":"2024-03-24T12:35:45.646982+0000","last_clean":"2024-03-24T12:35:45.646982+0000","last_became_active":"2024-03-24T12:26:07.329487+0000","last_became_peered":"2024-03-24T12:26:07.329487+0000","last_unstale":"2024-03-24T12:35:45.646982+0000","last_undegraded":"2024-03-24T12:35:45.646982+0000","last_fullsized":"2024-03-24T12:35:45.646982+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:35:09.007492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:09.007492+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.347607+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647926+0000","last_change":"2024-03-24T12:34:54.776782+0000","last_active":"2024-03-24T12:35:45.647926+0000","last_peered":"2024-03-24T12:35:45.647926+0000","last_clean":"2024-03-24T12:35:45.647926+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:35:45.647926+0000","last_undegraded":"2024-03-24T12:35:45.647926+0000","last_fullsized":"2024-03-24T12:35:45.647926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:54.776673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.776673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:06.865253+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":316,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.072972+0000","last_change":"2024-03-24T12:34:52.887121+0000","last_active":"2024-03-24T12:35:41.072972+0000","last_peered":"2024-03-24T12:35:41.072972+0000","last_clean":"2024-03-24T12:35:41.072972+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:35:41.072972+0000","last_undegraded":"2024-03-24T12:35:41.072972+0000","last_fullsized":"2024-03-24T12:35:41.072972+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:34:52.886768+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:52.886768+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:55.685280+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":302,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636848+0000","last_change":"2024-03-24T12:35:01.913414+0000","last_active":"2024-03-24T12:35:45.636848+0000","last_peered":"2024-03-24T12:35:45.636848+0000","last_clean":"2024-03-24T12:35:45.636848+0000","last_became_active":"2024-03-24T12:26:09.346062+0000","last_became_peered":"2024-03-24T12:26:09.346062+0000","last_unstale":"2024-03-24T12:35:45.636848+0000","last_undegraded":"2024-03-24T12:35:45.636848+0000","last_fullsized":"2024-03-24T12:35:45.636848+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:01.913287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:01.913287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.142232+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"93'9","reported_seq":415,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.073373+0000","last_change":"2024-03-24T12:33:06.165605+0000","last_active":"2024-03-24T12:35:41.073373+0000","last_peered":"2024-03-24T12:35:41.073373+0000","last_clean":"2024-03-24T12:35:41.073373+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:35:41.073373+0000","last_undegraded":"2024-03-24T12:35:41.073373+0000","last_fullsized":"2024-03-24T12:35:41.073373+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'9","last_scrub_stamp":"2024-03-24T12:33:06.165466+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:33:06.165466+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:46.970812+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647987+0000","last_change":"2024-03-24T12:33:26.284194+0000","last_active":"2024-03-24T12:35:45.647987+0000","last_peered":"2024-03-24T12:35:45.647987+0000","last_clean":"2024-03-24T12:35:45.647987+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:35:45.647987+0000","last_undegraded":"2024-03-24T12:35:45.647987+0000","last_fullsized":"2024-03-24T12:35:45.647987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:26.283885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:26.283885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:50.284575+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"88'13","reported_seq":331,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647089+0000","last_change":"2024-03-24T12:35:08.002157+0000","last_active":"2024-03-24T12:35:45.647089+0000","last_peered":"2024-03-24T12:35:45.647089+0000","last_clean":"2024-03-24T12:35:45.647089+0000","last_became_active":"2024-03-24T12:26:05.358213+0000","last_became_peered":"2024-03-24T12:26:05.358213+0000","last_unstale":"2024-03-24T12:35:45.647089+0000","last_undegraded":"2024-03-24T12:35:45.647089+0000","last_fullsized":"2024-03-24T12:35:45.647089+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:35:08.002032+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.002032+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:17.551062+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.644269+0000","last_change":"2024-03-24T12:34:54.995904+0000","last_active":"2024-03-24T12:35:45.644269+0000","last_peered":"2024-03-24T12:35:45.644269+0000","last_clean":"2024-03-24T12:35:45.644269+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:35:45.644269+0000","last_undegraded":"2024-03-24T12:35:45.644269+0000","last_fullsized":"2024-03-24T12:35:45.644269+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:54.995668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.995668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:21.864194+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":288,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.079155+0000","last_change":"2024-03-24T12:33:18.355672+0000","last_active":"2024-03-24T12:35:41.079155+0000","last_peered":"2024-03-24T12:35:41.079155+0000","last_clean":"2024-03-24T12:35:41.079155+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:35:41.079155+0000","last_undegraded":"2024-03-24T12:35:41.079155+0000","last_fullsized":"2024-03-24T12:35:41.079155+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:18.355449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.355449+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":321,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647606+0000","last_change":"2024-03-24T12:34:50.800957+0000","last_active":"2024-03-24T12:35:45.647606+0000","last_peered":"2024-03-24T12:35:45.647606+0000","last_clean":"2024-03-24T12:35:45.647606+0000","last_became_active":"2024-03-24T12:32:43.256833+0000","last_became_peered":"2024-03-24T12:32:43.256833+0000","last_unstale":"2024-03-24T12:35:45.647606+0000","last_undegraded":"2024-03-24T12:35:45.647606+0000","last_fullsized":"2024-03-24T12:35:45.647606+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:50.800852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:50.800852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:20.220743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":319,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.649166+0000","last_change":"2024-03-24T12:33:18.443409+0000","last_active":"2024-03-24T12:35:45.649166+0000","last_peered":"2024-03-24T12:35:45.649166+0000","last_clean":"2024-03-24T12:35:45.649166+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:35:45.649166+0000","last_undegraded":"2024-03-24T12:35:45.649166+0000","last_fullsized":"2024-03-24T12:35:45.649166+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:18.443276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:18.443276+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:50.284544+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647812+0000","last_change":"2024-03-24T12:35:04.719690+0000","last_active":"2024-03-24T12:35:45.647812+0000","last_peered":"2024-03-24T12:35:45.647812+0000","last_clean":"2024-03-24T12:35:45.647812+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:35:45.647812+0000","last_undegraded":"2024-03-24T12:35:45.647812+0000","last_fullsized":"2024-03-24T12:35:45.647812+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.719553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.719553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:30.298796+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":299,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646404+0000","last_change":"2024-03-24T12:35:10.995737+0000","last_active":"2024-03-24T12:35:45.646404+0000","last_peered":"2024-03-24T12:35:45.646404+0000","last_clean":"2024-03-24T12:35:45.646404+0000","last_became_active":"2024-03-24T12:26:07.329117+0000","last_became_peered":"2024-03-24T12:26:07.329117+0000","last_unstale":"2024-03-24T12:35:45.646404+0000","last_undegraded":"2024-03-24T12:35:45.646404+0000","last_fullsized":"2024-03-24T12:35:45.646404+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.995618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.995618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:39.944517+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":315,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.642933+0000","last_change":"2024-03-24T12:34:47.189025+0000","last_active":"2024-03-24T12:35:45.642933+0000","last_peered":"2024-03-24T12:35:45.642933+0000","last_clean":"2024-03-24T12:35:45.642933+0000","last_became_active":"2024-03-24T12:26:03.161414+0000","last_became_peered":"2024-03-24T12:26:03.161414+0000","last_unstale":"2024-03-24T12:35:45.642933+0000","last_undegraded":"2024-03-24T12:35:45.642933+0000","last_fullsized":"2024-03-24T12:35:45.642933+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.188916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.188916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:49.981138+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":309,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.079337+0000","last_change":"2024-03-24T12:35:15.907723+0000","last_active":"2024-03-24T12:35:41.079337+0000","last_peered":"2024-03-24T12:35:41.079337+0000","last_clean":"2024-03-24T12:35:41.079337+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:35:41.079337+0000","last_undegraded":"2024-03-24T12:35:41.079337+0000","last_fullsized":"2024-03-24T12:35:41.079337+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T12:35:15.907580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.907580+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.415309+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.4","version":"201'12392","reported_seq":12564,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:47.585416+0000","last_change":"2024-03-24T12:34:34.953934+0000","last_active":"2024-03-24T12:35:47.585416+0000","last_peered":"2024-03-24T12:35:47.585416+0000","last_clean":"2024-03-24T12:35:47.585416+0000","last_became_active":"2024-03-24T12:32:41.565218+0000","last_became_peered":"2024-03-24T12:32:41.565218+0000","last_unstale":"2024-03-24T12:35:47.585416+0000","last_undegraded":"2024-03-24T12:35:47.585416+0000","last_fullsized":"2024-03-24T12:35:47.585416+0000","mapping_epoch":158,"log_start":"188'7100","ondisk_log_start":"188'7100","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'7823","last_scrub_stamp":"2024-03-24T12:34:34.953754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:34.953754+0000","objects_scrubbed":6714,"log_size":5292,"log_dups_size":0,"ondisk_log_size":5292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1748,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136183808,"num_objects":2078,"num_object_clones":0,"num_object_copies":6234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12392,"num_write_kb":463040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646741+0000","last_change":"2024-03-24T12:33:34.170601+0000","last_active":"2024-03-24T12:35:45.646741+0000","last_peered":"2024-03-24T12:35:45.646741+0000","last_clean":"2024-03-24T12:35:45.646741+0000","last_became_active":"2024-03-24T12:26:09.643602+0000","last_became_peered":"2024-03-24T12:26:09.643602+0000","last_unstale":"2024-03-24T12:35:45.646741+0000","last_undegraded":"2024-03-24T12:35:45.646741+0000","last_fullsized":"2024-03-24T12:35:45.646741+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.170473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.170473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":319,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.648503+0000","last_change":"2024-03-24T12:34:40.851414+0000","last_active":"2024-03-24T12:35:45.648503+0000","last_peered":"2024-03-24T12:35:45.648503+0000","last_clean":"2024-03-24T12:35:45.648503+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:35:45.648503+0000","last_undegraded":"2024-03-24T12:35:45.648503+0000","last_fullsized":"2024-03-24T12:35:45.648503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:34:40.851280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:40.851280+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:07.220024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":328,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646048+0000","last_change":"2024-03-24T12:33:26.095777+0000","last_active":"2024-03-24T12:35:45.646048+0000","last_peered":"2024-03-24T12:35:45.646048+0000","last_clean":"2024-03-24T12:35:45.646048+0000","last_became_active":"2024-03-24T12:26:05.358217+0000","last_became_peered":"2024-03-24T12:26:05.358217+0000","last_unstale":"2024-03-24T12:35:45.646048+0000","last_undegraded":"2024-03-24T12:35:45.646048+0000","last_fullsized":"2024-03-24T12:35:45.646048+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:33:26.094785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:26.094785+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":435,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647826+0000","last_change":"2024-03-24T12:34:53.783253+0000","last_active":"2024-03-24T12:35:45.647826+0000","last_peered":"2024-03-24T12:35:45.647826+0000","last_clean":"2024-03-24T12:35:45.647826+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:35:45.647826+0000","last_undegraded":"2024-03-24T12:35:45.647826+0000","last_fullsized":"2024-03-24T12:35:45.647826+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:34:53.783165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.783165+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:15.706424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.5","version":"201'12001","reported_seq":12167,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:47.419093+0000","last_change":"2024-03-24T12:34:42.447980+0000","last_active":"2024-03-24T12:35:47.419093+0000","last_peered":"2024-03-24T12:35:47.419093+0000","last_clean":"2024-03-24T12:35:47.419093+0000","last_became_active":"2024-03-24T12:32:41.045244+0000","last_became_peered":"2024-03-24T12:32:41.045244+0000","last_unstale":"2024-03-24T12:35:47.419093+0000","last_undegraded":"2024-03-24T12:35:47.419093+0000","last_fullsized":"2024-03-24T12:35:47.419093+0000","mapping_epoch":158,"log_start":"183'5600","ondisk_log_start":"183'5600","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'7895","last_scrub_stamp":"2024-03-24T12:34:42.447811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:42.447811+0000","objects_scrubbed":6138,"log_size":6401,"log_dups_size":0,"ondisk_log_size":6401,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:59.127848+0000","scrub_duration":1249,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131268608,"num_objects":2003,"num_object_clones":0,"num_object_copies":6009,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2003,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12001,"num_write_kb":448128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":285,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.072777+0000","last_change":"2024-03-24T12:33:20.675885+0000","last_active":"2024-03-24T12:35:41.072777+0000","last_peered":"2024-03-24T12:35:41.072777+0000","last_clean":"2024-03-24T12:35:41.072777+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:35:41.072777+0000","last_undegraded":"2024-03-24T12:35:41.072777+0000","last_fullsized":"2024-03-24T12:35:41.072777+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:20.675649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:20.675649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":267,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.648471+0000","last_change":"2024-03-24T12:35:02.723283+0000","last_active":"2024-03-24T12:35:45.648471+0000","last_peered":"2024-03-24T12:35:45.648471+0000","last_clean":"2024-03-24T12:35:45.648471+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:35:45.648471+0000","last_undegraded":"2024-03-24T12:35:45.648471+0000","last_fullsized":"2024-03-24T12:35:45.648471+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:02.723170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.723170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:26.611016+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"140'17","reported_seq":339,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636184+0000","last_change":"2024-03-24T12:33:30.333141+0000","last_active":"2024-03-24T12:35:45.636184+0000","last_peered":"2024-03-24T12:35:45.636184+0000","last_clean":"2024-03-24T12:35:45.636184+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:35:45.636184+0000","last_undegraded":"2024-03-24T12:35:45.636184+0000","last_fullsized":"2024-03-24T12:35:45.636184+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'17","last_scrub_stamp":"2024-03-24T12:33:30.332974+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:33:30.332974+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.742376+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.643308+0000","last_change":"2024-03-24T12:35:11.997564+0000","last_active":"2024-03-24T12:35:45.643308+0000","last_peered":"2024-03-24T12:35:45.643308+0000","last_clean":"2024-03-24T12:35:45.643308+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:35:45.643308+0000","last_undegraded":"2024-03-24T12:35:45.643308+0000","last_fullsized":"2024-03-24T12:35:45.643308+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:11.997339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:11.997339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.100592+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.6","version":"201'12258","reported_seq":12426,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.931522+0000","last_change":"2024-03-24T12:34:41.649606+0000","last_active":"2024-03-24T12:35:45.931522+0000","last_peered":"2024-03-24T12:35:45.931522+0000","last_clean":"2024-03-24T12:35:45.931522+0000","last_became_active":"2024-03-24T12:32:41.041303+0000","last_became_peered":"2024-03-24T12:32:41.041303+0000","last_unstale":"2024-03-24T12:35:45.931522+0000","last_undegraded":"2024-03-24T12:35:45.931522+0000","last_fullsized":"2024-03-24T12:35:45.931522+0000","mapping_epoch":158,"log_start":"185'6400","ondisk_log_start":"185'6400","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'8059","last_scrub_stamp":"2024-03-24T12:34:41.649439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:41.649439+0000","objects_scrubbed":6312,"log_size":5858,"log_dups_size":0,"ondisk_log_size":5858,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:00.198020+0000","scrub_duration":3527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131465216,"num_objects":2006,"num_object_clones":0,"num_object_copies":6018,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2006,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12258,"num_write_kb":456448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647330+0000","last_change":"2024-03-24T12:34:45.838849+0000","last_active":"2024-03-24T12:35:45.647330+0000","last_peered":"2024-03-24T12:35:45.647330+0000","last_clean":"2024-03-24T12:35:45.647330+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:35:45.647330+0000","last_undegraded":"2024-03-24T12:35:45.647330+0000","last_fullsized":"2024-03-24T12:35:45.647330+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:45.838705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:45.838705+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:02.368329+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.643775+0000","last_change":"2024-03-24T12:35:16.019453+0000","last_active":"2024-03-24T12:35:45.643775+0000","last_peered":"2024-03-24T12:35:45.643775+0000","last_clean":"2024-03-24T12:35:45.643775+0000","last_became_active":"2024-03-24T12:26:07.331416+0000","last_became_peered":"2024-03-24T12:26:07.331416+0000","last_unstale":"2024-03-24T12:35:45.643775+0000","last_undegraded":"2024-03-24T12:35:45.643775+0000","last_fullsized":"2024-03-24T12:35:45.643775+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:16.019255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.019255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:22.005776+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.648242+0000","last_change":"2024-03-24T12:33:34.204979+0000","last_active":"2024-03-24T12:35:45.648242+0000","last_peered":"2024-03-24T12:35:45.648242+0000","last_clean":"2024-03-24T12:35:45.648242+0000","last_became_active":"2024-03-24T12:26:03.165214+0000","last_became_peered":"2024-03-24T12:26:03.165214+0000","last_unstale":"2024-03-24T12:35:45.648242+0000","last_undegraded":"2024-03-24T12:35:45.648242+0000","last_fullsized":"2024-03-24T12:35:45.648242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.204829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":342,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.643414+0000","last_change":"2024-03-24T12:34:51.459238+0000","last_active":"2024-03-24T12:35:45.643414+0000","last_peered":"2024-03-24T12:35:45.643414+0000","last_clean":"2024-03-24T12:35:45.643414+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:35:45.643414+0000","last_undegraded":"2024-03-24T12:35:45.643414+0000","last_fullsized":"2024-03-24T12:35:45.643414+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:34:51.459093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:51.459093+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:54.535337+0000","scrub_duration":248,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"200'12051","reported_seq":12217,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:43.653490+0000","last_change":"2024-03-24T12:34:40.602767+0000","last_active":"2024-03-24T12:35:43.653490+0000","last_peered":"2024-03-24T12:35:43.653490+0000","last_clean":"2024-03-24T12:35:43.653490+0000","last_became_active":"2024-03-24T12:32:41.042042+0000","last_became_peered":"2024-03-24T12:32:41.042042+0000","last_unstale":"2024-03-24T12:35:43.653490+0000","last_undegraded":"2024-03-24T12:35:43.653490+0000","last_fullsized":"2024-03-24T12:35:43.653490+0000","mapping_epoch":158,"log_start":"188'7100","ondisk_log_start":"188'7100","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'8067","last_scrub_stamp":"2024-03-24T12:34:40.602560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:40.602560+0000","objects_scrubbed":6331,"log_size":4951,"log_dups_size":0,"ondisk_log_size":4951,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:09.923583+0000","scrub_duration":3051,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150142976,"num_objects":2291,"num_object_clones":0,"num_object_copies":6873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12051,"num_write_kb":458944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.648330+0000","last_change":"2024-03-24T12:35:09.948925+0000","last_active":"2024-03-24T12:35:45.648330+0000","last_peered":"2024-03-24T12:35:45.648330+0000","last_clean":"2024-03-24T12:35:45.648330+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:35:45.648330+0000","last_undegraded":"2024-03-24T12:35:45.648330+0000","last_fullsized":"2024-03-24T12:35:45.648330+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:09.948402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:09.948402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.433285+0000","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.643510+0000","last_change":"2024-03-24T12:35:10.999806+0000","last_active":"2024-03-24T12:35:45.643510+0000","last_peered":"2024-03-24T12:35:45.643510+0000","last_clean":"2024-03-24T12:35:45.643510+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:35:45.643510+0000","last_undegraded":"2024-03-24T12:35:45.643510+0000","last_fullsized":"2024-03-24T12:35:45.643510+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.999487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.999487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:39.057856+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":322,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646361+0000","last_change":"2024-03-24T12:35:15.994912+0000","last_active":"2024-03-24T12:35:45.646361+0000","last_peered":"2024-03-24T12:35:45.646361+0000","last_clean":"2024-03-24T12:35:45.646361+0000","last_became_active":"2024-03-24T12:26:03.162562+0000","last_became_peered":"2024-03-24T12:26:03.162562+0000","last_unstale":"2024-03-24T12:35:45.646361+0000","last_undegraded":"2024-03-24T12:35:45.646361+0000","last_fullsized":"2024-03-24T12:35:45.646361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:35:15.994784+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.994784+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:16.011588+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":327,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646856+0000","last_change":"2024-03-24T12:33:17.435894+0000","last_active":"2024-03-24T12:35:45.646856+0000","last_peered":"2024-03-24T12:35:45.646856+0000","last_clean":"2024-03-24T12:35:45.646856+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:35:45.646856+0000","last_undegraded":"2024-03-24T12:35:45.646856+0000","last_fullsized":"2024-03-24T12:35:45.646856+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:33:17.435756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:17.435756+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:50.284473+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"201'12137","reported_seq":12304,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:47.577789+0000","last_change":"2024-03-24T12:34:38.651157+0000","last_active":"2024-03-24T12:35:47.577789+0000","last_peered":"2024-03-24T12:35:47.577789+0000","last_clean":"2024-03-24T12:35:47.577789+0000","last_became_active":"2024-03-24T12:32:41.043709+0000","last_became_peered":"2024-03-24T12:32:41.043709+0000","last_unstale":"2024-03-24T12:35:47.577789+0000","last_undegraded":"2024-03-24T12:35:47.577789+0000","last_fullsized":"2024-03-24T12:35:47.577789+0000","mapping_epoch":158,"log_start":"187'6800","ondisk_log_start":"187'6800","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'7870","last_scrub_stamp":"2024-03-24T12:34:38.651008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:38.651008+0000","objects_scrubbed":6332,"log_size":5337,"log_dups_size":0,"ondisk_log_size":5337,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:56.743064+0000","scrub_duration":2447,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130875392,"num_objects":1997,"num_object_clones":0,"num_object_copies":5991,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1997,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12137,"num_write_kb":452288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":293,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.075807+0000","last_change":"2024-03-24T12:35:07.993235+0000","last_active":"2024-03-24T12:35:41.075807+0000","last_peered":"2024-03-24T12:35:41.075807+0000","last_clean":"2024-03-24T12:35:41.075807+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:35:41.075807+0000","last_undegraded":"2024-03-24T12:35:41.075807+0000","last_fullsized":"2024-03-24T12:35:41.075807+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:07.993108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:35:07.993108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:18.965706+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":316,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646638+0000","last_change":"2024-03-24T12:34:53.972192+0000","last_active":"2024-03-24T12:35:45.646638+0000","last_peered":"2024-03-24T12:35:45.646638+0000","last_clean":"2024-03-24T12:35:45.646638+0000","last_became_active":"2024-03-24T12:26:03.165771+0000","last_became_peered":"2024-03-24T12:26:03.165771+0000","last_unstale":"2024-03-24T12:35:45.646638+0000","last_undegraded":"2024-03-24T12:35:45.646638+0000","last_fullsized":"2024-03-24T12:35:45.646638+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:34:53.972073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.972073+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:55.404283+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":315,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636133+0000","last_change":"2024-03-24T12:34:55.729278+0000","last_active":"2024-03-24T12:35:45.636133+0000","last_peered":"2024-03-24T12:35:45.636133+0000","last_clean":"2024-03-24T12:35:45.636133+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:35:45.636133+0000","last_undegraded":"2024-03-24T12:35:45.636133+0000","last_fullsized":"2024-03-24T12:35:45.636133+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:55.729129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:55.729129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:22.046705+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"90'38","reported_seq":476,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.066820+0000","last_change":"2024-03-24T12:32:55.403983+0000","last_active":"2024-03-24T12:35:41.066820+0000","last_peered":"2024-03-24T12:35:41.066820+0000","last_clean":"2024-03-24T12:35:41.066820+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:35:41.066820+0000","last_undegraded":"2024-03-24T12:35:41.066820+0000","last_fullsized":"2024-03-24T12:35:41.066820+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'38","last_scrub_stamp":"2024-03-24T12:32:55.403873+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:32:55.403873+0000","objects_scrubbed":10,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.374116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":299,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647133+0000","last_change":"2024-03-24T12:34:43.416737+0000","last_active":"2024-03-24T12:35:45.647133+0000","last_peered":"2024-03-24T12:35:45.647133+0000","last_clean":"2024-03-24T12:35:45.647133+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:35:45.647133+0000","last_undegraded":"2024-03-24T12:35:45.647133+0000","last_fullsized":"2024-03-24T12:35:45.647133+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:43.416627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:43.416627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:07.477996+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"200'11964","reported_seq":12121,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:44.333842+0000","last_change":"2024-03-24T12:35:03.249200+0000","last_active":"2024-03-24T12:35:44.333842+0000","last_peered":"2024-03-24T12:35:44.333842+0000","last_clean":"2024-03-24T12:35:44.333842+0000","last_became_active":"2024-03-24T12:32:41.045113+0000","last_became_peered":"2024-03-24T12:32:41.045113+0000","last_unstale":"2024-03-24T12:35:44.333842+0000","last_undegraded":"2024-03-24T12:35:44.333842+0000","last_fullsized":"2024-03-24T12:35:44.333842+0000","mapping_epoch":158,"log_start":"188'7100","ondisk_log_start":"188'7100","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'9142","last_scrub_stamp":"2024-03-24T12:35:03.248772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:35:03.248772+0000","objects_scrubbed":5152,"log_size":4864,"log_dups_size":0,"ondisk_log_size":4864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:29.461790+0000","scrub_duration":2292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148504576,"num_objects":2266,"num_object_clones":0,"num_object_copies":6798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11964,"num_write_kb":455360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.648317+0000","last_change":"2024-03-24T12:35:17.750703+0000","last_active":"2024-03-24T12:35:45.648317+0000","last_peered":"2024-03-24T12:35:45.648317+0000","last_clean":"2024-03-24T12:35:45.648317+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:35:45.648317+0000","last_undegraded":"2024-03-24T12:35:45.648317+0000","last_fullsized":"2024-03-24T12:35:45.648317+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:17.750601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:17.750601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:23.838416+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.067359+0000","last_change":"2024-03-24T12:33:19.170350+0000","last_active":"2024-03-24T12:35:41.067359+0000","last_peered":"2024-03-24T12:35:41.067359+0000","last_clean":"2024-03-24T12:35:41.067359+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:35:41.067359+0000","last_undegraded":"2024-03-24T12:35:41.067359+0000","last_fullsized":"2024-03-24T12:35:41.067359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:19.170147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.170147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:42.268933+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":322,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646639+0000","last_change":"2024-03-24T12:34:53.064043+0000","last_active":"2024-03-24T12:35:45.646639+0000","last_peered":"2024-03-24T12:35:45.646639+0000","last_clean":"2024-03-24T12:35:45.646639+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:35:45.646639+0000","last_undegraded":"2024-03-24T12:35:45.646639+0000","last_fullsized":"2024-03-24T12:35:45.646639+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:53.063885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.063885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:00.946389+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"84'18","reported_seq":341,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.066872+0000","last_change":"2024-03-24T12:33:02.509385+0000","last_active":"2024-03-24T12:35:41.066872+0000","last_peered":"2024-03-24T12:35:41.066872+0000","last_clean":"2024-03-24T12:35:41.066872+0000","last_became_active":"2024-03-24T12:26:05.315639+0000","last_became_peered":"2024-03-24T12:26:05.315639+0000","last_unstale":"2024-03-24T12:35:41.066872+0000","last_undegraded":"2024-03-24T12:35:41.066872+0000","last_fullsized":"2024-03-24T12:35:41.066872+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:33:02.509218+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:31:08.237387+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.509218+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:42.268849+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.644035+0000","last_change":"2024-03-24T12:35:13.031821+0000","last_active":"2024-03-24T12:35:45.644035+0000","last_peered":"2024-03-24T12:35:45.644035+0000","last_clean":"2024-03-24T12:35:45.644035+0000","last_became_active":"2024-03-24T12:26:07.343010+0000","last_became_peered":"2024-03-24T12:26:07.343010+0000","last_unstale":"2024-03-24T12:35:45.644035+0000","last_undegraded":"2024-03-24T12:35:45.644035+0000","last_fullsized":"2024-03-24T12:35:45.644035+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:13.031712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:13.031712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:24.248838+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"201'12128","reported_seq":12294,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:47.390374+0000","last_change":"2024-03-24T12:34:42.660114+0000","last_active":"2024-03-24T12:35:47.390374+0000","last_peered":"2024-03-24T12:35:47.390374+0000","last_clean":"2024-03-24T12:35:47.390374+0000","last_became_active":"2024-03-24T12:32:41.042370+0000","last_became_peered":"2024-03-24T12:32:41.042370+0000","last_unstale":"2024-03-24T12:35:47.390374+0000","last_undegraded":"2024-03-24T12:35:47.390374+0000","last_fullsized":"2024-03-24T12:35:47.390374+0000","mapping_epoch":158,"log_start":"183'5700","ondisk_log_start":"183'5700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'8097","last_scrub_stamp":"2024-03-24T12:34:42.659934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:42.659934+0000","objects_scrubbed":6161,"log_size":6428,"log_dups_size":0,"ondisk_log_size":6428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:52.223799+0000","scrub_duration":2450,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134479872,"num_objects":2052,"num_object_clones":0,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12128,"num_write_kb":453760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646386+0000","last_change":"2024-03-24T12:35:18.057402+0000","last_active":"2024-03-24T12:35:45.646386+0000","last_peered":"2024-03-24T12:35:45.646386+0000","last_clean":"2024-03-24T12:35:45.646386+0000","last_became_active":"2024-03-24T12:26:09.347821+0000","last_became_peered":"2024-03-24T12:26:09.347821+0000","last_unstale":"2024-03-24T12:35:45.646386+0000","last_undegraded":"2024-03-24T12:35:45.646386+0000","last_fullsized":"2024-03-24T12:35:45.646386+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:18.057286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:18.057286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:46.147860+0000","scrub_duration":50,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":568,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647439+0000","last_change":"2024-03-24T12:35:16.741056+0000","last_active":"2024-03-24T12:35:45.647439+0000","last_peered":"2024-03-24T12:35:45.647439+0000","last_clean":"2024-03-24T12:35:45.647439+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:35:45.647439+0000","last_undegraded":"2024-03-24T12:35:45.647439+0000","last_fullsized":"2024-03-24T12:35:45.647439+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:35:16.740813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.740813+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.220003+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":305,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.077685+0000","last_change":"2024-03-24T12:33:24.345396+0000","last_active":"2024-03-24T12:35:41.077685+0000","last_peered":"2024-03-24T12:35:41.077685+0000","last_clean":"2024-03-24T12:35:41.077685+0000","last_became_active":"2024-03-24T12:26:03.166467+0000","last_became_peered":"2024-03-24T12:26:03.166467+0000","last_unstale":"2024-03-24T12:35:41.077685+0000","last_undegraded":"2024-03-24T12:35:41.077685+0000","last_fullsized":"2024-03-24T12:35:41.077685+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:24.345279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:24.345279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":450,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.645589+0000","last_change":"2024-03-24T12:34:54.253870+0000","last_active":"2024-03-24T12:35:45.645589+0000","last_peered":"2024-03-24T12:35:45.645589+0000","last_clean":"2024-03-24T12:35:45.645589+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:35:45.645589+0000","last_undegraded":"2024-03-24T12:35:45.645589+0000","last_fullsized":"2024-03-24T12:35:45.645589+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-24T12:34:54.253723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.253723+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:12.669804+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":299,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.647913+0000","last_change":"2024-03-24T12:34:47.802764+0000","last_active":"2024-03-24T12:35:45.647913+0000","last_peered":"2024-03-24T12:35:45.647913+0000","last_clean":"2024-03-24T12:35:45.647913+0000","last_became_active":"2024-03-24T12:26:07.337548+0000","last_became_peered":"2024-03-24T12:26:07.337548+0000","last_unstale":"2024-03-24T12:35:45.647913+0000","last_undegraded":"2024-03-24T12:35:45.647913+0000","last_fullsized":"2024-03-24T12:35:45.647913+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.802609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.802609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.819650+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"200'11650","reported_seq":11810,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:43.663132+0000","last_change":"2024-03-24T12:34:52.725759+0000","last_active":"2024-03-24T12:35:43.663132+0000","last_peered":"2024-03-24T12:35:43.663132+0000","last_clean":"2024-03-24T12:35:43.663132+0000","last_became_active":"2024-03-24T12:32:41.565379+0000","last_became_peered":"2024-03-24T12:32:41.565379+0000","last_unstale":"2024-03-24T12:35:43.663132+0000","last_undegraded":"2024-03-24T12:35:43.663132+0000","last_fullsized":"2024-03-24T12:35:43.663132+0000","mapping_epoch":158,"log_start":"187'6700","ondisk_log_start":"187'6700","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'8328","last_scrub_stamp":"2024-03-24T12:34:52.725350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:52.725350+0000","objects_scrubbed":5683,"log_size":4950,"log_dups_size":0,"ondisk_log_size":4950,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:58.480014+0000","scrub_duration":2941,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":150863872,"num_objects":2302,"num_object_clones":0,"num_object_copies":6906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11650,"num_write_kb":446464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":309,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.064880+0000","last_change":"2024-03-24T12:34:59.974404+0000","last_active":"2024-03-24T12:35:41.064880+0000","last_peered":"2024-03-24T12:35:41.064880+0000","last_clean":"2024-03-24T12:35:41.064880+0000","last_became_active":"2024-03-24T12:26:09.349435+0000","last_became_peered":"2024-03-24T12:26:09.349435+0000","last_unstale":"2024-03-24T12:35:41.064880+0000","last_undegraded":"2024-03-24T12:35:41.064880+0000","last_fullsized":"2024-03-24T12:35:41.064880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:59.974279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:59.974279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:22.996463+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.648174+0000","last_change":"2024-03-24T12:33:34.251764+0000","last_active":"2024-03-24T12:35:45.648174+0000","last_peered":"2024-03-24T12:35:45.648174+0000","last_clean":"2024-03-24T12:35:45.648174+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:35:45.648174+0000","last_undegraded":"2024-03-24T12:35:45.648174+0000","last_fullsized":"2024-03-24T12:35:45.648174+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.251636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.251636+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.251636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:50.284669+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":902,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:46.149460+0000","last_change":"2024-03-24T12:35:15.719490+0000","last_active":"2024-03-24T12:35:46.149460+0000","last_peered":"2024-03-24T12:35:46.149460+0000","last_clean":"2024-03-24T12:35:46.149460+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:35:46.149460+0000","last_undegraded":"2024-03-24T12:35:46.149460+0000","last_fullsized":"2024-03-24T12:35:46.149460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:35:15.719354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.719354+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:18.484367+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":577,"num_read_kb":572,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":321,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.648839+0000","last_change":"2024-03-24T12:34:35.069650+0000","last_active":"2024-03-24T12:35:45.648839+0000","last_peered":"2024-03-24T12:35:45.648839+0000","last_clean":"2024-03-24T12:35:45.648839+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:35:45.648839+0000","last_undegraded":"2024-03-24T12:35:45.648839+0000","last_fullsized":"2024-03-24T12:35:45.648839+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:34:35.069369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:35.069369+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:04.642714+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646769+0000","last_change":"2024-03-24T12:34:52.137065+0000","last_active":"2024-03-24T12:35:45.646769+0000","last_peered":"2024-03-24T12:35:45.646769+0000","last_clean":"2024-03-24T12:35:45.646769+0000","last_became_active":"2024-03-24T12:26:07.326457+0000","last_became_peered":"2024-03-24T12:26:07.326457+0000","last_unstale":"2024-03-24T12:35:45.646769+0000","last_undegraded":"2024-03-24T12:35:45.646769+0000","last_fullsized":"2024-03-24T12:35:45.646769+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:52.136918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:52.136918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:04.242917+0000","scrub_duration":189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"200'12097","reported_seq":12251,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:44.363297+0000","last_change":"2024-03-24T12:35:04.336817+0000","last_active":"2024-03-24T12:35:44.363297+0000","last_peered":"2024-03-24T12:35:44.363297+0000","last_clean":"2024-03-24T12:35:44.363297+0000","last_became_active":"2024-03-24T12:32:41.563612+0000","last_became_peered":"2024-03-24T12:32:41.563612+0000","last_unstale":"2024-03-24T12:35:44.363297+0000","last_undegraded":"2024-03-24T12:35:44.363297+0000","last_fullsized":"2024-03-24T12:35:44.363297+0000","mapping_epoch":158,"log_start":"188'7200","ondisk_log_start":"188'7200","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'9353","last_scrub_stamp":"2024-03-24T12:35:04.336536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.336536+0000","objects_scrubbed":5058,"log_size":4897,"log_dups_size":0,"ondisk_log_size":4897,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:16.783132+0000","scrub_duration":2389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146210816,"num_objects":2231,"num_object_clones":0,"num_object_copies":6693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12097,"num_write_kb":458496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":301,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.066707+0000","last_change":"2024-03-24T12:34:47.787323+0000","last_active":"2024-03-24T12:35:41.066707+0000","last_peered":"2024-03-24T12:35:41.066707+0000","last_clean":"2024-03-24T12:35:41.066707+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:35:41.066707+0000","last_undegraded":"2024-03-24T12:35:41.066707+0000","last_fullsized":"2024-03-24T12:35:41.066707+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.785509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.785509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:02.500786+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.066688+0000","last_change":"2024-03-24T12:34:54.953694+0000","last_active":"2024-03-24T12:35:41.066688+0000","last_peered":"2024-03-24T12:35:41.066688+0000","last_clean":"2024-03-24T12:35:41.066688+0000","last_became_active":"2024-03-24T12:26:03.167244+0000","last_became_peered":"2024-03-24T12:26:03.167244+0000","last_unstale":"2024-03-24T12:35:41.066688+0000","last_undegraded":"2024-03-24T12:35:41.066688+0000","last_fullsized":"2024-03-24T12:35:41.066688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:54.953570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.953570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:05.662761+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"195'27","reported_seq":347,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646971+0000","last_change":"2024-03-24T12:35:16.981335+0000","last_active":"2024-03-24T12:35:45.646971+0000","last_peered":"2024-03-24T12:35:45.646971+0000","last_clean":"2024-03-24T12:35:45.646971+0000","last_became_active":"2024-03-24T12:26:05.356745+0000","last_became_peered":"2024-03-24T12:26:05.356745+0000","last_unstale":"2024-03-24T12:35:45.646971+0000","last_undegraded":"2024-03-24T12:35:45.646971+0000","last_fullsized":"2024-03-24T12:35:45.646971+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'27","last_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_deep_scrub":"195'27","last_deep_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.981148+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:23.598941+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.643495+0000","last_change":"2024-03-24T12:34:44.164236+0000","last_active":"2024-03-24T12:35:45.643495+0000","last_peered":"2024-03-24T12:35:45.643495+0000","last_clean":"2024-03-24T12:35:45.643495+0000","last_became_active":"2024-03-24T12:26:07.337303+0000","last_became_peered":"2024-03-24T12:26:07.337303+0000","last_unstale":"2024-03-24T12:35:45.643495+0000","last_undegraded":"2024-03-24T12:35:45.643495+0000","last_fullsized":"2024-03-24T12:35:45.643495+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:44.164114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.164114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:46.833106+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"201'12365","reported_seq":12529,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.948971+0000","last_change":"2024-03-24T12:34:51.920874+0000","last_active":"2024-03-24T12:35:45.948971+0000","last_peered":"2024-03-24T12:35:45.948971+0000","last_clean":"2024-03-24T12:35:45.948971+0000","last_became_active":"2024-03-24T12:32:41.565563+0000","last_became_peered":"2024-03-24T12:32:41.565563+0000","last_unstale":"2024-03-24T12:35:45.948971+0000","last_undegraded":"2024-03-24T12:35:45.948971+0000","last_fullsized":"2024-03-24T12:35:45.948971+0000","mapping_epoch":158,"log_start":"185'6500","ondisk_log_start":"185'6500","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"191'8699","last_scrub_stamp":"2024-03-24T12:34:51.920651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:51.920651+0000","objects_scrubbed":5891,"log_size":5865,"log_dups_size":0,"ondisk_log_size":5865,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:17.483495+0000","scrub_duration":2177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141885440,"num_objects":2165,"num_object_clones":0,"num_object_copies":6495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2165,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12365,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":287,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646359+0000","last_change":"2024-03-24T12:33:12.332284+0000","last_active":"2024-03-24T12:35:45.646359+0000","last_peered":"2024-03-24T12:35:45.646359+0000","last_clean":"2024-03-24T12:35:45.646359+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:35:45.646359+0000","last_undegraded":"2024-03-24T12:35:45.646359+0000","last_fullsized":"2024-03-24T12:35:45.646359+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:12.332162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:12.332162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:50.284508+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":315,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646963+0000","last_change":"2024-03-24T12:35:00.924220+0000","last_active":"2024-03-24T12:35:45.646963+0000","last_peered":"2024-03-24T12:35:45.646963+0000","last_clean":"2024-03-24T12:35:45.646963+0000","last_became_active":"2024-03-24T12:26:03.168184+0000","last_became_peered":"2024-03-24T12:26:03.168184+0000","last_unstale":"2024-03-24T12:35:45.646963+0000","last_undegraded":"2024-03-24T12:35:45.646963+0000","last_fullsized":"2024-03-24T12:35:45.646963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:00.924106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.924106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:20.399991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"138'15","reported_seq":329,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.068854+0000","last_change":"2024-03-24T12:33:07.518493+0000","last_active":"2024-03-24T12:35:41.068854+0000","last_peered":"2024-03-24T12:35:41.068854+0000","last_clean":"2024-03-24T12:35:41.068854+0000","last_became_active":"2024-03-24T12:26:05.316739+0000","last_became_peered":"2024-03-24T12:26:05.316739+0000","last_unstale":"2024-03-24T12:35:41.068854+0000","last_undegraded":"2024-03-24T12:35:41.068854+0000","last_fullsized":"2024-03-24T12:35:41.068854+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:33:07.518368+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.518368+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:42.268878+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":424,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646658+0000","last_change":"2024-03-24T12:34:55.749516+0000","last_active":"2024-03-24T12:35:45.646658+0000","last_peered":"2024-03-24T12:35:45.646658+0000","last_clean":"2024-03-24T12:35:45.646658+0000","last_became_active":"2024-03-24T12:26:07.336080+0000","last_became_peered":"2024-03-24T12:26:07.336080+0000","last_unstale":"2024-03-24T12:35:45.646658+0000","last_undegraded":"2024-03-24T12:35:45.646658+0000","last_fullsized":"2024-03-24T12:35:45.646658+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:34:55.749381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:55.749381+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:16.630589+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.d","version":"201'12174","reported_seq":12339,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:47.551849+0000","last_change":"2024-03-24T12:34:48.325053+0000","last_active":"2024-03-24T12:35:47.551849+0000","last_peered":"2024-03-24T12:35:47.551849+0000","last_clean":"2024-03-24T12:35:47.551849+0000","last_became_active":"2024-03-24T12:32:41.563736+0000","last_became_peered":"2024-03-24T12:32:41.563736+0000","last_unstale":"2024-03-24T12:35:47.551849+0000","last_undegraded":"2024-03-24T12:35:47.551849+0000","last_fullsized":"2024-03-24T12:35:47.551849+0000","mapping_epoch":158,"log_start":"188'6900","ondisk_log_start":"188'6900","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'8351","last_scrub_stamp":"2024-03-24T12:34:48.324926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.324926+0000","objects_scrubbed":5851,"log_size":5274,"log_dups_size":0,"ondisk_log_size":5274,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:09.073002+0000","scrub_duration":2162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129040408,"num_objects":1970,"num_object_clones":0,"num_object_copies":5910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1970,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12174,"num_write_kb":452545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.648378+0000","last_change":"2024-03-24T12:33:38.172365+0000","last_active":"2024-03-24T12:35:45.648378+0000","last_peered":"2024-03-24T12:35:45.648378+0000","last_clean":"2024-03-24T12:35:45.648378+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:35:45.648378+0000","last_undegraded":"2024-03-24T12:35:45.648378+0000","last_fullsized":"2024-03-24T12:35:45.648378+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:38.172239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:38.172239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":308,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.648935+0000","last_change":"2024-03-24T12:33:29.273583+0000","last_active":"2024-03-24T12:35:45.648935+0000","last_peered":"2024-03-24T12:35:45.648935+0000","last_clean":"2024-03-24T12:35:45.648935+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:35:45.648935+0000","last_undegraded":"2024-03-24T12:35:45.648935+0000","last_fullsized":"2024-03-24T12:35:45.648935+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:29.273325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:29.273325+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:50.284640+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"93'21","reported_seq":339,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636069+0000","last_change":"2024-03-24T12:35:14.829619+0000","last_active":"2024-03-24T12:35:45.636069+0000","last_peered":"2024-03-24T12:35:45.636069+0000","last_clean":"2024-03-24T12:35:45.636069+0000","last_became_active":"2024-03-24T12:26:05.315454+0000","last_became_peered":"2024-03-24T12:26:05.315454+0000","last_unstale":"2024-03-24T12:35:45.636069+0000","last_undegraded":"2024-03-24T12:35:45.636069+0000","last_fullsized":"2024-03-24T12:35:45.636069+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'21","last_scrub_stamp":"2024-03-24T12:35:14.829466+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:35:14.829466+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:35.823180+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":422,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.080563+0000","last_change":"2024-03-24T12:33:35.289329+0000","last_active":"2024-03-24T12:35:41.080563+0000","last_peered":"2024-03-24T12:35:41.080563+0000","last_clean":"2024-03-24T12:35:41.080563+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:35:41.080563+0000","last_undegraded":"2024-03-24T12:35:41.080563+0000","last_fullsized":"2024-03-24T12:35:41.080563+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:33:35.288932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:35.288932+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:46.971193+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.c","version":"201'11968","reported_seq":12139,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.818025+0000","last_change":"2024-03-24T12:34:34.361654+0000","last_active":"2024-03-24T12:35:45.818025+0000","last_peered":"2024-03-24T12:35:45.818025+0000","last_clean":"2024-03-24T12:35:45.818025+0000","last_became_active":"2024-03-24T12:32:41.036596+0000","last_became_peered":"2024-03-24T12:32:41.036596+0000","last_unstale":"2024-03-24T12:35:45.818025+0000","last_undegraded":"2024-03-24T12:35:45.818025+0000","last_fullsized":"2024-03-24T12:35:45.818025+0000","mapping_epoch":158,"log_start":"185'6100","ondisk_log_start":"185'6100","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'7551","last_scrub_stamp":"2024-03-24T12:34:34.361066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:34.361066+0000","objects_scrubbed":6609,"log_size":5868,"log_dups_size":0,"ondisk_log_size":5868,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:52.114579+0000","scrub_duration":2368,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":138674176,"num_objects":2116,"num_object_clones":0,"num_object_copies":6348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11968,"num_write_kb":450688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636452+0000","last_change":"2024-03-24T12:33:35.381730+0000","last_active":"2024-03-24T12:35:45.636452+0000","last_peered":"2024-03-24T12:35:45.636452+0000","last_clean":"2024-03-24T12:35:45.636452+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:35:45.636452+0000","last_undegraded":"2024-03-24T12:35:45.636452+0000","last_fullsized":"2024-03-24T12:35:45.636452+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:35.381614+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:35.381614+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.742327+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":297,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.078942+0000","last_change":"2024-03-24T12:33:15.388058+0000","last_active":"2024-03-24T12:35:41.078942+0000","last_peered":"2024-03-24T12:35:41.078942+0000","last_clean":"2024-03-24T12:35:41.078942+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:35:41.078942+0000","last_undegraded":"2024-03-24T12:35:41.078942+0000","last_fullsized":"2024-03-24T12:35:41.078942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:15.387796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:15.387796+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"74'18","reported_seq":333,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.080439+0000","last_change":"2024-03-24T12:33:33.420555+0000","last_active":"2024-03-24T12:35:41.080439+0000","last_peered":"2024-03-24T12:35:41.080439+0000","last_clean":"2024-03-24T12:35:41.080439+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:35:41.080439+0000","last_undegraded":"2024-03-24T12:35:41.080439+0000","last_fullsized":"2024-03-24T12:35:41.080439+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'18","last_scrub_stamp":"2024-03-24T12:33:33.420394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:33.420394+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.635917+0000","last_change":"2024-03-24T12:34:34.454336+0000","last_active":"2024-03-24T12:35:45.635917+0000","last_peered":"2024-03-24T12:35:45.635917+0000","last_clean":"2024-03-24T12:35:45.635917+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:35:45.635917+0000","last_undegraded":"2024-03-24T12:35:45.635917+0000","last_fullsized":"2024-03-24T12:35:45.635917+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:34.454215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:34.454215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:56.494180+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"200'12018","reported_seq":12181,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:43.657497+0000","last_change":"2024-03-24T12:34:48.526866+0000","last_active":"2024-03-24T12:35:43.657497+0000","last_peered":"2024-03-24T12:35:43.657497+0000","last_clean":"2024-03-24T12:35:43.657497+0000","last_became_active":"2024-03-24T12:32:41.042732+0000","last_became_peered":"2024-03-24T12:32:41.042732+0000","last_unstale":"2024-03-24T12:35:43.657497+0000","last_undegraded":"2024-03-24T12:35:43.657497+0000","last_fullsized":"2024-03-24T12:35:43.657497+0000","mapping_epoch":158,"log_start":"188'7000","ondisk_log_start":"188'7000","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"190'8368","last_scrub_stamp":"2024-03-24T12:34:48.526733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.526733+0000","objects_scrubbed":6006,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:01.870203+0000","scrub_duration":2757,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151257088,"num_objects":2308,"num_object_clones":0,"num_object_copies":6924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12018,"num_write_kb":458432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":295,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646852+0000","last_change":"2024-03-24T12:33:36.210705+0000","last_active":"2024-03-24T12:35:45.646852+0000","last_peered":"2024-03-24T12:35:45.646852+0000","last_clean":"2024-03-24T12:35:45.646852+0000","last_became_active":"2024-03-24T12:26:09.351061+0000","last_became_peered":"2024-03-24T12:26:09.351061+0000","last_unstale":"2024-03-24T12:35:45.646852+0000","last_undegraded":"2024-03-24T12:35:45.646852+0000","last_fullsized":"2024-03-24T12:35:45.646852+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:36.210440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.210440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":61,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"140'9","reported_seq":433,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636764+0000","last_change":"2024-03-24T12:34:44.703519+0000","last_active":"2024-03-24T12:35:45.636764+0000","last_peered":"2024-03-24T12:35:45.636764+0000","last_clean":"2024-03-24T12:35:45.636764+0000","last_became_active":"2024-03-24T12:26:07.328595+0000","last_became_peered":"2024-03-24T12:26:07.328595+0000","last_unstale":"2024-03-24T12:35:45.636764+0000","last_undegraded":"2024-03-24T12:35:45.636764+0000","last_fullsized":"2024-03-24T12:35:45.636764+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'9","last_scrub_stamp":"2024-03-24T12:34:44.703379+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.703379+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:57.223693+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":297,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.080361+0000","last_change":"2024-03-24T12:33:08.247372+0000","last_active":"2024-03-24T12:35:41.080361+0000","last_peered":"2024-03-24T12:35:41.080361+0000","last_clean":"2024-03-24T12:35:41.080361+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:35:41.080361+0000","last_undegraded":"2024-03-24T12:35:41.080361+0000","last_fullsized":"2024-03-24T12:35:41.080361+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:08.247253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.247253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"92'22","reported_seq":318,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.643642+0000","last_change":"2024-03-24T12:34:50.250429+0000","last_active":"2024-03-24T12:35:45.643642+0000","last_peered":"2024-03-24T12:35:45.643642+0000","last_clean":"2024-03-24T12:35:45.643642+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:35:45.643642+0000","last_undegraded":"2024-03-24T12:35:45.643642+0000","last_fullsized":"2024-03-24T12:35:45.643642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-24T12:34:50.250291+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:34:50.250291+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:51.281964+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"201'12133","reported_seq":12291,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:47.410709+0000","last_change":"2024-03-24T12:34:59.406398+0000","last_active":"2024-03-24T12:35:47.410709+0000","last_peered":"2024-03-24T12:35:47.410709+0000","last_clean":"2024-03-24T12:35:47.410709+0000","last_became_active":"2024-03-24T12:32:41.044207+0000","last_became_peered":"2024-03-24T12:32:41.044207+0000","last_unstale":"2024-03-24T12:35:47.410709+0000","last_undegraded":"2024-03-24T12:35:47.410709+0000","last_fullsized":"2024-03-24T12:35:47.410709+0000","mapping_epoch":158,"log_start":"188'7300","ondisk_log_start":"188'7300","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"192'8943","last_scrub_stamp":"2024-03-24T12:34:59.405885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:59.405885+0000","objects_scrubbed":5256,"log_size":4833,"log_dups_size":0,"ondisk_log_size":4833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:18.542470+0000","scrub_duration":1672,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132186112,"num_objects":2017,"num_object_clones":0,"num_object_copies":6051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12133,"num_write_kb":452800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":285,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.072847+0000","last_change":"2024-03-24T12:33:19.377017+0000","last_active":"2024-03-24T12:35:41.072847+0000","last_peered":"2024-03-24T12:35:41.072847+0000","last_clean":"2024-03-24T12:35:41.072847+0000","last_became_active":"2024-03-24T12:26:09.643739+0000","last_became_peered":"2024-03-24T12:26:09.643739+0000","last_unstale":"2024-03-24T12:35:41.072847+0000","last_undegraded":"2024-03-24T12:35:41.072847+0000","last_fullsized":"2024-03-24T12:35:41.072847+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:19.376894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.376894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.646689+0000","last_change":"2024-03-24T12:35:15.004983+0000","last_active":"2024-03-24T12:35:45.646689+0000","last_peered":"2024-03-24T12:35:45.646689+0000","last_clean":"2024-03-24T12:35:45.646689+0000","last_became_active":"2024-03-24T12:26:07.332766+0000","last_became_peered":"2024-03-24T12:26:07.332766+0000","last_unstale":"2024-03-24T12:35:45.646689+0000","last_undegraded":"2024-03-24T12:35:45.646689+0000","last_fullsized":"2024-03-24T12:35:45.646689+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:15.004831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.004831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.429614+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":307,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.635952+0000","last_change":"2024-03-24T12:33:08.165239+0000","last_active":"2024-03-24T12:35:45.635952+0000","last_peered":"2024-03-24T12:35:45.635952+0000","last_clean":"2024-03-24T12:35:45.635952+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:35:45.635952+0000","last_undegraded":"2024-03-24T12:35:45.635952+0000","last_fullsized":"2024-03-24T12:35:45.635952+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:08.165107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:08.165107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.742311+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":330,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636670+0000","last_change":"2024-03-24T12:34:53.872190+0000","last_active":"2024-03-24T12:35:45.636670+0000","last_peered":"2024-03-24T12:35:45.636670+0000","last_clean":"2024-03-24T12:35:45.636670+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:35:45.636670+0000","last_undegraded":"2024-03-24T12:35:45.636670+0000","last_fullsized":"2024-03-24T12:35:45.636670+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:34:53.872101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.872101+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:23.514337+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"200'11971","reported_seq":12138,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:43.671479+0000","last_change":"2024-03-24T12:34:33.907750+0000","last_active":"2024-03-24T12:35:43.671479+0000","last_peered":"2024-03-24T12:35:43.671479+0000","last_clean":"2024-03-24T12:35:43.671479+0000","last_became_active":"2024-03-24T12:32:41.563886+0000","last_became_peered":"2024-03-24T12:32:41.563886+0000","last_unstale":"2024-03-24T12:35:43.671479+0000","last_undegraded":"2024-03-24T12:35:43.671479+0000","last_fullsized":"2024-03-24T12:35:43.671479+0000","mapping_epoch":158,"log_start":"188'7000","ondisk_log_start":"188'7000","created":158,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'7610","last_scrub_stamp":"2024-03-24T12:34:33.907525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:34:33.907525+0000","objects_scrubbed":6738,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:50.675694+0000","scrub_duration":2380,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151715840,"num_objects":2315,"num_object_clones":0,"num_object_copies":6945,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2315,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11971,"num_write_kb":457152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.643802+0000","last_change":"2024-03-24T12:34:48.341578+0000","last_active":"2024-03-24T12:35:45.643802+0000","last_peered":"2024-03-24T12:35:45.643802+0000","last_clean":"2024-03-24T12:35:45.643802+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:35:45.643802+0000","last_undegraded":"2024-03-24T12:35:45.643802+0000","last_fullsized":"2024-03-24T12:35:45.643802+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:48.341457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.341457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:14.005235+0000","scrub_duration":236,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":313,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.066935+0000","last_change":"2024-03-24T12:35:04.121930+0000","last_active":"2024-03-24T12:35:41.066935+0000","last_peered":"2024-03-24T12:35:41.066935+0000","last_clean":"2024-03-24T12:35:41.066935+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:35:41.066935+0000","last_undegraded":"2024-03-24T12:35:41.066935+0000","last_fullsized":"2024-03-24T12:35:41.066935+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.121802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.121802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:28.728829+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":315,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.643272+0000","last_change":"2024-03-24T12:34:50.252743+0000","last_active":"2024-03-24T12:35:45.643272+0000","last_peered":"2024-03-24T12:35:45.643272+0000","last_clean":"2024-03-24T12:35:45.643272+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:35:45.643272+0000","last_undegraded":"2024-03-24T12:35:45.643272+0000","last_fullsized":"2024-03-24T12:35:45.643272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:50.252634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:50.252634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:59.548446+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"94'25","reported_seq":339,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636458+0000","last_change":"2024-03-24T12:34:43.556976+0000","last_active":"2024-03-24T12:35:45.636458+0000","last_peered":"2024-03-24T12:35:45.636458+0000","last_clean":"2024-03-24T12:35:45.636458+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:35:45.636458+0000","last_undegraded":"2024-03-24T12:35:45.636458+0000","last_fullsized":"2024-03-24T12:35:45.636458+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'25","last_scrub_stamp":"2024-03-24T12:34:43.556836+0000","last_deep_scrub":"94'25","last_deep_scrub_stamp":"2024-03-24T12:28:02.436198+0000","last_clean_scrub_stamp":"2024-03-24T12:34:43.556836+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.0","version":"200'12083","reported_seq":24857,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:44.323984+0000","last_change":"2024-03-24T12:35:01.894367+0000","last_active":"2024-03-24T12:35:44.323984+0000","last_peered":"2024-03-24T12:35:44.323984+0000","last_clean":"2024-03-24T12:35:44.323984+0000","last_became_active":"2024-03-24T12:34:49.326034+0000","last_became_peered":"2024-03-24T12:34:49.326034+0000","last_unstale":"2024-03-24T12:35:44.323984+0000","last_undegraded":"2024-03-24T12:35:44.323984+0000","last_fullsized":"2024-03-24T12:35:44.323984+0000","mapping_epoch":190,"log_start":"188'7200","ondisk_log_start":"188'7200","created":158,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'9180","last_scrub_stamp":"2024-03-24T12:35:01.892660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:32:40.019252+0000","last_clean_scrub_stamp":"2024-03-24T12:35:01.892660+0000","objects_scrubbed":5168,"log_size":4883,"log_dups_size":0,"ondisk_log_size":4883,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:06.389031+0000","scrub_duration":1912,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144769024,"num_objects":2209,"num_object_clones":0,"num_object_copies":6627,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2209,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12083,"num_write_kb":457344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6302,"num_bytes_recovered":413007872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":303,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636786+0000","last_change":"2024-03-24T12:33:10.193697+0000","last_active":"2024-03-24T12:35:45.636786+0000","last_peered":"2024-03-24T12:35:45.636786+0000","last_clean":"2024-03-24T12:35:45.636786+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:35:45.636786+0000","last_undegraded":"2024-03-24T12:35:45.636786+0000","last_fullsized":"2024-03-24T12:35:45.636786+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:10.193581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:10.193581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:48.742279+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":433,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.644514+0000","last_change":"2024-03-24T12:35:00.998847+0000","last_active":"2024-03-24T12:35:45.644514+0000","last_peered":"2024-03-24T12:35:45.644514+0000","last_clean":"2024-03-24T12:35:45.644514+0000","last_became_active":"2024-03-24T12:26:07.334798+0000","last_became_peered":"2024-03-24T12:26:07.334798+0000","last_unstale":"2024-03-24T12:35:45.644514+0000","last_undegraded":"2024-03-24T12:35:45.644514+0000","last_fullsized":"2024-03-24T12:35:45.644514+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:35:00.998713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.998713+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:06.501313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":297,"reported_epoch":200,"state":"active+clean","last_fresh":"2024-03-24T12:35:41.072521+0000","last_change":"2024-03-24T12:33:21.383831+0000","last_active":"2024-03-24T12:35:41.072521+0000","last_peered":"2024-03-24T12:35:41.072521+0000","last_clean":"2024-03-24T12:35:41.072521+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:35:41.072521+0000","last_undegraded":"2024-03-24T12:35:41.072521+0000","last_fullsized":"2024-03-24T12:35:41.072521+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.383703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.383703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":326,"reported_epoch":201,"state":"active+clean","last_fresh":"2024-03-24T12:35:45.636342+0000","last_change":"2024-03-24T12:34:48.616552+0000","last_active":"2024-03-24T12:35:45.636342+0000","last_peered":"2024-03-24T12:35:45.636342+0000","last_clean":"2024-03-24T12:35:45.636342+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:35:45.636342+0000","last_undegraded":"2024-03-24T12:35:45.636342+0000","last_fullsized":"2024-03-24T12:35:45.636342+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:34:48.616461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.616461+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:35:58.876656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2249523224,"num_objects":34326,"num_object_clones":0,"num_object_copies":102978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34326,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":193390,"num_write_kb":7286849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6302,"num_bytes_recovered":413007872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6864187392,"data_stored":6864175176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85690,"ondisk_log_size":85690,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":577,"num_read_kb":572,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":464,"ondisk_log_size":464,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037881,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":890976,"kb_used_data":652948,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92854176,"statfs":{"total":95995035648,"available":95082676224,"internally_reserved":0,"allocated":668618752,"data_stored":668246519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":243726794},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332223,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":892804,"kb_used_data":654712,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":92852348,"statfs":{"total":95995035648,"available":95080804352,"internally_reserved":0,"allocated":670425088,"data_stored":670080915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":243726791},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575001,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1435812,"kb_used_data":1170012,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92309340,"statfs":{"total":95995035648,"available":94524764160,"internally_reserved":0,"allocated":1198092288,"data_stored":1197704720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,0,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738526,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1181388,"kb_used_data":829508,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92563764,"statfs":{"total":95995035648,"available":94785294336,"internally_reserved":0,"allocated":849416192,"data_stored":849052698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411476,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1273984,"kb_used_data":904056,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92471168,"statfs":{"total":95995035648,"available":94690476032,"internally_reserved":0,"allocated":925753344,"data_stored":925408111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117132,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1183064,"kb_used_data":863120,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92562088,"statfs":{"total":95995035648,"available":94783578112,"internally_reserved":0,"allocated":883834880,"data_stored":883459526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":327612871},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,1,3,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920905,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":875352,"kb_used_data":673228,"kb_used_omap":1,"kb_used_meta":201918,"kb_avail":92869800,"statfs":{"total":95995035648,"available":95098675200,"internally_reserved":0,"allocated":689385472,"data_stored":689020005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":206764485},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626564,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1316428,"kb_used_data":1050564,"kb_used_omap":1,"kb_used_meta":265790,"kb_avail":92428724,"statfs":{"total":95995035648,"available":94647013376,"internally_reserved":0,"allocated":1075777536,"data_stored":1075421389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":272169413},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":829816832,"data_stored":829816832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1186594816,"data_stored":1186594816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":916393984,"data_stored":916389912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":873922560,"data_stored":873922560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":675282944,"data_stored":675282944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1066602496,"data_stored":1066598424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":662896640,"data_stored":662896640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":652677120,"data_stored":652673048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:35:52.058 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-24T12:35:52.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:52.159 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:52.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:52.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:52.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:52.304 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:52.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:52.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:52.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:52.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:52.488 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:35:52.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:52.604 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:52.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:52.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:52.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:52.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:52.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:52.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:52.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:52.880 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:52.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:52.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:53.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:53.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:53.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:53.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:53.157 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:53.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:53.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:53.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:53.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:53.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:53.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:53.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:53.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:53.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:53.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:53.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:53.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:53.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:53.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:53.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:53.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:53.829 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:53.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:53.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:53.874 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:53 smithi029 ceph-mon[29919]: pgmap v394: 148 pgs: 148 active+clean; 1.8 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:35:53.874 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:53 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:35:53.874 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:53 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:35:53.874 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:53 smithi029 ceph-mon[29919]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T12:35:53.875 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:53 smithi029 ceph-mon[41500]: pgmap v394: 148 pgs: 148 active+clean; 1.8 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:35:53.875 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:53 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:35:53.875 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:53 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:35:53.875 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:53 smithi029 ceph-mon[41500]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T12:35:53.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:53.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:53.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:54.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:54.104 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:54.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:54.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:53 smithi190 ceph-mon[33748]: pgmap v394: 148 pgs: 148 active+clean; 1.8 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-24T12:35:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:53 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:35:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:53 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:35:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:53 smithi190 ceph-mon[33748]: osdmap e202: 8 total, 8 up, 8 in 2024-03-24T12:35:54.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:54.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:54.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:54.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:54.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:54.379 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:54.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:54.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:54.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:54.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:54.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:54.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:54.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:54.650 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:54.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:54.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:54.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:54.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:54.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:54.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:54.910 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:35:54.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:54 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3220335415' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:35:54.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:54 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3220335415' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:35:54.935 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:54.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:55.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:55.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:55.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:54 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3220335415' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:35:55.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:55.214 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:55.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:55.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:55.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:55.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:55.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:55.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:55.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:55.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:55.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:55.633 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:55.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:55.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:55.735 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:55.758 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:55.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:55.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:55.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:55.894 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:55.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:55.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:55.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:55 smithi029 ceph-mon[29919]: pgmap v396: 148 pgs: 148 active+clean; 1.6 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:35:55.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:55 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:35:55.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:55 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3220335415' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:35:55.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:55 smithi029 ceph-mon[29919]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T12:35:55.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:55 smithi029 ceph-mon[41500]: pgmap v396: 148 pgs: 148 active+clean; 1.6 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:35:55.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:55 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:35:55.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:55 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3220335415' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:35:55.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:55 smithi029 ceph-mon[41500]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T12:35:55.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:56.026 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:56.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:56.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:56.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5 2024-03-24T12:35:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:55 smithi190 ceph-mon[33748]: pgmap v396: 148 pgs: 148 active+clean; 1.6 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-03-24T12:35:56.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:55 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:35:56.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:55 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3220335415' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:35:56.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:55 smithi190 ceph-mon[33748]: osdmap e203: 8 total, 8 up, 8 in 2024-03-24T12:35:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:56.167 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:56.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.5' 2024-03-24T12:35:56.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:56.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:56.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:56.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:56.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:56.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:56.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:56.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:56.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:56.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:56.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:56.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:56.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:56.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:56.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:56.836 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:56.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:56.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:56.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:56.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:56.978 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:56.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:57.059 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:35:57.059 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:35:57.060 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:35:57.060 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T12:35:57.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.5 2024-03-24T12:35:57.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:57.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:56 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:35:57.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:56 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:35:57.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:57.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:57.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:57.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:56 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:35:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:56 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:35:57.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:56 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:35:57.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:56 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:35:57.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:57.273 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:57.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:57.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:57.375 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:57.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:35:57 smithi190 systemd[1]: Stopping Ceph osd.5 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:35:57.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:57.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:57.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:57.535 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:57.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:57.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:57.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:57.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:57.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:57.796 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:57.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:57.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:57.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:57 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:35:57.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:57 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:35:57.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:57 smithi190 ceph-mon[33748]: pgmap v398: 148 pgs: 148 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T12:35:57.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:57 smithi190 ceph-mon[33748]: 4.14 scrub starts 2024-03-24T12:35:57.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:57 smithi190 ceph-mon[33748]: 4.14 scrub ok 2024-03-24T12:35:57.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:57 smithi190 ceph-mon[33748]: osd.5 marked itself down and dead 2024-03-24T12:35:57.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:35:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[45195]: 2024-03-24T12:35:57.536+0000 7f1cfabb6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:35:57.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:35:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[45195]: 2024-03-24T12:35:57.536+0000 7f1cfabb6700 -1 osd.5 203 *** Got signal Terminated *** 2024-03-24T12:35:57.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:35:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[45195]: 2024-03-24T12:35:57.536+0000 7f1cfabb6700 -1 osd.5 203 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:35:57.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:57.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:35:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:35:57.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[41500]: pgmap v398: 148 pgs: 148 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[41500]: 4.14 scrub starts 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[41500]: 4.14 scrub ok 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[41500]: osd.5 marked itself down and dead 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[29919]: pgmap v398: 148 pgs: 148 active+clean; 1.6 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[29919]: 4.14 scrub starts 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[29919]: 4.14 scrub ok 2024-03-24T12:35:57.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:57 smithi029 ceph-mon[29919]: osd.5 marked itself down and dead 2024-03-24T12:35:57.920 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:57.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:57.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:58.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:58.046 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:58.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:58.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:58.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:58.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:58.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:58.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:58.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:58.296 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:58.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:58.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:35:58 smithi190 podman[101458]: 2024-03-24 12:35:58.123752761 +0000 UTC m=+0.735193323 container died b7ef4ba1fac8f1fd64ced180fb01cd8b6c68816c3f0369cc4ee51002c42d3ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T12:35:58.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:35:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:35:58.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:58.423 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:58.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:58.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:35:58.552 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:58.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:58.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:35:58.654 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:58.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:58.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:58.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:58.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:35:58 smithi190 podman[101458]: 2024-03-24 12:35:58.602151723 +0000 UTC m=+1.213592313 container cleanup b7ef4ba1fac8f1fd64ced180fb01cd8b6c68816c3f0369cc4ee51002c42d3ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T12:35:58.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:35:58 smithi190 bash[101458]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5 2024-03-24T12:35:58.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:58.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:58.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:58.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:58.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:58.938 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:58.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:58.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:58 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:35:58.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:58 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:35:58.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:58 smithi029 ceph-mon[29919]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T12:35:58.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:58 smithi029 ceph-mon[29919]: osdmap e204: 8 total, 7 up, 8 in 2024-03-24T12:35:58.968 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:58 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:58.970 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:58 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:35:58.970 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:58 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:35:58.970 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:58 smithi029 ceph-mon[41500]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T12:35:58.970 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:58 smithi029 ceph-mon[41500]: osdmap e204: 8 total, 7 up, 8 in 2024-03-24T12:35:58.970 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:58 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:59.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:35:59.090 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:59.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:35:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:58 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:35:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:58 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:35:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:58 smithi190 ceph-mon[33748]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T12:35:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:58 smithi190 ceph-mon[33748]: osdmap e204: 8 total, 7 up, 8 in 2024-03-24T12:35:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:35:59.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:59.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:59.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:59.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:59.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:35:59.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:59.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:35:59.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:35:59.497 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:59.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:59.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:35:59.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:35:59.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:59.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:59.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:35:59.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:35:59 smithi190 podman[101526]: 2024-03-24 12:35:59.429663292 +0000 UTC m=+1.309673748 container remove b7ef4ba1fac8f1fd64ced180fb01cd8b6c68816c3f0369cc4ee51002c42d3ccb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2) 2024-03-24T12:35:59.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:35:59.773 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:35:59.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:59.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:35:59.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:35:59.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:59 smithi029 ceph-mon[41500]: 3.7 scrub starts 2024-03-24T12:35:59.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:59 smithi029 ceph-mon[41500]: 3.7 scrub ok 2024-03-24T12:35:59.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:59 smithi029 ceph-mon[41500]: pgmap v400: 148 pgs: 3 peering, 1 stale+active+clean+scrubbing, 20 stale+active+clean, 124 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:35:59.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:35:59 smithi029 ceph-mon[41500]: osdmap e205: 8 total, 7 up, 8 in 2024-03-24T12:35:59.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:59 smithi029 ceph-mon[29919]: 3.7 scrub starts 2024-03-24T12:35:59.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:59 smithi029 ceph-mon[29919]: 3.7 scrub ok 2024-03-24T12:35:59.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:59 smithi029 ceph-mon[29919]: pgmap v400: 148 pgs: 3 peering, 1 stale+active+clean+scrubbing, 20 stale+active+clean, 124 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:35:59.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:35:59 smithi029 ceph-mon[29919]: osdmap e205: 8 total, 7 up, 8 in 2024-03-24T12:35:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:35:59.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:35:59.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:35:59.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:35:59 smithi190 podman[101753]: 2024-03-24 12:35:59.771113899 +0000 UTC m=+0.022388482 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:35:59.960 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:59 smithi190 ceph-mon[33748]: 3.7 scrub starts 2024-03-24T12:35:59.960 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:59 smithi190 ceph-mon[33748]: 3.7 scrub ok 2024-03-24T12:35:59.960 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:59 smithi190 ceph-mon[33748]: pgmap v400: 148 pgs: 3 peering, 1 stale+active+clean+scrubbing, 20 stale+active+clean, 124 active+clean; 1.4 GiB data, 6.6 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:35:59.960 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:35:59 smithi190 ceph-mon[33748]: osdmap e205: 8 total, 7 up, 8 in 2024-03-24T12:35:59.995 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:35:59.996 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:35:59.996 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:36:00.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:00.004 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:36:00.041 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:00.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:00.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:00.043 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:36:00.084 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:36:00.123 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:36:00.123 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:36:00.123 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:36:00.123 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:36:00.123 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:36:00.123 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:36:00.123 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:36:00.123 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:36:00.123 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:36:00.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:00.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:00.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:00.325 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:00.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:00.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:35:59 smithi190 podman[101753]: 2024-03-24 12:35:59.964499155 +0000 UTC m=+0.215773716 container create e33f6d271d581651f5dc81ab624dea41ff16e0ebace059a2f539a126c8cabbd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:36:00.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:00.449 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:00.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:00.479 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:36:00.479 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:36:00.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:00.592 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:00.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:00.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:00.734 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:00.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:00.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:00.881 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:00.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:00.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:00.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:01.020 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:01.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:01.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:01.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:36:00 smithi190 podman[101753]: 2024-03-24 12:36:00.684984306 +0000 UTC m=+0.936258887 container init e33f6d271d581651f5dc81ab624dea41ff16e0ebace059a2f539a126c8cabbd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T12:36:01.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:36:00 smithi190 podman[101753]: 2024-03-24 12:36:00.695752791 +0000 UTC m=+0.947027363 container start e33f6d271d581651f5dc81ab624dea41ff16e0ebace059a2f539a126c8cabbd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T12:36:01.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:00 smithi190 ceph-mon[33748]: 4.b scrub starts 2024-03-24T12:36:01.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:00 smithi190 ceph-mon[33748]: 4.b scrub ok 2024-03-24T12:36:01.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:00 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:36:01.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:00 smithi190 ceph-mon[33748]: 4.4 scrub ok 2024-03-24T12:36:01.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:00 smithi190 ceph-mon[33748]: 5.1a scrub starts 2024-03-24T12:36:01.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:00 smithi190 ceph-mon[33748]: 5.1a scrub ok 2024-03-24T12:36:01.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:00 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1084139449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:01.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:01.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:01.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[41500]: 4.b scrub starts 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[41500]: 4.b scrub ok 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[41500]: 4.4 scrub ok 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[41500]: 5.1a scrub starts 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[41500]: 5.1a scrub ok 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1084139449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[29919]: 4.b scrub starts 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[29919]: 4.b scrub ok 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[29919]: 4.4 scrub ok 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[29919]: 5.1a scrub starts 2024-03-24T12:36:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[29919]: 5.1a scrub ok 2024-03-24T12:36:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:00 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1084139449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:01.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:36:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:36:01.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:36:01.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:01.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:01.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:01.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:01.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:01.444 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:01.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:01.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:01.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:01.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:01.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:01.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:01.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T12:36:01.678 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:01.767 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:01.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:01.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:01.814 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:01.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:01.870 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:36:01.870 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:36:01.870 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[41500]: pgmap v402: 148 pgs: 3 active+undersized+degraded, 17 peering, 11 stale+active+clean, 7 active+undersized, 110 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 878 op/s; 2486/64500 objects degraded (3.854%) 2024-03-24T12:36:01.870 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[41500]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T12:36:01.870 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[41500]: Health check failed: Degraded data redundancy: 2486/64500 objects degraded (3.854%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:01.870 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[41500]: 3.9 scrub starts 2024-03-24T12:36:01.871 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[41500]: 3.9 scrub ok 2024-03-24T12:36:01.871 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[41500]: 4.1d scrub starts 2024-03-24T12:36:01.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:36:01.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:36:01.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[29919]: pgmap v402: 148 pgs: 3 active+undersized+degraded, 17 peering, 11 stale+active+clean, 7 active+undersized, 110 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 878 op/s; 2486/64500 objects degraded (3.854%) 2024-03-24T12:36:01.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[29919]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T12:36:01.871 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[29919]: Health check failed: Degraded data redundancy: 2486/64500 objects degraded (3.854%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:01.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[29919]: 3.9 scrub starts 2024-03-24T12:36:01.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[29919]: 3.9 scrub ok 2024-03-24T12:36:01.872 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:01 smithi029 ceph-mon[29919]: 4.1d scrub starts 2024-03-24T12:36:01.912 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:01.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:01.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:02.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:02.042 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:02.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:02.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:02.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:36:01 smithi190 podman[101753]: 2024-03-24 12:36:01.793881274 +0000 UTC m=+2.045155848 container attach e33f6d271d581651f5dc81ab624dea41ff16e0ebace059a2f539a126c8cabbd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-deactivate, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T12:36:02.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:36:01 smithi190 podman[101753]: 2024-03-24 12:36:01.794413742 +0000 UTC m=+2.045688298 container died e33f6d271d581651f5dc81ab624dea41ff16e0ebace059a2f539a126c8cabbd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T12:36:02.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:01 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:36:02.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:01 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:36:02.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:01 smithi190 ceph-mon[33748]: pgmap v402: 148 pgs: 3 active+undersized+degraded, 17 peering, 11 stale+active+clean, 7 active+undersized, 110 active+clean; 1.3 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 878 op/s; 2486/64500 objects degraded (3.854%) 2024-03-24T12:36:02.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:01 smithi190 ceph-mon[33748]: Health check failed: Reduced data availability: 3 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T12:36:02.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:01 smithi190 ceph-mon[33748]: Health check failed: Degraded data redundancy: 2486/64500 objects degraded (3.854%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:02.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:01 smithi190 ceph-mon[33748]: 3.9 scrub starts 2024-03-24T12:36:02.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:01 smithi190 ceph-mon[33748]: 3.9 scrub ok 2024-03-24T12:36:02.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:01 smithi190 ceph-mon[33748]: 4.1d scrub starts 2024-03-24T12:36:02.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:02.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:02.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:02.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:02.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:02.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:02.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:02.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:02.476 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:02.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:02.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:02.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:02.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:02.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:02.717 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:02.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T12:36:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:02.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:02.790 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:36:02.790 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T12:36:02.790 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2024-03-24T12:36:02.790 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi190.front.sepia.ceph.com 2024-03-24T12:36:02.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-03-24T12:36:02.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:02.876 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:02.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:02.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:36:02.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:36:02.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[29919]: 4.1d scrub ok 2024-03-24T12:36:02.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:36:02.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:36:02.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[29919]: 5.14 scrub starts 2024-03-24T12:36:02.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[29919]: 5.14 scrub ok 2024-03-24T12:36:02.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:02.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:36:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:36:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[41500]: 4.1d scrub ok 2024-03-24T12:36:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:36:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:36:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[41500]: 5.14 scrub starts 2024-03-24T12:36:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[41500]: 5.14 scrub ok 2024-03-24T12:36:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:02.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:02 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:02.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:03.015 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:36:03.017 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:03.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:03.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:03.115 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:03.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:03.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:02 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:36:03.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:02 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:36:03.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:02 smithi190 ceph-mon[33748]: 4.1d scrub ok 2024-03-24T12:36:03.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:02 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:36:03.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:02 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:36:03.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:02 smithi190 ceph-mon[33748]: 5.14 scrub starts 2024-03-24T12:36:03.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:02 smithi190 ceph-mon[33748]: 5.14 scrub ok 2024-03-24T12:36:03.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:02 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:03.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:02 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:03.150 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:03.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:03.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:03.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:36:03.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:03.332 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:03.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:03.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:03.434 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:03.482 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:03.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:03.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:03.630 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:03.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:03.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:03.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:03.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:03.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:03.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:03 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:36:03.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:03 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:36:03.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:03 smithi190 ceph-mon[33748]: pgmap v403: 148 pgs: 17 active+undersized+degraded, 17 peering, 23 active+undersized, 91 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 129 B/s rd, 0 B/s wr, 945 op/s; 6734/57942 objects degraded (11.622%) 2024-03-24T12:36:03.905 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:03.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:03.952 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:03 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:36:03.952 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:03 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:36:03.952 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:03 smithi029 ceph-mon[41500]: pgmap v403: 148 pgs: 17 active+undersized+degraded, 17 peering, 23 active+undersized, 91 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 129 B/s rd, 0 B/s wr, 945 op/s; 6734/57942 objects degraded (11.622%) 2024-03-24T12:36:03.953 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:03 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:36:03.953 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:03 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:36:03.953 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:03 smithi029 ceph-mon[29919]: pgmap v403: 148 pgs: 17 active+undersized+degraded, 17 peering, 23 active+undersized, 91 active+clean; 1.2 GiB data, 6.1 GiB used, 709 GiB / 715 GiB avail; 129 B/s rd, 0 B/s wr, 945 op/s; 6734/57942 objects degraded (11.622%) 2024-03-24T12:36:04.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:04.034 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:04.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:04.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:04.177 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:04.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:04.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:04.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:04.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:04.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:04.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:04.322 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:36:04.412 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:04.435 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:04.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:04.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:04.579 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:04.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:04.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:04.710 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:04.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:04.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:04.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:04.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: 8.5 scrub starts 2024-03-24T12:36:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:36:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: 8.5 scrub ok 2024-03-24T12:36:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:36:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:36:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:36:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:36:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:36:04.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: 6.3 scrub starts 2024-03-24T12:36:04.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: 6.3 scrub ok 2024-03-24T12:36:04.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:36:04.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:04 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: 8.5 scrub starts 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: 8.5 scrub ok 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: 6.3 scrub starts 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: 6.3 scrub ok 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:36:04.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: 8.5 scrub starts 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: 8.5 scrub ok 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: 6.3 scrub starts 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: 6.3 scrub ok 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:36:04.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:04 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:36:04.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:04.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:04.997 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:04.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:05.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:05.141 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:05.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:05.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:05.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:05.280 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:05.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:05.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:05.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:05.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:05.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:05.545 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:05.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:05.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:05.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:05.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:05.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:05.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:05.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:05.817 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:05.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:05.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:05.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:05.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:05.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:05.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:36:05.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:36:05.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[29919]: 3.12 scrub starts 2024-03-24T12:36:05.979 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[29919]: 3.12 scrub ok 2024-03-24T12:36:05.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[29919]: pgmap v404: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1023 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 127 B/s rd, 0 B/s wr, 1.13k op/s; 8091/49677 objects degraded (16.287%) 2024-03-24T12:36:05.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[29919]: 3.1a scrub starts 2024-03-24T12:36:05.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[29919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-03-24T12:36:05.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[29919]: osdmap e206: 8 total, 7 up, 8 in 2024-03-24T12:36:05.980 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[29919]: 3.1a scrub ok 2024-03-24T12:36:05.980 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:36:05.980 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:36:05.980 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[41500]: 3.12 scrub starts 2024-03-24T12:36:05.980 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[41500]: 3.12 scrub ok 2024-03-24T12:36:05.980 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[41500]: pgmap v404: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1023 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 127 B/s rd, 0 B/s wr, 1.13k op/s; 8091/49677 objects degraded (16.287%) 2024-03-24T12:36:05.981 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[41500]: 3.1a scrub starts 2024-03-24T12:36:05.981 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[41500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-03-24T12:36:05.981 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[41500]: osdmap e206: 8 total, 7 up, 8 in 2024-03-24T12:36:05.981 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:05 smithi029 ceph-mon[41500]: 3.1a scrub ok 2024-03-24T12:36:06.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:06.108 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:06.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:05 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:36:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:05 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:36:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:05 smithi190 ceph-mon[33748]: 3.12 scrub starts 2024-03-24T12:36:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:05 smithi190 ceph-mon[33748]: 3.12 scrub ok 2024-03-24T12:36:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:05 smithi190 ceph-mon[33748]: pgmap v404: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1023 MiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 127 B/s rd, 0 B/s wr, 1.13k op/s; 8091/49677 objects degraded (16.287%) 2024-03-24T12:36:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:05 smithi190 ceph-mon[33748]: 3.1a scrub starts 2024-03-24T12:36:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:05 smithi190 ceph-mon[33748]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-03-24T12:36:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:05 smithi190 ceph-mon[33748]: osdmap e206: 8 total, 7 up, 8 in 2024-03-24T12:36:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:05 smithi190 ceph-mon[33748]: 3.1a scrub ok 2024-03-24T12:36:06.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:06.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:06.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:06.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:06.339 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:06.347 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:36:06.347 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:36:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:06.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:06.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:06.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:06.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:06.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:06.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:06.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:06.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:06.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:36:06.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:06.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:06.817 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:06.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:06.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:06.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:06.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:06.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:07.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:07.086 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:07.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:07.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:06 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1811669441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:36:07.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:06 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1811669441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:36:07.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:06 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1811669441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:36:07.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:07.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:07.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:07.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:07.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:07.357 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:07.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:07.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:07.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:07.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:07.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:07.577 INFO:teuthology.orchestra.run.smithi190.stdout:8.a 2024-03-24T12:36:07.577 INFO:teuthology.orchestra.run.smithi190.stdout:8.9 2024-03-24T12:36:07.577 INFO:teuthology.orchestra.run.smithi190.stdout:8.4 2024-03-24T12:36:07.577 INFO:teuthology.orchestra.run.smithi190.stdout:8.0 2024-03-24T12:36:07.577 INFO:teuthology.orchestra.run.smithi190.stdout:6.d 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:6.c 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:6.4 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:6.1e 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:8.3 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:6.1c 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:6.7 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:6.11 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:4.1 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:6.1a 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:4.12 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:6.a 2024-03-24T12:36:07.578 INFO:teuthology.orchestra.run.smithi190.stdout:5.2 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:3.d 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:6.1f 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:3.4 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:4.15 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:3.1c 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:3.c 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:4.8 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:3.2 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:5.16 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:3.1f 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:5.7 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:3.5 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:8.2 2024-03-24T12:36:07.579 INFO:teuthology.orchestra.run.smithi190.stdout:3.10 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:3.1d 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:3.17 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:8.d 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:5.1b 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:2.2 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:3.16 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:4.1f 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:6.1d 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:5.d 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:4.3 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:4.7 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:5.12 2024-03-24T12:36:07.580 INFO:teuthology.orchestra.run.smithi190.stdout:5.15 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:5.3 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:5.19 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:6.19 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:5.4 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:4.2 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:5.6 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:5.b 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:5.f 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:3.11 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:5.e 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:8.8 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:6.17 2024-03-24T12:36:07.581 INFO:teuthology.orchestra.run.smithi190.stdout:6.10 2024-03-24T12:36:07.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:07.607 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:36:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:36:07.630 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:07.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:07.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:07.759 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:07.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:07.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:07.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T12:36:07.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: 5.1e scrub starts 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: 5.1e scrub ok 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: pgmap v406: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 957 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 8091/46521 objects degraded (17.392%) 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 8091/46521 objects degraded (17.392%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1811669441' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: osdmap e207: 8 total, 7 up, 8 in 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:36:07.880 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: 5.1e scrub starts 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: 5.1e scrub ok 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: pgmap v406: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 957 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 8091/46521 objects degraded (17.392%) 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 8091/46521 objects degraded (17.392%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1811669441' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: osdmap e207: 8 total, 7 up, 8 in 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:36:07.881 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:07 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:36:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: 5.1e scrub starts 2024-03-24T12:36:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: 5.1e scrub ok 2024-03-24T12:36:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:36:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:36:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: pgmap v406: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 957 MiB data, 5.6 GiB used, 710 GiB / 715 GiB avail; 0 B/s wr, 1.01k op/s; 8091/46521 objects degraded (17.392%) 2024-03-24T12:36:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 8091/46521 objects degraded (17.392%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:36:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1811669441' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:36:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: osdmap e207: 8 total, 7 up, 8 in 2024-03-24T12:36:07.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:36:07.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:36:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:07.908 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:07.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:08.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:08.060 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:08.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:08.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:08.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:08.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:08.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:08.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:08.328 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:08.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:08.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:08.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:36:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:36:08.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:08.454 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:08.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:08.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:08.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:08.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:08.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:08.724 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:08.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:08.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:08.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 4.1f --file /var/log/ceph/exp.4.1f.5 2024-03-24T12:36:08.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:08.886 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:08.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:08.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:08.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:09.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:09.012 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:09.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:09.014 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:36:09.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:08 smithi029 ceph-mon[29919]: 8.1 scrub starts 2024-03-24T12:36:09.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:08 smithi029 ceph-mon[29919]: 8.1 scrub ok 2024-03-24T12:36:09.090 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:09.090 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:08 smithi029 ceph-mon[41500]: 8.1 scrub starts 2024-03-24T12:36:09.090 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:08 smithi029 ceph-mon[41500]: 8.1 scrub ok 2024-03-24T12:36:09.090 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:09.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:09.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:08 smithi190 ceph-mon[33748]: 8.1 scrub starts 2024-03-24T12:36:09.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:08 smithi190 ceph-mon[33748]: 8.1 scrub ok 2024-03-24T12:36:09.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:09.152 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:09.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:09.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:09.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:09.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:09.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:09.440 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:09.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:09.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:09.542 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:09.564 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:09.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:09.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:09.666 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:09.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:09.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:09.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:09.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:09.831 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:09.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:09 smithi029 ceph-mon[41500]: pgmap v408: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 843 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 127 B/s rd, 0 B/s wr, 978 op/s; 6612/41019 objects degraded (16.119%) 2024-03-24T12:36:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:09 smithi029 ceph-mon[41500]: 3.8 scrub starts 2024-03-24T12:36:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:09 smithi029 ceph-mon[41500]: 3.8 scrub ok 2024-03-24T12:36:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:09 smithi029 ceph-mon[29919]: pgmap v408: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 843 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 127 B/s rd, 0 B/s wr, 978 op/s; 6612/41019 objects degraded (16.119%) 2024-03-24T12:36:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:09 smithi029 ceph-mon[29919]: 3.8 scrub starts 2024-03-24T12:36:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:09 smithi029 ceph-mon[29919]: 3.8 scrub ok 2024-03-24T12:36:09.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:09.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:09.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:10.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:10.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:10.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:10.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:10.123 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:09 smithi190 ceph-mon[33748]: pgmap v408: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 843 MiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 127 B/s rd, 0 B/s wr, 978 op/s; 6612/41019 objects degraded (16.119%) 2024-03-24T12:36:10.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:09 smithi190 ceph-mon[33748]: 3.8 scrub starts 2024-03-24T12:36:10.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:09 smithi190 ceph-mon[33748]: 3.8 scrub ok 2024-03-24T12:36:10.188 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:36:10.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:10.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:10.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:10.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:10.377 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:10.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:10.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:10.502 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:10.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:10.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:10.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:10.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:10.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:10.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:10.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:10.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:10.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:10.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:10.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:10.942 INFO:teuthology.orchestra.run.smithi190.stderr:Exporting 4.1f info 4.1f( v 74'11 (0'0,74'11] local-lis/les=68/69 n=5 ec=68/68 lis/c=68/68 les/c/f=69/69/0 sis=68) 2024-03-24T12:36:10.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:f93251ce:::obj_delete_at_hint.0000000042:head# 2024-03-24T12:36:10.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:fce5a501:::obj_delete_at_hint.0000000075:head# 2024-03-24T12:36:10.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:fd6b5df8:::obj_delete_at_hint.0000000120:head# 2024-03-24T12:36:10.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:fd8d39b6:::obj_delete_at_hint.0000000030:head# 2024-03-24T12:36:10.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #4:fde4dd55:gc::gc.26:head# 2024-03-24T12:36:10.943 INFO:teuthology.orchestra.run.smithi190.stderr:Export successful 2024-03-24T12:36:11.013 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:11.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:10 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:36:11.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:10 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:36:11.051 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:11.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:11.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:10 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:36:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:10 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:36:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:10 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:36:11.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:10 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:36:11.176 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:11.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:11.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:11.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:11.302 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:11.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:11.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:11.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:11.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:11.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:11.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:11.564 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:11.565 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:11.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:11.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:11.704 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:11.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:11.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:11.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:11.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:11.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:11.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:11.964 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:11.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:11.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:12.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:12.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:12.106 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:12.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:12.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:11 smithi190 ceph-mon[33748]: pgmap v409: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 665 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s; 5168/32481 objects degraded (15.911%) 2024-03-24T12:36:12.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:11 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:36:12.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:11 smithi190 ceph-mon[33748]: 3.13 scrub ok 2024-03-24T12:36:12.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:11 smithi029 ceph-mon[41500]: pgmap v409: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 665 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s; 5168/32481 objects degraded (15.911%) 2024-03-24T12:36:12.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:11 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:36:12.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:11 smithi029 ceph-mon[41500]: 3.13 scrub ok 2024-03-24T12:36:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:11 smithi029 ceph-mon[29919]: pgmap v409: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 665 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.06k op/s; 5168/32481 objects degraded (15.911%) 2024-03-24T12:36:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:11 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:36:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:11 smithi029 ceph-mon[29919]: 3.13 scrub ok 2024-03-24T12:36:12.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 4.1f 2024-03-24T12:36:12.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:12.232 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:12.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:12.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:12.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:12.371 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:12.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:12.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:12.450 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:36:12.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:12.508 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:12.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:12.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:12.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:12.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:12.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:12.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:12.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:12.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:12.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:12.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:12.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:12 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 4078/25782 objects degraded (15.817%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:12.902 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:12.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:13.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:13.042 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:13.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:13.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:13.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:13.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:12 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 4078/25782 objects degraded (15.817%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:13.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:12 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 4078/25782 objects degraded (15.817%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:13.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:13.180 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:13.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:13.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:13.319 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:13.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:13.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:13.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:13.445 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:13.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:13.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:13.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:13.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:13.583 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:13.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:13.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:13.728 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:13.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:13.769 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:36:13.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:13.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:13.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:13.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:13.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:14.004 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:14.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:14.057 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:13 smithi190 ceph-mon[33748]: pgmap v410: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 525 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 1.03k op/s; 4078/25782 objects degraded (15.817%) 2024-03-24T12:36:14.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:13 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:36:14.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:13 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:36:14.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:13 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:36:14.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:13 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:36:14.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:13 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:36:14.058 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:13 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:36:14.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:14.142 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:14.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:14.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:14.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[41500]: pgmap v410: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 525 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 1.03k op/s; 4078/25782 objects degraded (15.817%) 2024-03-24T12:36:14.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:36:14.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:36:14.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:36:14.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:36:14.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:36:14.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:36:14.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[29919]: pgmap v410: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 525 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 1.03k op/s; 4078/25782 objects degraded (15.817%) 2024-03-24T12:36:14.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:36:14.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:36:14.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:36:14.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:36:14.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:36:14.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:13 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:36:14.247 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:14.271 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:14.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:14.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:14.373 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:14.405 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:14.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:14.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:14.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:14.540 INFO:teuthology.orchestra.run.smithi190.stdout: marking collection for removal 2024-03-24T12:36:14.541 INFO:teuthology.orchestra.run.smithi190.stdout:setting '_remove' omap key 2024-03-24T12:36:14.541 INFO:teuthology.orchestra.run.smithi190.stdout:finish_remove_pgs 4.1f_head removing 4.1f 2024-03-24T12:36:14.542 INFO:teuthology.orchestra.run.smithi190.stdout:Remove successful 2024-03-24T12:36:14.547 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:14.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:14.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:14.673 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:14.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:14.774 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:14.800 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:14.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:14.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:14.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:15.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:15.063 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:14 smithi190 ceph-mon[33748]: 3.1 deep-scrub starts 2024-03-24T12:36:15.063 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:14 smithi190 ceph-mon[33748]: 3.1 deep-scrub ok 2024-03-24T12:36:15.077 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:15.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:15.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:14 smithi029 ceph-mon[29919]: 3.1 deep-scrub starts 2024-03-24T12:36:15.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:14 smithi029 ceph-mon[29919]: 3.1 deep-scrub ok 2024-03-24T12:36:15.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:14 smithi029 ceph-mon[41500]: 3.1 deep-scrub starts 2024-03-24T12:36:15.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:14 smithi029 ceph-mon[41500]: 3.1 deep-scrub ok 2024-03-24T12:36:15.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:15.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:15.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:15.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:15.330 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:15.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:15.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:15.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:15.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:15.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:15.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:15.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:15.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:15.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:15.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:15.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:15.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:15.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.1f.5 2024-03-24T12:36:15.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:15.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:15.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:15.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:15.978 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:36:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:16.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:16.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:16.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:16.150 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:16.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:16.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:16.286 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:16.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:16 smithi190 ceph-mon[33748]: pgmap v411: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 389 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 1.09k op/s; 2997/19248 objects degraded (15.570%) 2024-03-24T12:36:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:16 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:36:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:16 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:36:16.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:16.411 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:16.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:16 smithi029 ceph-mon[41500]: pgmap v411: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 389 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 1.09k op/s; 2997/19248 objects degraded (15.570%) 2024-03-24T12:36:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:16 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:36:16.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:16 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:36:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:16 smithi029 ceph-mon[29919]: pgmap v411: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 389 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 36 KiB/s rd, 0 B/s wr, 1.09k op/s; 2997/19248 objects degraded (15.570%) 2024-03-24T12:36:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:16 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:36:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:16 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:36:16.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:16.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:16.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:16.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:16.688 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:16.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:16.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:16.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:16.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:16.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:16.938 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:16.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:17.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:17.087 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:17.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:17.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:17.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:17.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:17.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:17.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:17 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:36:17.398 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:17.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:17.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:17.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:17 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:36:17.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:17 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:36:17.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:17.534 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:36:17.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:17.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:17.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:17.682 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:17.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:17.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:17.827 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:17.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:17.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:17.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:36:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:36:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:36:17.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:17.951 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:17.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:17.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:18.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:18.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:36:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:36:18.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:18 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:36:18.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:18 smithi190 ceph-mon[33748]: pgmap v412: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 335 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 1.06k op/s; 2997/16635 objects degraded (18.016%) 2024-03-24T12:36:18.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:18.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:18.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:18 smithi190 ceph-mon[33748]: 6.12 scrub starts 2024-03-24T12:36:18.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:18 smithi190 ceph-mon[33748]: 6.12 scrub ok 2024-03-24T12:36:18.094 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:18.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:18.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:18.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:18.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:18.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:18.287 INFO:teuthology.orchestra.run.smithi190.stdout:get_pg_num_history pg_num_history pg_num_history(e203 pg_nums {1={20=1},2={66=3},3={66=32},4={68=32},5={70=32},6={72=32},7={109=26},8={158=16}} deleted_pools 153,7) 2024-03-24T12:36:18.287 INFO:teuthology.orchestra.run.smithi190.stdout:Importing pgid 4.1f 2024-03-24T12:36:18.287 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:f93251ce:::obj_delete_at_hint.0000000042:head# 2024-03-24T12:36:18.287 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-03-24T12:36:18.287 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:fce5a501:::obj_delete_at_hint.0000000075:head# 2024-03-24T12:36:18.287 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-03-24T12:36:18.288 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:fd6b5df8:::obj_delete_at_hint.0000000120:head# 2024-03-24T12:36:18.288 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-03-24T12:36:18.288 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:fd8d39b6:::obj_delete_at_hint.0000000030:head# 2024-03-24T12:36:18.288 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-03-24T12:36:18.289 INFO:teuthology.orchestra.run.smithi190.stdout:Write #4:fde4dd55:gc::gc.26:head# 2024-03-24T12:36:18.289 INFO:teuthology.orchestra.run.smithi190.stdout:snapset 0=[]:{} 2024-03-24T12:36:18.289 INFO:teuthology.orchestra.run.smithi190.stdout:write_pg epoch 189 info 4.1f( v 74'11 (0'0,74'11] local-lis/les=68/69 n=5 ec=68/68 lis/c=68/68 les/c/f=69/69/0 sis=68) 2024-03-24T12:36:18.289 INFO:teuthology.orchestra.run.smithi190.stdout:Import successful 2024-03-24T12:36:18.328 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:18.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:18.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:18.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:18.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:36:18.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[29919]: pgmap v412: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 335 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 1.06k op/s; 2997/16635 objects degraded (18.016%) 2024-03-24T12:36:18.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:18.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:18.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[29919]: 6.12 scrub starts 2024-03-24T12:36:18.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[29919]: 6.12 scrub ok 2024-03-24T12:36:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:36:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[41500]: pgmap v412: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 335 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 40 KiB/s rd, 0 B/s wr, 1.06k op/s; 2997/16635 objects degraded (18.016%) 2024-03-24T12:36:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[41500]: 6.12 scrub starts 2024-03-24T12:36:18.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:18 smithi029 ceph-mon[41500]: 6.12 scrub ok 2024-03-24T12:36:18.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:18.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:18.495 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:18.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:18.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:18.644 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:18.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:18.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:18.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:18.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:18.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:18.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:18.920 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:18.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:19.022 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:19.072 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:19.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:19.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:19 smithi190 ceph-mon[33748]: 8.c scrub starts 2024-03-24T12:36:19.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:19 smithi190 ceph-mon[33748]: 8.c scrub ok 2024-03-24T12:36:19.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:19.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:19 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:36:19.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:19 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:36:19.140 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Removed 113857 objects 2024-03-24T12:36:19.140 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Clean up completed and total clean up time :113.447 2024-03-24T12:36:19.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:19 smithi029 ceph-mon[29919]: 8.c scrub starts 2024-03-24T12:36:19.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:19 smithi029 ceph-mon[29919]: 8.c scrub ok 2024-03-24T12:36:19.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:19.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:19 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:36:19.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:19 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:36:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:19 smithi029 ceph-mon[41500]: 8.c scrub starts 2024-03-24T12:36:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:19 smithi029 ceph-mon[41500]: 8.c scrub ok 2024-03-24T12:36:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:19 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:36:19.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:19 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:36:19.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:19.197 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:19.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:19.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:19.291 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:19.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:19.323 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:19.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:19.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:19.333 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:36:19.352 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:36:19.371 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:36:19.390 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:36:19.411 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:36:19.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:19.432 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:36:19.450 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:19.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:19.453 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:36:19.453 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:36:19.453 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:36:19.454 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:36:19.454 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:36:19.454 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:36:19.454 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:36:19.454 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:36:19.454 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:36:19.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/exp.4.1f.5 2024-03-24T12:36:19.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:19.576 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:19.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:19.678 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:19.702 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:19.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:19.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:19.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:19.829 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:19.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:19.931 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:19.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:19.957 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:19.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:20.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:20.082 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:20.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:20.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:20.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:20.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:20.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:20.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:20.335 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:20.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:20 smithi190 ceph-mon[33748]: pgmap v413: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 243 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 1.03k op/s; 1820/12231 objects degraded (14.880%) 2024-03-24T12:36:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:20 smithi190 ceph-mon[33748]: 3.14 scrub starts 2024-03-24T12:36:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:20 smithi190 ceph-mon[33748]: 3.14 scrub ok 2024-03-24T12:36:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:20 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:36:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:20 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:36:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:20 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 1820/12231 objects degraded (14.880%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:20 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1220899143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:20.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[41500]: pgmap v413: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 243 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 1.03k op/s; 1820/12231 objects degraded (14.880%) 2024-03-24T12:36:20.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[41500]: 3.14 scrub starts 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[41500]: 3.14 scrub ok 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 1820/12231 objects degraded (14.880%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1220899143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[29919]: pgmap v413: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 243 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 38 KiB/s rd, 0 B/s wr, 1.03k op/s; 1820/12231 objects degraded (14.880%) 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[29919]: 3.14 scrub starts 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[29919]: 3.14 scrub ok 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 1820/12231 objects degraded (14.880%), 24 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:20 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1220899143' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:20.437 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:20.461 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:20.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:20.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:20.589 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:20.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:20.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:20.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:20.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:20.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:36:20.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:20.848 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:20.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:20.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:20.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:20.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:21.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:21.102 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:21.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:21.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:21.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:21.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:21.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:21.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:21.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:21.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:21.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:21.370 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:21 smithi029 ceph-mon[41500]: pgmap v414: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 84 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 1.07k op/s; 577/4623 objects degraded (12.481%) 2024-03-24T12:36:21.370 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:21 smithi029 ceph-mon[29919]: pgmap v414: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 84 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 1.07k op/s; 577/4623 objects degraded (12.481%) 2024-03-24T12:36:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:21 smithi190 ceph-mon[33748]: pgmap v414: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 84 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 1.07k op/s; 577/4623 objects degraded (12.481%) 2024-03-24T12:36:21.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:21.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:21.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:21.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:21.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:21.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:21.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:21.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:21.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:21.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:21.761 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:36:21.762 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:36:21.848 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:21.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:21.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:21.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:21.998 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:21.999 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:22.099 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:22.129 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:22.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:22.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:22.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:22.255 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:22.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:22.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:22.259 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:22.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:22.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:22.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:22.383 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:22 smithi029 ceph-mon[41500]: 3.b scrub starts 2024-03-24T12:36:22.383 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:22 smithi029 ceph-mon[41500]: 3.b scrub ok 2024-03-24T12:36:22.383 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:22 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:36:22.383 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:22 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:36:22.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:22 smithi029 ceph-mon[29919]: 3.b scrub starts 2024-03-24T12:36:22.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:22 smithi029 ceph-mon[29919]: 3.b scrub ok 2024-03-24T12:36:22.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:22 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:36:22.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:22 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:36:22.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:22 smithi190 ceph-mon[33748]: 3.b scrub starts 2024-03-24T12:36:22.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:22 smithi190 ceph-mon[33748]: 3.b scrub ok 2024-03-24T12:36:22.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:22 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:36:22.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:22 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:36:22.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:22.507 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:22.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:22.609 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:22.631 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:22.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:22.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:22.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:22.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:22.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:22.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:22.887 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:22.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:22.989 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:23.013 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:23.014 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:23.071 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-24T12:36:23.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-24T12:36:23.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:23.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:23.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:23.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:23.267 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:23.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:23.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:23 smithi190 ceph-mon[33748]: 4.1a scrub starts 2024-03-24T12:36:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:23 smithi190 ceph-mon[33748]: 4.1a scrub ok 2024-03-24T12:36:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:23 smithi190 ceph-mon[33748]: osdmap e208: 8 total, 7 up, 8 in 2024-03-24T12:36:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:23 smithi190 ceph-mon[33748]: 6.b scrub starts 2024-03-24T12:36:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:23 smithi190 ceph-mon[33748]: 6.b scrub ok 2024-03-24T12:36:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:23 smithi190 ceph-mon[33748]: pgmap v416: 148 pgs: 20 active+undersized+degraded, 37 active+undersized, 91 active+clean; 8.8 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 867 op/s; 114/987 objects degraded (11.550%) 2024-03-24T12:36:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:23 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:36:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:23 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:36:23.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:23.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:23.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[29919]: 4.1a scrub starts 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[29919]: 4.1a scrub ok 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[29919]: osdmap e208: 8 total, 7 up, 8 in 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[29919]: 6.b scrub starts 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[29919]: 6.b scrub ok 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[29919]: pgmap v416: 148 pgs: 20 active+undersized+degraded, 37 active+undersized, 91 active+clean; 8.8 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 867 op/s; 114/987 objects degraded (11.550%) 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[41500]: 4.1a scrub starts 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[41500]: 4.1a scrub ok 2024-03-24T12:36:23.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[41500]: osdmap e208: 8 total, 7 up, 8 in 2024-03-24T12:36:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[41500]: 6.b scrub starts 2024-03-24T12:36:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[41500]: 6.b scrub ok 2024-03-24T12:36:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[41500]: pgmap v416: 148 pgs: 20 active+undersized+degraded, 37 active+undersized, 91 active+clean; 8.8 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 867 op/s; 114/987 objects degraded (11.550%) 2024-03-24T12:36:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:36:23.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:23 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:36:23.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:23.527 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:23.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:23.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:23.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:23.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:23.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:23.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:23.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:36:23.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:23.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:23.799 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:36:23.799 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:36:23.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:36:23.799 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:36:23.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.799 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "MOSDPGPushReply(7.12 121/112 [PushReplyOp(7:4c958bf0:::benchmark_data_smithi029_84282_object20613:head)])", 2024-03-24T12:36:23.799 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:29:37.982382+0000", 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 405.81947521900003, 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33331592199999999, 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:37.982382+0000", 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:37.982382+0000", 2024-03-24T12:36:23.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:37.982382+0000", 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2899999999999999e-07 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:37.982386+0000", 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6279999999999998e-06 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:37.982394+0000", 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2130000000000008e-06 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:37.982404+0000", 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4210000000000001e-06 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:38.315534+0000", 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33313013200000002 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:29:38.315698+0000", 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016419899999999999 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:33075 8.3 8:cdce902a:::benchmark_data_smithi029_108893_object33074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:10.312637+0000", 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 193.48922041700001, 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.359386382, 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.803 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 33075 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312637+0000", 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312637+0000", 2024-03-24T12:36:23.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312638+0000", 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2810000000000001e-06 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312659+0000", 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0469999999999999e-05 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312668+0000", 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2410000000000001e-06 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312717+0000", 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8871e-05 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312768+0000", 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1350000000000001e-05 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312886+0000", 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000117804 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313037+0000", 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015158200000000001 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.314789+0000", 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0017516299999999999 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.314796+0000", 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7909999999999999e-06 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.671921+0000", 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.35712503600000001 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.671980+0000", 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8655e-05 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.672023+0000", 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3671000000000003e-05 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:33076 8.7 8:e25cf78f:::benchmark_data_smithi029_108893_object33075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:10.312970+0000", 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 193.488887659, 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.36400421500000002, 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.809 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 33076 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312970+0000", 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312970+0000", 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312971+0000", 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8800000000000001e-07 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.312994+0000", 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3441000000000001e-05 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313004+0000", 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7999999999999993e-06 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313012+0000", 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9659999999999996e-06 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313065+0000", 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3288999999999998e-05 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313179+0000", 2024-03-24T12:36:23.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0001138 2024-03-24T12:36:23.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,6", 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313300+0000", 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012110199999999999 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.314665+0000", 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0013650940000000001 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.672217+0000", 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.35755171000000002 2024-03-24T12:36:23.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.676871+0000", 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0046542170000000004 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.676932+0000", 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0693999999999999e-05 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.676974+0000", 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2213999999999999e-05 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:33078 8.1 8:8fb21a8a:::benchmark_data_smithi029_108893_object33077:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:10.313076+0000", 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 193.488781192, 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.36556627200000003, 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.816 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 33078 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313076+0000", 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313076+0000", 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313077+0000", 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1990000000000001e-06 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313105+0000", 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7858999999999999e-05 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313116+0000", 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0504e-05 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313124+0000", 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0790000000000001e-06 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313165+0000", 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0930000000000003e-05 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313253+0000", 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8807000000000005e-05 2024-03-24T12:36:23.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.313347+0000", 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3994999999999997e-05 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.314594+0000", 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001246605 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.671285+0000", 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.35669087999999999 2024-03-24T12:36:23.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.678423+0000", 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0071383380000000001 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.678600+0000", 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000177016 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.678642+0000", 2024-03-24T12:36:23.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.206e-05 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.24815.0:33078 8.1 e170/158 ondisk, result = 0)", 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:10.314925+0000", 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 193.486932478, 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.35637014299999997, 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.314925+0000", 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.314925+0000", 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.314925+0000", 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3499999999999996e-07 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.314929+0000", 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4359999999999998e-06 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.314936+0000", 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8329999999999996e-06 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.314940+0000", 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5850000000000001e-06 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.671267+0000", 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.35632644000000002 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.671277+0000", 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0458999999999999e-05 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.671295+0000", 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7855000000000001e-05 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.825 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.24815.0:33075 8.3 e170/158 ondisk, result = 0)", 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:10.315028+0000", 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 193.486829019, 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.35700172299999999, 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.315028+0000", 2024-03-24T12:36:23.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.315028+0000", 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.315029+0000", 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7e-07 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.827 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.315031+0000", 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6649999999999999e-06 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.671658+0000", 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.356626414 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.671668+0000", 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.059e-05 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.671899+0000", 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023101 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.671907+0000", 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1319999999999994e-06 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:10.672030+0000", 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012254200000000001 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.829 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.24815.0:44991 8.b e173/158 8:d68ba037:::benchmark_data_smithi029_108893_object44990:head v 173'2717, mlcod=173'2713)", 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:20.273727+0000", 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 183.528130167, 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.334671895, 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.273727+0000", 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.273727+0000", 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.273728+0000", 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2999999999999999e-07 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.273754+0000", 2024-03-24T12:36:23.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6529e-05 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.273764+0000", 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.825e-06 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.273775+0000", 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0577999999999999e-05 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.273898+0000", 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012305500000000001 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.273921+0000", 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3034e-05 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.608346+0000", 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33442487999999998 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.833 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.608399+0000", 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3164000000000001e-05 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:45002 8.3 8:cf894ea8:::benchmark_data_smithi029_108893_object45001:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:20.278133+0000", 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 183.523724313, 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.333515799, 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 45002 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.278133+0000", 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.278133+0000", 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.278134+0000", 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1200000000000001e-07 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.278156+0000", 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2305000000000001e-05 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.278168+0000", 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1381e-05 2024-03-24T12:36:23.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.278183+0000", 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5569999999999998e-05 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280487+0000", 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002304202 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280614+0000", 2024-03-24T12:36:23.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000126809 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280750+0000", 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013538100000000001 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.608474+0000", 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32772459999999998 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.608671+0000", 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019684999999999999 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.611557+0000", 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028855299999999999 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.611611+0000", 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4104999999999998e-05 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.611649+0000", 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8154000000000001e-05 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:45007 8.3 8:c0e1991c:::benchmark_data_smithi029_108893_object45006:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:20.280401+0000", 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 183.52145584499999, 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33514502099999999, 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.840 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 45007 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280401+0000", 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280401+0000", 2024-03-24T12:36:23.841 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280402+0000", 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4e-07 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280425+0000", 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3139999999999999e-05 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.842 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280436+0000", 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0248000000000001e-05 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280443+0000", 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.588e-06 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.608710+0000", 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.328266646 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.608943+0000", 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000232876 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.609128+0000", 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018551200000000001 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.611770+0000", 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026416249999999999 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.613200+0000", 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0014305120000000001 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.615447+0000", 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0022467619999999998 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.615502+0000", 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4962e-05 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.615546+0000", 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4310000000000001e-05 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:45008 8.3 8:ca4d8ab1:::benchmark_data_smithi029_108893_object45007:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:20.280457+0000", 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 183.521400008, 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33953772300000001, 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.846 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 45008 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280457+0000", 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.847 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280457+0000", 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280458+0000", 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5899999999999996e-07 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280480+0000", 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2317999999999999e-05 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280488+0000", 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6040000000000004e-06 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.280494+0000", 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5699999999999998e-06 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.613243+0000", 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33274863999999998 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.613369+0000", 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000126198 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.613512+0000", 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014234499999999999 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.615615+0000", 2024-03-24T12:36:23.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0021032260000000001 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.619860+0000", 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0042454440000000001 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.619877+0000", 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6364000000000001e-05 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.619948+0000", 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1503999999999998e-05 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.619995+0000", 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6950999999999998e-05 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:45012 8.3 8:c9beb8d1:::benchmark_data_smithi029_108893_object45011:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:20.282247+0000", 2024-03-24T12:36:23.852 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 183.519610314, 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33795349899999999, 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 45012 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.282247+0000", 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.854 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.854 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.854 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.282247+0000", 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.282248+0000", 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1900000000000002e-07 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.282269+0000", 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1511000000000001e-05 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.855 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.282277+0000", 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6179999999999997e-06 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.282283+0000", 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6499999999999999e-06 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.615790+0000", 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33350645600000001 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.615934+0000", 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014432599999999999 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.616091+0000", 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015642100000000001 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.620050+0000", 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003959036 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.620068+0000", 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8263000000000001e-05 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.620111+0000", 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2687e-05 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.620163+0000", 2024-03-24T12:36:23.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2210999999999999e-05 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.620200+0000", 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7700999999999997e-05 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:45014 8.3 8:c15bbde1:::benchmark_data_smithi029_108893_object45013:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e173)", 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:33:20.283816+0000", 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 183.51804148400001, 2024-03-24T12:36:23.859 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.33891811900000002, 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 45014 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.283816+0000", 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.283816+0000", 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.283817+0000", 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0100000000000004e-07 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.283837+0000", 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0434000000000001e-05 2024-03-24T12:36:23.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.608329+0000", 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.32449249299999999 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.608343+0000", 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3400999999999999e-05 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.617994+0000", 2024-03-24T12:36:23.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0096509939999999995 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.618140+0000", 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014640899999999999 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.618281+0000", 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014105300000000001 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.620216+0000", 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001935071 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.622271+0000", 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020550949999999998 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.622643+0000", 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00037154499999999998 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.864 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.622694+0000", 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1224999999999998e-05 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:33:20.622734+0000", 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9697999999999998e-05 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:132118 8.3 8:c934d450:::benchmark_data_smithi029_108893_object18258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:34:45.999360+0000", 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 97.802497446999993, 2024-03-24T12:36:23.865 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.44550312199999997, 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 132118 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999360+0000", 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999360+0000", 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999361+0000", 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.189e-06 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999367+0000", 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6570000000000002e-06 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999378+0000", 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0954e-05 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999386+0000", 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2630000000000003e-06 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.010021+0000", 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010634727 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.010155+0000", 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000134232 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.010275+0000", 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000120445 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.013680+0000", 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0034047539999999999 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.439925+0000", 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42624483899999999 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.444736+0000", 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0048109759999999998 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.444797+0000", 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0776999999999998e-05 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.444863+0000", 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6309000000000005e-05 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:132138 8.a 8:583ccc1f:::benchmark_data_smithi029_108893_object18278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:34:46.016245+0000", 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 97.785612439999994, 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.436156192, 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.871 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 132138 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016245+0000", 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016245+0000", 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016246+0000", 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2279999999999999e-06 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016250+0000", 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.172e-06 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016259+0000", 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2359999999999993e-06 2024-03-24T12:36:23.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016266+0000", 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1489999999999999e-06 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.439996+0000", 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42373073500000003 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440135+0000", 2024-03-24T12:36:23.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013908400000000001 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440278+0000", 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014265999999999999 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.449225+0000", 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0089471140000000008 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.449357+0000", 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000131564 2024-03-24T12:36:23.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.452245+0000", 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002888467 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.452351+0000", 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105548 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.452401+0000", 2024-03-24T12:36:23.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0235000000000003e-05 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.24815.0:132118 8.3 e189/158 ondisk, result = 0)", 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:34:46.017746+0000", 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 97.784111641999999, 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42219177899999999, 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.017746+0000", 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.017746+0000", 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.017746+0000", 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7999999999999995e-07 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.017751+0000", 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3649999999999997e-06 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.439811+0000", 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42206070200000001 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.439823+0000", 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1274000000000001e-05 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.439895+0000", 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2612000000000002e-05 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.439906+0000", 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0712e-05 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.439937+0000", 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1433999999999998e-05 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.880 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.24815.0:132141 8.f e189/158 8:f6572891:::benchmark_data_smithi029_108893_object18281:head v 189'8315, mlcod=189'8313)", 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:34:46.018414+0000", 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 97.783442835000002, 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42895093499999998, 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.018414+0000", 2024-03-24T12:36:23.881 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.018414+0000", 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.018415+0000", 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5799999999999999e-07 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.882 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.018421+0000", 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2330000000000003e-06 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440292+0000", 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42187096600000001 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440304+0000", 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1215e-05 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.883 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440368+0000", 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4675e-05 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440392+0000", 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3594000000000001e-05 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.447323+0000", 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0069309710000000002 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.884 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.447365+0000", 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2623000000000002e-05 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:206534 8.a 8:549f3ac9:::benchmark_data_smithi029_108893_object92674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:57.688035+0000", 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 26.113822275, 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0529943820000001, 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.885 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 206534 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688035+0000", 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688035+0000", 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.886 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688037+0000", 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5540000000000001e-06 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688042+0000", 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9219999999999996e-06 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688058+0000", 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6325e-05 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.887 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688067+0000", 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5780000000000005e-06 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688128+0000", 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0321000000000001e-05 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688265+0000", 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013724000000000001 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688443+0000", 2024-03-24T12:36:23.888 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000178054 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.708590+0000", 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.020147302999999998 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.708600+0000", 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6330000000000008e-06 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.740862+0000", 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.032261922 2024-03-24T12:36:23.889 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.741029+0000", 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016752999999999999 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:206542 8.a 8:5adb6f6b:::benchmark_data_smithi029_108893_object92682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:57.711565+0000", 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 26.090292716, 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0341805719999999, 2024-03-24T12:36:23.890 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 206542 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711565+0000", 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711565+0000", 2024-03-24T12:36:23.891 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711566+0000", 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.761e-06 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711571+0000", 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0980000000000001e-06 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711584+0000", 2024-03-24T12:36:23.892 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2625e-05 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711590+0000", 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.93e-06 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711637+0000", 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6998e-05 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711645+0000", 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7009999999999996e-06 2024-03-24T12:36:23.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.741788+0000", 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.030143448 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.741954+0000", 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016632800000000001 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2", 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.742106+0000", 2024-03-24T12:36:23.894 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015137899999999999 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.745416+0000", 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0033100439999999998 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.745629+0000", 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021347200000000001 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.745670+0000", 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0614000000000001e-05 2024-03-24T12:36:23.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.745745+0000", 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5173999999999995e-05 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:2089 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 93828934255616 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:58.114923+0000", 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 25.686934682, 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.60225106100000003, 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:36:23.896 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 2089 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114923+0000", 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114923+0000", 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.897 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114924+0000", 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.753e-06 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114930+0000", 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3329999999999996e-06 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114945+0000", 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5656999999999999e-05 2024-03-24T12:36:23.898 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114955+0000", 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7140000000000005e-06 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.115031+0000", 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6396999999999994e-05 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.115044+0000", 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2275e-05 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.899 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.711561+0000", 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.59651754199999996 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.711857+0000", 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029541399999999997 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.717174+0000", 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0053169760000000002 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.900 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:2091 5.15 5:a93a5511:::notify.2:head [watch ping cookie 93828934255616 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:58.327336+0000", 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 25.474521456000002, 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38515318199999998, 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 2091 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:36:23.901 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327336+0000", 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327336+0000", 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327338+0000", 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.699e-06 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:36:23.902 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327344+0000", 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0680000000000002e-06 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327359+0000", 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5658000000000002e-05 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327369+0000", 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0002e-05 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.903 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327502+0000", 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000132986 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327519+0000", 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6269999999999998e-05 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.712226+0000", 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.384707034 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.712318+0000", 2024-03-24T12:36:23.904 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2596000000000006e-05 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.712489+0000", 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017086900000000001 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:23.905 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:36:23.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:23.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:23.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:24.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:24.057 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:24.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:24.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:24.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:24.187 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:24.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:24.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:24.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:24.312 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:24.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:24 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:36:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:24 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:36:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:24 smithi190 ceph-mon[33748]: 4.d deep-scrub starts 2024-03-24T12:36:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:24 smithi190 ceph-mon[33748]: 4.d deep-scrub ok 2024-03-24T12:36:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:24 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:36:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:24 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:36:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:24 smithi190 ceph-mon[33748]: osdmap e209: 8 total, 7 up, 8 in 2024-03-24T12:36:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:24 smithi190 ceph-mon[33748]: 5.17 scrub starts 2024-03-24T12:36:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:24 smithi190 ceph-mon[33748]: 5.17 scrub ok 2024-03-24T12:36:24.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:24.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:36:24.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:36:24.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[41500]: 4.d deep-scrub starts 2024-03-24T12:36:24.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[41500]: 4.d deep-scrub ok 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[41500]: osdmap e209: 8 total, 7 up, 8 in 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[41500]: 5.17 scrub starts 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[41500]: 5.17 scrub ok 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[29919]: 4.d deep-scrub starts 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[29919]: 4.d deep-scrub ok 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:36:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:36:24.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[29919]: osdmap e209: 8 total, 7 up, 8 in 2024-03-24T12:36:24.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[29919]: 5.17 scrub starts 2024-03-24T12:36:24.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:24 smithi029 ceph-mon[29919]: 5.17 scrub ok 2024-03-24T12:36:24.437 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:24.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:24.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:24.526 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:36:24.526 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:36:24.527 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-24T12:36:24.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:36:24.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:24.598 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:24.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:24.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:24.727 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:24.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:24.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:24.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:24.851 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:24.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:24.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:24.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:24.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:25.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:25.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:25.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:25.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:25.322 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[41500]: 5.1 scrub starts 2024-03-24T12:36:25.322 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[41500]: 5.1 scrub ok 2024-03-24T12:36:25.322 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:36:25.322 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:36:25.322 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:36:25.322 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:36:25.323 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[41500]: pgmap v418: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 676 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:25.323 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 114/987 objects degraded (11.550%), 20 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:25.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[29919]: 5.1 scrub starts 2024-03-24T12:36:25.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[29919]: 5.1 scrub ok 2024-03-24T12:36:25.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:36:25.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:36:25.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:36:25.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:36:25.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[29919]: pgmap v418: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 676 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:25.323 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:25 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 114/987 objects degraded (11.550%), 20 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:25.337 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:25.365 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:25.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:25 smithi190 ceph-mon[33748]: 5.1 scrub starts 2024-03-24T12:36:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:25 smithi190 ceph-mon[33748]: 5.1 scrub ok 2024-03-24T12:36:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:25 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:36:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:25 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:36:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:25 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:36:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:25 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:36:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:25 smithi190 ceph-mon[33748]: pgmap v418: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 0 B/s wr, 676 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:25.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:25 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 114/987 objects degraded (11.550%), 20 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:25.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:25.490 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:25.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:25.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:25.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:25.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:25.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:25.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T12:36:25.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:25.742 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:25.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:25.844 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:25.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:25.870 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:25.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:25.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:25.996 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:25.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:26.065 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:26.065 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:26.065 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:26.097 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:26.128 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:26.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:26.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:26.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:26.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:26.357 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[41500]: 8.7 scrub starts 2024-03-24T12:36:26.357 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[41500]: 8.7 scrub ok 2024-03-24T12:36:26.357 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:36:26.357 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[41500]: 4.0 scrub starts 2024-03-24T12:36:26.357 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[41500]: 4.0 scrub ok 2024-03-24T12:36:26.357 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[41500]: 8.6 scrub starts 2024-03-24T12:36:26.357 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[41500]: 8.6 scrub ok 2024-03-24T12:36:26.357 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[29919]: 8.7 scrub starts 2024-03-24T12:36:26.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[29919]: 8.7 scrub ok 2024-03-24T12:36:26.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:36:26.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[29919]: 4.0 scrub starts 2024-03-24T12:36:26.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[29919]: 4.0 scrub ok 2024-03-24T12:36:26.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[29919]: 8.6 scrub starts 2024-03-24T12:36:26.358 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:26 smithi029 ceph-mon[29919]: 8.6 scrub ok 2024-03-24T12:36:26.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:26.381 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:26.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:26 smithi190 ceph-mon[33748]: 8.7 scrub starts 2024-03-24T12:36:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:26 smithi190 ceph-mon[33748]: 8.7 scrub ok 2024-03-24T12:36:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:26 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:36:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:26 smithi190 ceph-mon[33748]: 4.0 scrub starts 2024-03-24T12:36:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:26 smithi190 ceph-mon[33748]: 4.0 scrub ok 2024-03-24T12:36:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:26 smithi190 ceph-mon[33748]: 8.6 scrub starts 2024-03-24T12:36:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:26 smithi190 ceph-mon[33748]: 8.6 scrub ok 2024-03-24T12:36:26.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:26.506 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:26.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:26.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:26.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:26.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:26.632 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:26.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:26.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:26.757 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:26.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:26.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:26.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:26.881 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:26.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:26.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:26.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:27.006 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:27.007 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:27.108 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:27.132 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:27.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:27.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:27.233 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:36:27.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:27.258 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:27.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:27.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:27.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:27 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:36:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:27 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:36:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:36:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:36:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:27 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2456930185' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:36:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:27 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:36:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:27 smithi190 ceph-mon[33748]: pgmap v419: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 485 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:27.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:27.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:27.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:36:27.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:36:27.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:36:27.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:36:27.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:27.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2456930185' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:36:27.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:36:27.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[41500]: pgmap v419: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 485 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:36:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:36:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:36:27.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:36:27.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:27.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2456930185' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:36:27.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:36:27.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-mon[29919]: pgmap v419: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 485 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:27.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:27.509 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:27.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:27.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:27.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:27.634 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:27.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:27.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:27.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:27.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:27.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:27.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:27.889 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:27.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:27.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:36:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:36:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:36:27.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:28.028 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:28.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:28.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:28.153 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:28.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:28.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:28.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:28.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:36:28.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:36:28.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:36:28.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:36:28.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[29919]: osdmap e210: 8 total, 7 up, 8 in 2024-03-24T12:36:28.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:28.269 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:36:28.269 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:36:28.269 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:36:28.269 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:36:28.269 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[41500]: osdmap e210: 8 total, 7 up, 8 in 2024-03-24T12:36:28.269 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:28.279 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:28.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:28.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:36:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:36:28.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:28 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:36:28.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:28 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:36:28.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:28 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:36:28.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:28 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:36:28.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:28 smithi190 ceph-mon[33748]: osdmap e210: 8 total, 7 up, 8 in 2024-03-24T12:36:28.285 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:28.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:28.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:28.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:28.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:28.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:28.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:28.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:28.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:28.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:28.658 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:28.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:28.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:28.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:28.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:28.784 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:28.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:28.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:28.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:28.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:28.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:29.010 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:29.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:29.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:29.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:29.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:29.158 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:29.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:29.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:29.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:29.284 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:29.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:29.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:29.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:29 smithi190 ceph-mon[33748]: pgmap v421: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 111 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:29.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:29.410 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:29.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:29.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:29.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:29 smithi029 ceph-mon[29919]: pgmap v421: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 111 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:29.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:29 smithi029 ceph-mon[41500]: pgmap v421: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 111 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:29.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:29.535 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:29.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:29.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:29.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:29.660 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:29.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:29.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:29.786 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:29.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:29.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:29.910 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:29.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:29.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:36:30.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:30.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:30.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:30.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:30.162 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:30.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:30.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:30.288 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:30.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:30.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:30.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:30 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 53/594 objects degraded (8.923%), 16 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:30.322 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:30 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 53/594 objects degraded (8.923%), 16 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:30 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 53/594 objects degraded (8.923%), 16 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:30.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:30.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:30.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:30.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:30.540 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:30.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:30.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:30.666 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:30.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:30.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:30.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:30.791 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:30.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:30.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:30.918 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:30.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:31.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:31.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:31.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:31.158 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'unique_pool_1' removed 2024-03-24T12:36:31.158 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:31 smithi029 ceph-mon[41500]: pgmap v422: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 16 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:31.159 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:31 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3907333776' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:36:31.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:31 smithi029 ceph-mon[29919]: pgmap v422: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 16 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:31.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:31 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3907333776' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:36:31.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:31.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:31.273 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:31.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:31.298 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:31.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:31 smithi190 ceph-mon[33748]: pgmap v422: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 16 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:31 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3907333776' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:36:31.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:31.423 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:31.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:31.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:31.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:31.547 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:31.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:31.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:31.678 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:31.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:31.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:31.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:31.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:31.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:31.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:31.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:31.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:32.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:32.055 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:32.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:32.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:32.072 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T12:36:32.072 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T12:36:32.073 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-24T12:36:32.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-24T12:36:32.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:32.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:32.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:32.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:32.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:32.314 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:32.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:32.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:32 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3907333776' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:36:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:32 smithi190 ceph-mon[33748]: osdmap e211: 8 total, 7 up, 8 in 2024-03-24T12:36:32.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:32.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:32 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3907333776' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:36:32.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:32 smithi029 ceph-mon[29919]: osdmap e211: 8 total, 7 up, 8 in 2024-03-24T12:36:32.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:32 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3907333776' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:36:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:32 smithi029 ceph-mon[41500]: osdmap e211: 8 total, 7 up, 8 in 2024-03-24T12:36:32.438 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:32.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:32.526 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:36:32.526 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:36:32.526 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 211, 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "created": "2024-03-24T12:20:18.712998+0000", 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "modified": "2024-03-24T12:36:31.128070+0000", 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "last_up_change": "2024-03-24T12:35:57.677873+0000", 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "last_in_change": "2024-03-24T12:25:30.369664+0000", 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_num": 5801984, 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_set": [ 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "noscrub", 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "pglog_hardlimit", 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "purged_snapdirs", 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_deletes", 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "sortbitwise" 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:36:32.527 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_version": 18, 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_snapshot": "", 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_max": 8, 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd": 8, 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "min_compat_client": "luminous", 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "require_osd_release": "squid", 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_crimson": false, 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "pools": [ 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 1, 2024-03-24T12:36:32.528 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".mgr", 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:23:41.491597+0000", 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:36:32.529 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 1, 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 1, 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 1, 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 1, 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 1, 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:36:32.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "22", 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:36:32.531 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.532 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_max": 32, 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_min": 1 2024-03-24T12:36:32.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": {} 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 7.8899998664855957, 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 7.8899998664855957, 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.37999999523162842, 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 3, 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 3, 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:36:32.534 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 2, 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "datapool", 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:01.465080+0000", 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 8193, 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:36:32.535 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 3, 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 3, 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 3, 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 3, 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 3, 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:36:32.536 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "71", 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 2, 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 71, 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:36:32.537 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:36:32.538 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:36:32.539 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd": {} 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.6700000762939453, 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.6700000762939453, 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2, 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2, 2024-03-24T12:36:32.540 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 3, 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".rgw.root", 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:02.100021+0000", 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:36:32.541 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:36:32.542 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:36:32.543 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:36:32.543 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:36:32.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.543 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "68", 2024-03-24T12:36:32.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:32.544 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:36:32.544 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:36:32.544 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:36:32.544 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:36:32.544 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:36:32.544 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:36:32.544 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:36:32.545 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:36:32.546 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.9900000095367432, 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.9900000095367432, 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:36:32.547 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.75, 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.75, 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 4, 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:03.473862+0000", 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:36:32.548 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:36:32.549 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "70", 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:36:32.550 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:36:32.551 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:36:32.552 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.2400000095367432, 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.2400000095367432, 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.9700000286102295, 2024-03-24T12:36:32.553 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.9700000286102295, 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 5, 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:05.573828+0000", 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:36:32.554 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:36:32.555 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "72", 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:36:32.556 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:36:32.557 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:36:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.4900000095367432, 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.4900000095367432, 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.309999942779541, 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.309999942779541, 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:36:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 6, 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:07.645083+0000", 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:36:32.560 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:36:32.561 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "74", 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:36:32.562 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:36:32.563 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:36:32.564 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_bias": 4 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.9900000095367432, 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.9900000095367432, 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.87999999523162842, 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.75, 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.75, 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 1, 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 1, 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 1 2024-03-24T12:36:32.565 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b", 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 8, 2024-03-24T12:36:32.566 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 204, 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6801", 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6802", 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6804", 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.568 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6803", 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6801/2514818464", 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6802/2514818464", 2024-03-24T12:36:32.569 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6804/2514818464", 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6803/2514818464", 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc", 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:36:32.570 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 13, 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 204, 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6805", 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.571 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6806", 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.572 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6808", 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6807", 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.573 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6805/2923396572", 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6806/2923396572", 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6808/2923396572", 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6807/2923396572", 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.574 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "c3f83505-25ee-4217-b60a-081c36092e92", 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 18, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 204, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.575 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6809", 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6810", 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6812", 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.577 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6811", 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6809/126032915", 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6810/126032915", 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6812/126032915", 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6811/126032915", 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:36:32.578 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96", 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 27, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 204, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:36:32.579 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6813", 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.580 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6814", 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6816", 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.581 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6815", 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6813/1080171824", 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6814/1080171824", 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6816/1080171824", 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6815/1080171824", 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:36:32.582 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "5e420304-2798-4d11-9688-d0d89482008c", 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 33, 2024-03-24T12:36:32.583 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 204, 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6800", 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:36:32.584 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6801", 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6803", 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:36:32.585 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6802", 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6800/3690258869", 2024-03-24T12:36:32.586 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6801/3690258869", 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6803/3690258869", 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6802/3690258869", 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "1213431a-652c-4cbf-b32a-36335430c666", 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 0, 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:36:32.587 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 42, 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 204, 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6804", 2024-03-24T12:36:32.588 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6805", 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6807", 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.590 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6806", 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6804/1327220488", 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6805/1327220488", 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6807/1327220488", 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6806/1327220488", 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: "exists" 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c", 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 51, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 204, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:36:32.592 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6808", 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6809", 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:36:32.593 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6811", 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:36:32.594 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6810", 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6808/1295064976", 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6809/1295064976", 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6811/1295064976", 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6810/1295064976", 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:36:32.595 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda", 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 60, 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 204, 2024-03-24T12:36:32.596 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:36:32.597 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:36:32.597 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:36:32.597 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.597 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.597 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.597 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6812", 2024-03-24T12:36:32.598 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:36:32.598 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.598 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.598 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.598 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:36:32.598 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.598 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.598 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.598 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6813", 2024-03-24T12:36:32.598 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6815", 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.599 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6814", 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6812/110963949", 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6813/110963949", 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6815/110963949", 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6814/110963949", 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:36:32.600 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_xinfo": [ 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:36:22.775614+0000", 2024-03-24T12:36:32.601 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:36:21.503057+0000", 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.602 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:36:03.817780+0000", 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:36:32.603 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:35:52.606110+0000", 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:35:35.954695+0000", 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.604 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:35:04.518484+0000", 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 203 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:36:32.605 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:35:45.344951+0000", 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:35:31.824636+0000", 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:36:32.606 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap": [], 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_items": [ 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "3.11", 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 6, 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 5 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.607 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.b", 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.f", 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.608 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.12", 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 5 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:36:32.609 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_primaries": [], 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_temp": [], 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_temp": [], 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklist": { 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2633022206": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2612078930": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2365028933": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4113609697": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4130171883": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3697529912": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2929430696": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/327626505": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3715448582": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:36:32.610 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/861146182": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2109454792": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1631806715": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/852212806": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1823252077": "2024-03-25T12:21:15.151901+0000" 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "range_blocklist": {}, 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profiles": { 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "default": { 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-failure-domain": "osd", 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "k": "2", 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "m": "1", 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "plugin": "jerasure", 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "technique": "reed_sol_van" 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:36:32.611 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps_queue": [], 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: "new_removed_snaps": [], 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: "new_purged_snaps": [], 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_node_flags": {}, 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: "device_class_flags": {}, 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode": { 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_enabled": false, 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_bucket_count": 0, 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_bucket": 0 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:36:32.612 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:36:32.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:32.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:32.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:32.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:32.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:32.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:32.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:32.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:32.878 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:32.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:32.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:33.005 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:33.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:33.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:33.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:33.130 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:33.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:33.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:33.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:33.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:33.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:33.355 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:33 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/145337930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:36:33.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:33 smithi029 ceph-mon[41500]: pgmap v424: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:33.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:33.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:33.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:33 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3689627754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:33.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:33 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/145337930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:36:33.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:33 smithi029 ceph-mon[29919]: pgmap v424: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:33.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:33.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:33.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:33 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3689627754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:33.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:33.382 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:33.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:33.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:33.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:33.508 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:33.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:33.570 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:36:33.571 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:36:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:36:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:36:33.572 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:36:33.609 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:33 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/145337930' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:36:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:33 smithi190 ceph-mon[33748]: pgmap v424: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:33.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:33 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3689627754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:33.637 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:33.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:33.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:33.762 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:33.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:33.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:33.892 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:33.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:33.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:33.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:34.021 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:34.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:34.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:34.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:34.147 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:34.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:34.178 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:36:34.179 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:36:34.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:34.272 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:34.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:34.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:34.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:34.398 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:34.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:34.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:34.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:34.528 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:34.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:34.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:34.653 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:34.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:34.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:34.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:34.779 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:34.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:34.881 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:34.901 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:34.910 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:34.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:34.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:34.921 INFO:tasks.thrashosds.thrasher:j is {'epoch': 211, 'fsid': 'bc017092-e9d8-11ee-95cd-87774f69a715', 'created': '2024-03-24T12:20:18.712998+0000', 'modified': '2024-03-24T12:36:31.128070+0000', 'last_up_change': '2024-03-24T12:35:57.677873+0000', 'last_in_change': '2024-03-24T12:25:30.369664+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T12:23:41.491597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-24T12:26:01.465080+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T12:26:02.100021+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T12:26:03.473862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.9700000286102295, 'raw_score_stable': 1.9700000286102295, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T12:26:05.573828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.4900000095367432, 'score_stable': 1.4900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T12:26:07.645083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '0aa1c49b-7887-4f74-b03c-08cc0adec90b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6801', 'nonce': 2514818464}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6802', 'nonce': 2514818464}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6804', 'nonce': 2514818464}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6803', 'nonce': 2514818464}]}, 'public_addr': '172.21.15.29:6801/2514818464', 'cluster_addr': '172.21.15.29:6802/2514818464', 'heartbeat_back_addr': '172.21.15.29:6804/2514818464', 'heartbeat_front_addr': '172.21.15.29:6803/2514818464', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99d8ef8f-eb59-4882-ab51-b398e6b53bcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6805', 'nonce': 2923396572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6806', 'nonce': 2923396572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6808', 'nonce': 2923396572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6807', 'nonce': 2923396572}]}, 'public_addr': '172.21.15.29:6805/2923396572', 'cluster_addr': '172.21.15.29:6806/2923396572', 'heartbeat_back_addr': '172.21.15.29:6808/2923396572', 'heartbeat_front_addr': '172.21.15.29:6807/2923396572', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c3f83505-25ee-4217-b60a-081c36092e92', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6809', 'nonce': 126032915}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6810', 'nonce': 126032915}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6812', 'nonce': 126032915}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6811', 'nonce': 126032915}]}, 'public_addr': '172.21.15.29:6809/126032915', 'cluster_addr': '172.21.15.29:6810/126032915', 'heartbeat_back_addr': '172.21.15.29:6812/126032915', 'heartbeat_front_addr': '172.21.15.29:6811/126032915', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f5e479a8-459a-4315-a808-711a7b84bb96', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6813', 'nonce': 1080171824}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6814', 'nonce': 1080171824}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6816', 'nonce': 1080171824}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6815', 'nonce': 1080171824}]}, 'public_addr': '172.21.15.29:6813/1080171824', 'cluster_addr': '172.21.15.29:6814/1080171824', 'heartbeat_back_addr': '172.21.15.29:6816/1080171824', 'heartbeat_front_addr': '172.21.15.29:6815/1080171824', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '5e420304-2798-4d11-9688-d0d89482008c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 3690258869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6801', 'nonce': 3690258869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6803', 'nonce': 3690258869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 3690258869}]}, 'public_addr': '172.21.15.190:6800/3690258869', 'cluster_addr': '172.21.15.190:6801/3690258869', 'heartbeat_back_addr': '172.21.15.190:6803/3690258869', 'heartbeat_front_addr': '172.21.15.190:6802/3690258869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1213431a-652c-4cbf-b32a-36335430c666', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 158, 'down_at': 204, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 1327220488}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6805', 'nonce': 1327220488}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6807', 'nonce': 1327220488}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 1327220488}]}, 'public_addr': '172.21.15.190:6804/1327220488', 'cluster_addr': '172.21.15.190:6805/1327220488', 'heartbeat_back_addr': '172.21.15.190:6807/1327220488', 'heartbeat_front_addr': '172.21.15.190:6806/1327220488', 'state': ['exists']}, {'osd': 6, 'uuid': '0712e511-0257-44f3-a82f-ee09ab957c0c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 1295064976}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6809', 'nonce': 1295064976}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6811', 'nonce': 1295064976}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 1295064976}]}, 'public_addr': '172.21.15.190:6808/1295064976', 'cluster_addr': '172.21.15.190:6809/1295064976', 'heartbeat_back_addr': '172.21.15.190:6811/1295064976', 'heartbeat_front_addr': '172.21.15.190:6810/1295064976', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7e36bde4-14d0-4e7e-9a82-2bb97859dbda', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 110963949}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6813', 'nonce': 110963949}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6815', 'nonce': 110963949}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 110963949}]}, 'public_addr': '172.21.15.190:6812/110963949', 'cluster_addr': '172.21.15.190:6813/110963949', 'heartbeat_back_addr': '172.21.15.190:6815/110963949', 'heartbeat_front_addr': '172.21.15.190:6814/110963949', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:36:22.775614+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:36:21.503057+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:36:03.817780+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:35:52.606110+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:35:35.954695+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:35:04.518484+0000', 'dead_epoch': 203}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:35:45.344951+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:35:31.824636+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.11', 'mappings': [{'from': 6, 'to': 5}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.12', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.29:0/2633022206': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2612078930': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2365028933': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4113609697': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4130171883': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/3697529912': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2929430696': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/327626505': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/3715448582': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/861146182': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2109454792': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/1631806715': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:6800/852212806': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/1823252077': '2024-03-25T12:21:15.151901+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:36:34.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:36:35.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:35.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:35.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:35.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:35.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:35.168 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:35.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:35.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:35.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:35.299 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:35.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:35.363 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:35.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:35.426 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:35.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:35.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:35.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:35.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:35.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:35.684 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:35.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:35.778 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:35 smithi029 ceph-mon[41500]: pgmap v425: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:35.778 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:35 smithi029 ceph-mon[29919]: pgmap v425: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:35.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:35.810 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:35.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:35.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:35 smithi190 ceph-mon[33748]: pgmap v425: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:35.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:35.935 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:35.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:36.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:36.059 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:36.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:36.161 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:36.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:36:36.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:36.185 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:36.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:36.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:36.314 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:36.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:36.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:36.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:36.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:36.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:36.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:36.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:36.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:36.694 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:36.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:36.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:36.819 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:36.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:36.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:36.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:36.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:36.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:36.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:37.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:37.076 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:37.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:37.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:37.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:37.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:37.269 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:37.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:37.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:37.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:37.335 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:37.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:37.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:37.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:37.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:37.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:37.724 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:37.751 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:37.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:37.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:37.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:37.877 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:37.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:37.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:37 smithi029 ceph-mon[29919]: pgmap v426: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 985 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:37.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:37 smithi029 ceph-mon[41500]: pgmap v426: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 985 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:37.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:36:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:36:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:36:37.968 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:37 smithi190 ceph-mon[33748]: pgmap v426: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 985 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:37.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:38.003 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:38.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:38.128 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:38.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:38.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:38.252 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:38.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:38.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:38.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:38.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:38.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:38.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:36:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:36:38.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:38.503 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:38.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:38.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:38.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:38.628 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:38.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:38.696 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:38 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:38.696 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:38 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:38.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:38.757 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:38.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:38.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:38.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:38.883 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:38.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:38.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:38.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:39.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:39.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:39.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:39.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:39.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:38 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:39.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:39.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:39.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:39.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:39.267 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:39.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:39.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:39.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:39.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:39.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:39.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:39.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:39.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:39.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:39.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:39.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:39.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:39.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:39.780 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:39.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:39.881 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:39.906 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:39.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:39.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:39 smithi029 ceph-mon[29919]: pgmap v427: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:39.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:39 smithi029 ceph-mon[41500]: pgmap v427: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:40.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:40.032 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:40.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:40.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:40.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:39 smithi190 ceph-mon[33748]: pgmap v427: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:40.158 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:40.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:40.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:40.282 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:40.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:40.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:40.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:40.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:40.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:40.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:40.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:40.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:40.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:40.658 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'unique_pool_2' created 2024-03-24T12:36:40.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:40.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:40.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:40.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:40.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:40.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:40.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:40 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/649161364' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T12:36:40.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:40 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/649161364' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T12:36:40.924 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:40.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:40.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:41.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:41.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:41.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:41.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:41.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:40 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/649161364' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-24T12:36:41.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:41.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:41.184 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:41.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:41.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:41.309 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:41.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:41.411 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:41.434 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:41.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:41.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:41.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:41.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:41.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:41.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:41.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:41.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:41.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:41.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:41.820 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:41.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:41.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:41.878 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:41 smithi029 ceph-mon[41500]: pgmap v428: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:41.878 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:41 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/649161364' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-24T12:36:41.879 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:41 smithi029 ceph-mon[41500]: osdmap e212: 8 total, 7 up, 8 in 2024-03-24T12:36:41.879 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:41 smithi029 ceph-mon[29919]: pgmap v428: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:41.879 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:41 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/649161364' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-24T12:36:41.879 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:41 smithi029 ceph-mon[29919]: osdmap e212: 8 total, 7 up, 8 in 2024-03-24T12:36:41.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:41.946 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:41.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:42.004 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:42.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:42.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:42.110 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:36:42.113 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:36:42.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:41 smithi190 ceph-mon[33748]: pgmap v428: 132 pgs: 16 active+undersized+degraded, 33 active+undersized, 83 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:42.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:41 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/649161364' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-24T12:36:42.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:41 smithi190 ceph-mon[33748]: osdmap e212: 8 total, 7 up, 8 in 2024-03-24T12:36:42.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:42.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:42.198 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:42.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:42.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:42.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:42.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:42.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:42.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:42.451 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:42.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:42.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:42.578 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:42.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:42.661 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:36:42.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:42.703 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:42.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:42.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:42.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:42.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:42.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:42.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-24T12:36:42.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:42 smithi029 ceph-mon[29919]: osdmap e213: 8 total, 7 up, 8 in 2024-03-24T12:36:42.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:42 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1204776765' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:36:42.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:42 smithi029 ceph-mon[41500]: osdmap e213: 8 total, 7 up, 8 in 2024-03-24T12:36:42.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:42 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1204776765' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:36:42.935 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:42.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:42.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:43.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:43.089 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:43.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:43.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:43.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:42 smithi190 ceph-mon[33748]: osdmap e213: 8 total, 7 up, 8 in 2024-03-24T12:36:43.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:42 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1204776765' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:36:43.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:43.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:43.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:43.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:43.321 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:43.345 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:43.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:43.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:43.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:43.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:43.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:43.541 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-03-24T12:36:41.655297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":53,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1981,"num_read_kb":1827,"num_write":901,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":964,"ondisk_log_size":964,"up":353,"acting":353,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":405,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3381676,"kb_used_data":930388,"kb_used_omap":12,"kb_used_meta":2451187,"kb_avail":746579540,"statfs":{"total":767960285184,"available":764497448960,"internally_reserved":0,"allocated":952717312,"data_stored":949784873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":2510016049},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.072960"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":326,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.513898+0000","last_change":"2024-03-24T12:35:58.701806+0000","last_active":"2024-03-24T12:36:31.513898+0000","last_peered":"2024-03-24T12:36:31.513898+0000","last_clean":"2024-03-24T12:35:54.676897+0000","last_became_active":"2024-03-24T12:35:58.701806+0000","last_became_peered":"2024-03-24T12:35:58.701806+0000","last_unstale":"2024-03-24T12:36:31.513898+0000","last_undegraded":"2024-03-24T12:36:31.513898+0000","last_fullsized":"2024-03-24T12:35:58.695795+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:08.859869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.859869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.338192+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":330,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.138527+0000","last_change":"2024-03-24T12:35:58.709334+0000","last_active":"2024-03-24T12:36:31.138527+0000","last_peered":"2024-03-24T12:36:31.138527+0000","last_clean":"2024-03-24T12:35:54.687134+0000","last_became_active":"2024-03-24T12:35:58.709334+0000","last_became_peered":"2024-03-24T12:35:58.709334+0000","last_unstale":"2024-03-24T12:36:31.138527+0000","last_undegraded":"2024-03-24T12:36:31.138527+0000","last_fullsized":"2024-03-24T12:35:58.702300+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:48.744937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.744937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.828340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":333,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-24T12:36:40.659611+0000","last_change":"2024-03-24T12:35:58.712452+0000","last_active":"2024-03-24T12:36:40.659611+0000","last_peered":"2024-03-24T12:36:40.659611+0000","last_clean":"2024-03-24T12:35:54.683285+0000","last_became_active":"2024-03-24T12:35:58.712452+0000","last_became_peered":"2024-03-24T12:35:58.712452+0000","last_unstale":"2024-03-24T12:36:40.659611+0000","last_undegraded":"2024-03-24T12:36:40.659611+0000","last_fullsized":"2024-03-24T12:35:58.707789+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:14.966328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:14.966328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"207'15","reported_seq":356,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.513749+0000","last_change":"2024-03-24T12:36:06.773401+0000","last_active":"2024-03-24T12:36:31.513749+0000","last_peered":"2024-03-24T12:36:31.513749+0000","last_clean":"2024-03-24T12:36:31.513749+0000","last_became_active":"2024-03-24T12:26:05.315506+0000","last_became_peered":"2024-03-24T12:26:05.315506+0000","last_unstale":"2024-03-24T12:36:31.513749+0000","last_undegraded":"2024-03-24T12:36:31.513749+0000","last_fullsized":"2024-03-24T12:36:31.513749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:36:06.773261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:36:06.773261+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:27.693083+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.599500+0000","last_change":"2024-03-24T12:35:07.031107+0000","last_active":"2024-03-24T12:36:31.599500+0000","last_peered":"2024-03-24T12:36:31.599500+0000","last_clean":"2024-03-24T12:36:31.599500+0000","last_became_active":"2024-03-24T12:26:09.645798+0000","last_became_peered":"2024-03-24T12:26:09.645798+0000","last_unstale":"2024-03-24T12:36:31.599500+0000","last_undegraded":"2024-03-24T12:36:31.599500+0000","last_fullsized":"2024-03-24T12:36:31.599500+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:07.030987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:07.030987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458808+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.513575+0000","last_change":"2024-03-24T12:35:12.883693+0000","last_active":"2024-03-24T12:36:31.513575+0000","last_peered":"2024-03-24T12:36:31.513575+0000","last_clean":"2024-03-24T12:36:31.513575+0000","last_became_active":"2024-03-24T12:26:07.335254+0000","last_became_peered":"2024-03-24T12:26:07.335254+0000","last_unstale":"2024-03-24T12:36:31.513575+0000","last_undegraded":"2024-03-24T12:36:31.513575+0000","last_fullsized":"2024-03-24T12:36:31.513575+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:12.883613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:12.883613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:38.551474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":328,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.596873+0000","last_change":"2024-03-24T12:35:02.993072+0000","last_active":"2024-03-24T12:36:31.596873+0000","last_peered":"2024-03-24T12:36:31.596873+0000","last_clean":"2024-03-24T12:36:31.596873+0000","last_became_active":"2024-03-24T12:26:03.164114+0000","last_became_peered":"2024-03-24T12:26:03.164114+0000","last_unstale":"2024-03-24T12:36:31.596873+0000","last_undegraded":"2024-03-24T12:36:31.596873+0000","last_fullsized":"2024-03-24T12:36:31.596873+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:02.992900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.992900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458720+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"207'34","reported_seq":373,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.597596+0000","last_change":"2024-03-24T12:33:36.263856+0000","last_active":"2024-03-24T12:36:31.597596+0000","last_peered":"2024-03-24T12:36:31.597596+0000","last_clean":"2024-03-24T12:36:31.597596+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:36:31.597596+0000","last_undegraded":"2024-03-24T12:36:31.597596+0000","last_fullsized":"2024-03-24T12:36:31.597596+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2024-03-24T12:33:36.263707+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.263707+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458658+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":325,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.658663+0000","last_change":"2024-03-24T12:34:58.789197+0000","last_active":"2024-03-24T12:36:40.658663+0000","last_peered":"2024-03-24T12:36:40.658663+0000","last_clean":"2024-03-24T12:36:40.658663+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:36:40.658663+0000","last_undegraded":"2024-03-24T12:36:40.658663+0000","last_fullsized":"2024-03-24T12:36:40.658663+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:58.789088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:34:58.789088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":24,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-24T12:36:40.659720+0000","last_change":"2024-03-24T12:35:58.711151+0000","last_active":"2024-03-24T12:36:40.659720+0000","last_peered":"2024-03-24T12:36:40.659720+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.711151+0000","last_became_peered":"2024-03-24T12:35:58.711151+0000","last_unstale":"2024-03-24T12:36:40.659720+0000","last_undegraded":"2024-03-24T12:36:40.659720+0000","last_fullsized":"2024-03-24T12:35:58.703392+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:44.161505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.161505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.515347+0000","last_change":"2024-03-24T12:35:58.706758+0000","last_active":"2024-03-24T12:36:31.515347+0000","last_peered":"2024-03-24T12:36:31.515347+0000","last_clean":"2024-03-24T12:26:02.143860+0000","last_became_active":"2024-03-24T12:35:58.706758+0000","last_became_peered":"2024-03-24T12:35:58.706758+0000","last_unstale":"2024-03-24T12:36:31.515347+0000","last_undegraded":"2024-03-24T12:36:31.515347+0000","last_fullsized":"2024-03-24T12:35:58.700092+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:33.177938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:33.177938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"207'15","reported_seq":356,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.514717+0000","last_change":"2024-03-24T12:36:21.714274+0000","last_active":"2024-03-24T12:36:31.514717+0000","last_peered":"2024-03-24T12:36:31.514717+0000","last_clean":"2024-03-24T12:36:31.514717+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:36:31.514717+0000","last_undegraded":"2024-03-24T12:36:31.514717+0000","last_fullsized":"2024-03-24T12:36:31.514717+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:36:21.714161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:36:21.714161+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:45.262610+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.138807+0000","last_change":"2024-03-24T12:35:58.718922+0000","last_active":"2024-03-24T12:36:31.138807+0000","last_peered":"2024-03-24T12:36:31.138807+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.718922+0000","last_became_peered":"2024-03-24T12:35:58.718922+0000","last_unstale":"2024-03-24T12:36:31.138807+0000","last_undegraded":"2024-03-24T12:36:31.138807+0000","last_fullsized":"2024-03-24T12:35:58.702792+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.155345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.155345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.828317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.133717+0000","last_change":"2024-03-24T12:35:59.708744+0000","last_active":"2024-03-24T12:36:31.133717+0000","last_peered":"2024-03-24T12:36:31.133717+0000","last_clean":"2024-03-24T12:36:31.133717+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:36:31.133717+0000","last_undegraded":"2024-03-24T12:36:31.133717+0000","last_fullsized":"2024-03-24T12:36:31.133717+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:59.708595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:35:59.708595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:10.451359+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.514671+0000","last_change":"2024-03-24T12:35:58.708742+0000","last_active":"2024-03-24T12:36:31.514671+0000","last_peered":"2024-03-24T12:36:31.514671+0000","last_clean":"2024-03-24T12:26:02.143860+0000","last_became_active":"2024-03-24T12:35:58.708742+0000","last_became_peered":"2024-03-24T12:35:58.708742+0000","last_unstale":"2024-03-24T12:36:31.514671+0000","last_undegraded":"2024-03-24T12:36:31.514671+0000","last_fullsized":"2024-03-24T12:35:58.699613+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:23.095706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:33:23.095706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"207'7","reported_seq":344,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.514017+0000","last_change":"2024-03-24T12:34:32.424665+0000","last_active":"2024-03-24T12:36:31.514017+0000","last_peered":"2024-03-24T12:36:31.514017+0000","last_clean":"2024-03-24T12:36:31.514017+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:36:31.514017+0000","last_undegraded":"2024-03-24T12:36:31.514017+0000","last_fullsized":"2024-03-24T12:36:31.514017+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-24T12:34:32.424537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:32.424537+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561435+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.514719+0000","last_change":"2024-03-24T12:35:58.706607+0000","last_active":"2024-03-24T12:36:31.514719+0000","last_peered":"2024-03-24T12:36:31.514719+0000","last_clean":"2024-03-24T12:35:54.678236+0000","last_became_active":"2024-03-24T12:35:58.706607+0000","last_became_peered":"2024-03-24T12:35:58.706607+0000","last_unstale":"2024-03-24T12:36:31.514719+0000","last_undegraded":"2024-03-24T12:36:31.514719+0000","last_fullsized":"2024-03-24T12:35:58.700244+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:49.652214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:49.652214+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561410+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":343,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.140154+0000","last_change":"2024-03-24T12:36:23.728799+0000","last_active":"2024-03-24T12:36:31.140154+0000","last_peered":"2024-03-24T12:36:31.140154+0000","last_clean":"2024-03-24T12:36:31.140154+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:36:31.140154+0000","last_undegraded":"2024-03-24T12:36:31.140154+0000","last_fullsized":"2024-03-24T12:36:31.140154+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:23.728717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:36:23.728717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:33.747800+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":329,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.660212+0000","last_change":"2024-03-24T12:34:46.648871+0000","last_active":"2024-03-24T12:36:40.660212+0000","last_peered":"2024-03-24T12:36:40.660212+0000","last_clean":"2024-03-24T12:36:40.660212+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:36:40.660212+0000","last_undegraded":"2024-03-24T12:36:40.660212+0000","last_fullsized":"2024-03-24T12:36:40.660212+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:46.648735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:34:46.648735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"207'27","reported_seq":374,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.135576+0000","last_change":"2024-03-24T12:35:04.954711+0000","last_active":"2024-03-24T12:36:31.135576+0000","last_peered":"2024-03-24T12:36:31.135576+0000","last_clean":"2024-03-24T12:36:31.135576+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:36:31.135576+0000","last_undegraded":"2024-03-24T12:36:31.135576+0000","last_fullsized":"2024-03-24T12:36:31.135576+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:35:04.954560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.954560+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":319,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.601874+0000","last_change":"2024-03-24T12:35:58.718321+0000","last_active":"2024-03-24T12:36:31.601874+0000","last_peered":"2024-03-24T12:36:31.601874+0000","last_clean":"2024-03-24T12:35:54.694978+0000","last_became_active":"2024-03-24T12:35:58.718321+0000","last_became_peered":"2024-03-24T12:35:58.718321+0000","last_unstale":"2024-03-24T12:36:31.601874+0000","last_undegraded":"2024-03-24T12:36:31.601874+0000","last_fullsized":"2024-03-24T12:35:58.706636+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:51.927644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:51.927644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.513618+0000","last_change":"2024-03-24T12:35:02.853101+0000","last_active":"2024-03-24T12:36:31.513618+0000","last_peered":"2024-03-24T12:36:31.513618+0000","last_clean":"2024-03-24T12:36:31.513618+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:36:31.513618+0000","last_undegraded":"2024-03-24T12:36:31.513618+0000","last_fullsized":"2024-03-24T12:36:31.513618+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:02.852982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.852982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561502+0000","scrub_duration":62,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.514821+0000","last_change":"2024-03-24T12:36:04.747596+0000","last_active":"2024-03-24T12:36:31.514821+0000","last_peered":"2024-03-24T12:36:31.514821+0000","last_clean":"2024-03-24T12:36:31.514821+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:36:31.514821+0000","last_undegraded":"2024-03-24T12:36:31.514821+0000","last_fullsized":"2024-03-24T12:36:31.514821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:04.747486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:04.747486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:11.361809+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"207'18","reported_seq":355,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.597518+0000","last_change":"2024-03-24T12:36:01.347255+0000","last_active":"2024-03-24T12:36:31.597518+0000","last_peered":"2024-03-24T12:36:31.597518+0000","last_clean":"2024-03-24T12:36:31.597518+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:36:31.597518+0000","last_undegraded":"2024-03-24T12:36:31.597518+0000","last_fullsized":"2024-03-24T12:36:31.597518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:36:01.347110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:36:01.347110+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:15.446692+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":319,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.134314+0000","last_change":"2024-03-24T12:35:58.711430+0000","last_active":"2024-03-24T12:36:31.134314+0000","last_peered":"2024-03-24T12:36:31.134314+0000","last_clean":"2024-03-24T12:35:54.688013+0000","last_became_active":"2024-03-24T12:35:58.711430+0000","last_became_peered":"2024-03-24T12:35:58.711430+0000","last_unstale":"2024-03-24T12:36:31.134314+0000","last_undegraded":"2024-03-24T12:35:58.697902+0000","last_fullsized":"2024-03-24T12:35:58.697784+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:34:42.820487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:42.820487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:32.023174+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":474,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:32.567624+0000","last_change":"2024-03-24T12:36:24.008629+0000","last_active":"2024-03-24T12:36:32.567624+0000","last_peered":"2024-03-24T12:36:32.567624+0000","last_clean":"2024-03-24T12:36:32.567624+0000","last_became_active":"2024-03-24T12:26:07.335213+0000","last_became_peered":"2024-03-24T12:26:07.335213+0000","last_unstale":"2024-03-24T12:36:32.567624+0000","last_undegraded":"2024-03-24T12:36:32.567624+0000","last_fullsized":"2024-03-24T12:36:32.567624+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:36:24.008518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:36:24.008518+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:45.240646+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":344,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.138223+0000","last_change":"2024-03-24T12:36:03.631710+0000","last_active":"2024-03-24T12:36:31.138223+0000","last_peered":"2024-03-24T12:36:31.138223+0000","last_clean":"2024-03-24T12:36:31.138223+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:36:31.138223+0000","last_undegraded":"2024-03-24T12:36:31.138223+0000","last_fullsized":"2024-03-24T12:36:31.138223+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:36:03.631586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:03.631586+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:18.860995+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"207'20","reported_seq":364,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.660792+0000","last_change":"2024-03-24T12:35:05.755602+0000","last_active":"2024-03-24T12:36:40.660792+0000","last_peered":"2024-03-24T12:36:40.660792+0000","last_clean":"2024-03-24T12:36:40.660792+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:36:40.660792+0000","last_undegraded":"2024-03-24T12:36:40.660792+0000","last_fullsized":"2024-03-24T12:36:40.660792+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:35:05.755456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:05.755456+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":326,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.140285+0000","last_change":"2024-03-24T12:35:58.705420+0000","last_active":"2024-03-24T12:36:31.140285+0000","last_peered":"2024-03-24T12:36:31.140285+0000","last_clean":"2024-03-24T12:35:54.688035+0000","last_became_active":"2024-03-24T12:35:58.705420+0000","last_became_peered":"2024-03-24T12:35:58.705420+0000","last_unstale":"2024-03-24T12:36:31.140285+0000","last_undegraded":"2024-03-24T12:36:31.140285+0000","last_fullsized":"2024-03-24T12:35:58.697777+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:03.555975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.555975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.828292+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":322,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.660951+0000","last_change":"2024-03-24T12:36:06.248136+0000","last_active":"2024-03-24T12:36:40.660951+0000","last_peered":"2024-03-24T12:36:40.660951+0000","last_clean":"2024-03-24T12:36:40.660951+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:36:40.660951+0000","last_undegraded":"2024-03-24T12:36:40.660951+0000","last_fullsized":"2024-03-24T12:36:40.660951+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:06.247360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:36:06.247360+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:18.538351+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.599184+0000","last_change":"2024-03-24T12:33:28.354892+0000","last_active":"2024-03-24T12:36:31.599184+0000","last_peered":"2024-03-24T12:36:31.599184+0000","last_clean":"2024-03-24T12:36:31.599184+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:36:31.599184+0000","last_undegraded":"2024-03-24T12:36:31.599184+0000","last_fullsized":"2024-03-24T12:36:31.599184+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:28.354367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.354367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458640+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"207'19","reported_seq":366,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.597767+0000","last_change":"2024-03-24T12:35:58.717530+0000","last_active":"2024-03-24T12:36:31.597767+0000","last_peered":"2024-03-24T12:36:31.597767+0000","last_clean":"2024-03-24T12:35:54.684607+0000","last_became_active":"2024-03-24T12:35:58.717530+0000","last_became_peered":"2024-03-24T12:35:58.717530+0000","last_unstale":"2024-03-24T12:36:31.597767+0000","last_undegraded":"2024-03-24T12:35:58.700976+0000","last_fullsized":"2024-03-24T12:35:58.700861+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:34:43.138814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:43.138814+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.017364+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":["6","1"],"object_location_counts":[{"shards":"1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":323,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.133695+0000","last_change":"2024-03-24T12:36:17.855254+0000","last_active":"2024-03-24T12:36:31.133695+0000","last_peered":"2024-03-24T12:36:31.133695+0000","last_clean":"2024-03-24T12:36:31.133695+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:36:31.133695+0000","last_undegraded":"2024-03-24T12:36:31.133695+0000","last_fullsized":"2024-03-24T12:36:31.133695+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:17.855129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:36:17.855129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:31.857303+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.596421+0000","last_change":"2024-03-24T12:35:56.943670+0000","last_active":"2024-03-24T12:36:31.596421+0000","last_peered":"2024-03-24T12:36:31.596421+0000","last_clean":"2024-03-24T12:36:31.596421+0000","last_became_active":"2024-03-24T12:26:07.335384+0000","last_became_peered":"2024-03-24T12:26:07.335384+0000","last_unstale":"2024-03-24T12:36:31.596421+0000","last_undegraded":"2024-03-24T12:36:31.596421+0000","last_fullsized":"2024-03-24T12:36:31.596421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:56.943584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:56.943584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:59.912834+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":332,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-24T12:36:40.659864+0000","last_change":"2024-03-24T12:35:58.718236+0000","last_active":"2024-03-24T12:36:40.659864+0000","last_peered":"2024-03-24T12:36:40.659864+0000","last_clean":"2024-03-24T12:35:54.683418+0000","last_became_active":"2024-03-24T12:35:58.718236+0000","last_became_peered":"2024-03-24T12:35:58.718236+0000","last_unstale":"2024-03-24T12:36:40.659864+0000","last_undegraded":"2024-03-24T12:36:40.659864+0000","last_fullsized":"2024-03-24T12:35:58.708687+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.720633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.720633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"207'8","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.599228+0000","last_change":"2024-03-24T12:33:27.357494+0000","last_active":"2024-03-24T12:36:31.599228+0000","last_peered":"2024-03-24T12:36:31.599228+0000","last_clean":"2024-03-24T12:36:31.599228+0000","last_became_active":"2024-03-24T12:26:05.319743+0000","last_became_peered":"2024-03-24T12:26:05.319743+0000","last_unstale":"2024-03-24T12:36:31.599228+0000","last_undegraded":"2024-03-24T12:36:31.599228+0000","last_fullsized":"2024-03-24T12:36:31.599228+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'4","last_scrub_stamp":"2024-03-24T12:33:27.355578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:27.355578+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458607+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.599744+0000","last_change":"2024-03-24T12:34:57.998315+0000","last_active":"2024-03-24T12:36:31.599744+0000","last_peered":"2024-03-24T12:36:31.599744+0000","last_clean":"2024-03-24T12:36:31.599744+0000","last_became_active":"2024-03-24T12:26:09.645177+0000","last_became_peered":"2024-03-24T12:26:09.645177+0000","last_unstale":"2024-03-24T12:36:31.599744+0000","last_undegraded":"2024-03-24T12:36:31.599744+0000","last_fullsized":"2024-03-24T12:36:31.599744+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:57.998162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:34:57.998162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458751+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.134244+0000","last_change":"2024-03-24T12:35:12.719879+0000","last_active":"2024-03-24T12:36:31.134244+0000","last_peered":"2024-03-24T12:36:31.134244+0000","last_clean":"2024-03-24T12:36:31.134244+0000","last_became_active":"2024-03-24T12:26:07.340013+0000","last_became_peered":"2024-03-24T12:26:07.340013+0000","last_unstale":"2024-03-24T12:36:31.134244+0000","last_undegraded":"2024-03-24T12:36:31.134244+0000","last_fullsized":"2024-03-24T12:36:31.134244+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:12.719663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:12.719663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.187535+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":29,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.134710+0000","last_change":"2024-03-24T12:35:58.708757+0000","last_active":"2024-03-24T12:36:31.134710+0000","last_peered":"2024-03-24T12:36:31.134710+0000","last_clean":"2024-03-24T12:26:03.166952+0000","last_became_active":"2024-03-24T12:35:58.708757+0000","last_became_peered":"2024-03-24T12:35:58.708757+0000","last_unstale":"2024-03-24T12:36:31.134710+0000","last_undegraded":"2024-03-24T12:35:58.698998+0000","last_fullsized":"2024-03-24T12:35:58.698185+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:28.059288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.059288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:32.023129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"207'19","reported_seq":354,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.597500+0000","last_change":"2024-03-24T12:34:54.934073+0000","last_active":"2024-03-24T12:36:31.597500+0000","last_peered":"2024-03-24T12:36:31.597500+0000","last_clean":"2024-03-24T12:36:31.597500+0000","last_became_active":"2024-03-24T12:26:05.358682+0000","last_became_peered":"2024-03-24T12:26:05.358682+0000","last_unstale":"2024-03-24T12:36:31.597500+0000","last_undegraded":"2024-03-24T12:36:31.597500+0000","last_fullsized":"2024-03-24T12:36:31.597500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:34:54.933925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.933925+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458736+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":320,"reported_epoch":212,"state":"active+undersized","last_fresh":"2024-03-24T12:36:40.658301+0000","last_change":"2024-03-24T12:35:58.706939+0000","last_active":"2024-03-24T12:36:40.658301+0000","last_peered":"2024-03-24T12:36:40.658301+0000","last_clean":"2024-03-24T12:35:54.681672+0000","last_became_active":"2024-03-24T12:35:58.706939+0000","last_became_peered":"2024-03-24T12:35:58.706939+0000","last_unstale":"2024-03-24T12:36:40.658301+0000","last_undegraded":"2024-03-24T12:36:40.658301+0000","last_fullsized":"2024-03-24T12:35:58.695715+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:53.895902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.895902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":319,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.599419+0000","last_change":"2024-03-24T12:33:31.383451+0000","last_active":"2024-03-24T12:36:31.599419+0000","last_peered":"2024-03-24T12:36:31.599419+0000","last_clean":"2024-03-24T12:36:31.599419+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:36:31.599419+0000","last_undegraded":"2024-03-24T12:36:31.599419+0000","last_fullsized":"2024-03-24T12:36:31.599419+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_clean_scrub_stamp":"2024-03-24T12:33:31.383330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458625+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.134826+0000","last_change":"2024-03-24T12:35:10.729253+0000","last_active":"2024-03-24T12:36:31.134826+0000","last_peered":"2024-03-24T12:36:31.134826+0000","last_clean":"2024-03-24T12:36:31.134826+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:36:31.134826+0000","last_undegraded":"2024-03-24T12:36:31.134826+0000","last_fullsized":"2024-03-24T12:36:31.134826+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.729138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.729138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"207'15","reported_seq":65,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.596829+0000","last_change":"2024-03-24T12:35:58.714993+0000","last_active":"2024-03-24T12:36:31.596829+0000","last_peered":"2024-03-24T12:36:31.596829+0000","last_clean":"2024-03-24T12:26:10.587572+0000","last_became_active":"2024-03-24T12:35:58.714993+0000","last_became_peered":"2024-03-24T12:35:58.714993+0000","last_unstale":"2024-03-24T12:36:31.596829+0000","last_undegraded":"2024-03-24T12:35:58.698252+0000","last_fullsized":"2024-03-24T12:35:58.698144+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:35:02.976915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.976915+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.017392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":["6","2"],"object_location_counts":[{"shards":"2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":318,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.597100+0000","last_change":"2024-03-24T12:35:58.714875+0000","last_active":"2024-03-24T12:36:31.597100+0000","last_peered":"2024-03-24T12:36:31.597100+0000","last_clean":"2024-03-24T12:35:54.689143+0000","last_became_active":"2024-03-24T12:35:58.714875+0000","last_became_peered":"2024-03-24T12:35:58.714875+0000","last_unstale":"2024-03-24T12:36:31.597100+0000","last_undegraded":"2024-03-24T12:36:31.597100+0000","last_fullsized":"2024-03-24T12:35:58.702457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.898988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.898988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458685+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":330,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.139064+0000","last_change":"2024-03-24T12:35:58.718562+0000","last_active":"2024-03-24T12:36:31.139064+0000","last_peered":"2024-03-24T12:36:31.139064+0000","last_clean":"2024-03-24T12:35:54.687557+0000","last_became_active":"2024-03-24T12:35:58.718562+0000","last_became_peered":"2024-03-24T12:35:58.718562+0000","last_unstale":"2024-03-24T12:36:31.139064+0000","last_undegraded":"2024-03-24T12:36:31.139064+0000","last_fullsized":"2024-03-24T12:35:58.697301+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.144883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.144883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.828309+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":343,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.515564+0000","last_change":"2024-03-24T12:36:18.761794+0000","last_active":"2024-03-24T12:36:31.515564+0000","last_peered":"2024-03-24T12:36:31.515564+0000","last_clean":"2024-03-24T12:36:31.515564+0000","last_became_active":"2024-03-24T12:26:03.163379+0000","last_became_peered":"2024-03-24T12:26:03.163379+0000","last_unstale":"2024-03-24T12:36:31.515564+0000","last_undegraded":"2024-03-24T12:36:31.515564+0000","last_fullsized":"2024-03-24T12:36:31.515564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:18.761708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:18.761708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:30.646150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"207'24","reported_seq":375,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.514464+0000","last_change":"2024-03-24T12:35:04.793820+0000","last_active":"2024-03-24T12:36:31.514464+0000","last_peered":"2024-03-24T12:36:31.514464+0000","last_clean":"2024-03-24T12:36:31.514464+0000","last_became_active":"2024-03-24T12:26:05.315348+0000","last_became_peered":"2024-03-24T12:26:05.315348+0000","last_unstale":"2024-03-24T12:36:31.514464+0000","last_undegraded":"2024-03-24T12:36:31.514464+0000","last_fullsized":"2024-03-24T12:36:31.514464+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'16","last_scrub_stamp":"2024-03-24T12:35:04.793689+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.793689+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561475+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":318,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.660619+0000","last_change":"2024-03-24T12:34:57.796652+0000","last_active":"2024-03-24T12:36:40.660619+0000","last_peered":"2024-03-24T12:36:40.660619+0000","last_clean":"2024-03-24T12:36:40.660619+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:36:40.660619+0000","last_undegraded":"2024-03-24T12:36:40.660619+0000","last_fullsized":"2024-03-24T12:36:40.660619+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:34:57.796520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:57.796520+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":77,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"205'9","reported_seq":56,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:33.335433+0000","last_change":"2024-03-24T12:35:58.710493+0000","last_active":"2024-03-24T12:36:33.335433+0000","last_peered":"2024-03-24T12:36:33.335433+0000","last_clean":"2024-03-24T12:26:10.493548+0000","last_became_active":"2024-03-24T12:35:58.710493+0000","last_became_peered":"2024-03-24T12:35:58.710493+0000","last_unstale":"2024-03-24T12:36:33.335433+0000","last_undegraded":"2024-03-24T12:35:58.697342+0000","last_fullsized":"2024-03-24T12:35:58.697213+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:35:09.007492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:09.007492+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:38.881561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.134032+0000","last_change":"2024-03-24T12:36:10.719570+0000","last_active":"2024-03-24T12:36:31.134032+0000","last_peered":"2024-03-24T12:36:31.134032+0000","last_clean":"2024-03-24T12:36:31.134032+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:36:31.134032+0000","last_undegraded":"2024-03-24T12:36:31.134032+0000","last_fullsized":"2024-03-24T12:36:31.134032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:10.719495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:10.719495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:22.803806+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"207'20","reported_seq":361,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.598342+0000","last_change":"2024-03-24T12:35:57.320380+0000","last_active":"2024-03-24T12:36:31.598342+0000","last_peered":"2024-03-24T12:36:31.598342+0000","last_clean":"2024-03-24T12:36:31.598342+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:36:31.598342+0000","last_undegraded":"2024-03-24T12:36:31.598342+0000","last_fullsized":"2024-03-24T12:36:31.598342+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-24T12:35:57.320252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:57.320252+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:59.299123+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":326,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.514914+0000","last_change":"2024-03-24T12:35:58.712757+0000","last_active":"2024-03-24T12:36:31.514914+0000","last_peered":"2024-03-24T12:36:31.514914+0000","last_clean":"2024-03-24T12:35:54.677408+0000","last_became_active":"2024-03-24T12:35:58.712757+0000","last_became_peered":"2024-03-24T12:35:58.712757+0000","last_unstale":"2024-03-24T12:36:31.514914+0000","last_undegraded":"2024-03-24T12:36:31.514914+0000","last_fullsized":"2024-03-24T12:35:58.700636+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:01.913287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:01.913287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561454+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"93'9","reported_seq":456,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:33.335400+0000","last_change":"2024-03-24T12:36:02.327855+0000","last_active":"2024-03-24T12:36:33.335400+0000","last_peered":"2024-03-24T12:36:33.335400+0000","last_clean":"2024-03-24T12:36:33.335400+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:36:33.335400+0000","last_undegraded":"2024-03-24T12:36:33.335400+0000","last_fullsized":"2024-03-24T12:36:33.335400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'9","last_scrub_stamp":"2024-03-24T12:36:02.327739+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:36:02.327739+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:05.195795+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":337,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.659865+0000","last_change":"2024-03-24T12:36:04.229737+0000","last_active":"2024-03-24T12:36:40.659865+0000","last_peered":"2024-03-24T12:36:40.659865+0000","last_clean":"2024-03-24T12:36:40.659865+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:36:40.659865+0000","last_undegraded":"2024-03-24T12:36:40.659865+0000","last_fullsized":"2024-03-24T12:36:40.659865+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:04.229621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:04.229621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:31.003720+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"207'19","reported_seq":98,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.135298+0000","last_change":"2024-03-24T12:35:58.751467+0000","last_active":"2024-03-24T12:36:31.135298+0000","last_peered":"2024-03-24T12:36:31.135298+0000","last_clean":"2024-03-24T12:27:16.240812+0000","last_became_active":"2024-03-24T12:35:58.751467+0000","last_became_peered":"2024-03-24T12:35:58.751467+0000","last_unstale":"2024-03-24T12:36:31.135298+0000","last_undegraded":"2024-03-24T12:35:58.700376+0000","last_fullsized":"2024-03-24T12:35:58.700265+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:35:08.002032+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.002032+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.036433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":331,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.135872+0000","last_change":"2024-03-24T12:36:22.236075+0000","last_active":"2024-03-24T12:36:31.135872+0000","last_peered":"2024-03-24T12:36:31.135872+0000","last_clean":"2024-03-24T12:36:31.135872+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:36:31.135872+0000","last_undegraded":"2024-03-24T12:36:31.135872+0000","last_fullsized":"2024-03-24T12:36:31.135872+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:22.235824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:36:22.235824+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:44.210402+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":319,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.601720+0000","last_change":"2024-03-24T12:36:23.475131+0000","last_active":"2024-03-24T12:36:31.601720+0000","last_peered":"2024-03-24T12:36:31.601720+0000","last_clean":"2024-03-24T12:36:31.601720+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:36:31.601720+0000","last_undegraded":"2024-03-24T12:36:31.601720+0000","last_fullsized":"2024-03-24T12:36:31.601720+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:23.475052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:36:23.475052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:47.942496+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":344,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.133730+0000","last_change":"2024-03-24T12:35:58.703902+0000","last_active":"2024-03-24T12:36:31.133730+0000","last_peered":"2024-03-24T12:36:31.133730+0000","last_clean":"2024-03-24T12:35:54.687144+0000","last_became_active":"2024-03-24T12:35:58.703902+0000","last_became_peered":"2024-03-24T12:35:58.703902+0000","last_unstale":"2024-03-24T12:36:31.133730+0000","last_undegraded":"2024-03-24T12:36:31.133730+0000","last_fullsized":"2024-03-24T12:35:58.698923+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:50.800852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:50.800852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:32.023182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"207'15","reported_seq":358,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.660844+0000","last_change":"2024-03-24T12:36:03.226122+0000","last_active":"2024-03-24T12:36:40.660844+0000","last_peered":"2024-03-24T12:36:40.660844+0000","last_clean":"2024-03-24T12:36:40.660844+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:36:40.660844+0000","last_undegraded":"2024-03-24T12:36:40.660844+0000","last_fullsized":"2024-03-24T12:36:40.660844+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:36:03.225747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:36:03.225747+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:05.523043+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.134476+0000","last_change":"2024-03-24T12:35:04.719690+0000","last_active":"2024-03-24T12:36:31.134476+0000","last_peered":"2024-03-24T12:36:31.134476+0000","last_clean":"2024-03-24T12:36:31.134476+0000","last_became_active":"2024-03-24T12:26:09.645901+0000","last_became_peered":"2024-03-24T12:26:09.645901+0000","last_unstale":"2024-03-24T12:36:31.134476+0000","last_undegraded":"2024-03-24T12:36:31.134476+0000","last_fullsized":"2024-03-24T12:36:31.134476+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.719553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.719553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":22,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.597017+0000","last_change":"2024-03-24T12:35:58.714488+0000","last_active":"2024-03-24T12:36:31.597017+0000","last_peered":"2024-03-24T12:36:31.597017+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.714488+0000","last_became_peered":"2024-03-24T12:35:58.714488+0000","last_unstale":"2024-03-24T12:36:31.597017+0000","last_undegraded":"2024-03-24T12:36:31.597017+0000","last_fullsized":"2024-03-24T12:35:58.700893+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.995618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.995618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":338,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.596540+0000","last_change":"2024-03-24T12:35:58.709475+0000","last_active":"2024-03-24T12:36:31.596540+0000","last_peered":"2024-03-24T12:36:31.596540+0000","last_clean":"2024-03-24T12:35:54.683912+0000","last_became_active":"2024-03-24T12:35:58.709475+0000","last_became_peered":"2024-03-24T12:35:58.709475+0000","last_unstale":"2024-03-24T12:36:31.596540+0000","last_undegraded":"2024-03-24T12:36:31.596540+0000","last_fullsized":"2024-03-24T12:35:58.697021+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.188916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.188916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.017377+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"207'12","reported_seq":341,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.601956+0000","last_change":"2024-03-24T12:35:15.907723+0000","last_active":"2024-03-24T12:36:31.601956+0000","last_peered":"2024-03-24T12:36:31.601956+0000","last_clean":"2024-03-24T12:36:31.601956+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:36:31.601956+0000","last_undegraded":"2024-03-24T12:36:31.601956+0000","last_fullsized":"2024-03-24T12:36:31.601956+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'6","last_scrub_stamp":"2024-03-24T12:35:15.907580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.907580+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.415309+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":212,"state":"creating+peering","last_fresh":"2024-03-24T12:36:40.657771+0000","last_change":"2024-03-24T12:36:40.657771+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.657771+0000","last_undegraded":"2024-03-24T12:36:40.657771+0000","last_fullsized":"2024-03-24T12:36:40.657771+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.597033+0000","last_change":"2024-03-24T12:35:58.709539+0000","last_active":"2024-03-24T12:36:31.597033+0000","last_peered":"2024-03-24T12:36:31.597033+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.709539+0000","last_became_peered":"2024-03-24T12:35:58.709539+0000","last_unstale":"2024-03-24T12:36:31.597033+0000","last_undegraded":"2024-03-24T12:36:31.597033+0000","last_fullsized":"2024-03-24T12:35:58.697194+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.170473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.170473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.017312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":347,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.135003+0000","last_change":"2024-03-24T12:36:12.738305+0000","last_active":"2024-03-24T12:36:31.135003+0000","last_peered":"2024-03-24T12:36:31.135003+0000","last_clean":"2024-03-24T12:36:31.135003+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:36:31.135003+0000","last_undegraded":"2024-03-24T12:36:31.135003+0000","last_fullsized":"2024-03-24T12:36:31.135003+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:36:12.738167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:12.738167+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:37.496976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"207'27","reported_seq":83,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.134113+0000","last_change":"2024-03-24T12:35:58.706252+0000","last_active":"2024-03-24T12:36:31.134113+0000","last_peered":"2024-03-24T12:36:31.134113+0000","last_clean":"2024-03-24T12:26:10.633417+0000","last_became_active":"2024-03-24T12:35:58.706252+0000","last_became_peered":"2024-03-24T12:35:58.706252+0000","last_unstale":"2024-03-24T12:36:31.134113+0000","last_undegraded":"2024-03-24T12:35:58.699574+0000","last_fullsized":"2024-03-24T12:35:58.699437+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-24T12:33:26.094785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:33:26.094785+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:32.023165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[{"shards":"6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"74'8","reported_seq":473,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.133935+0000","last_change":"2024-03-24T12:36:15.766023+0000","last_active":"2024-03-24T12:36:31.133935+0000","last_peered":"2024-03-24T12:36:31.133935+0000","last_clean":"2024-03-24T12:36:31.133935+0000","last_became_active":"2024-03-24T12:26:07.341418+0000","last_became_peered":"2024-03-24T12:26:07.341418+0000","last_unstale":"2024-03-24T12:36:31.133935+0000","last_undegraded":"2024-03-24T12:36:31.133935+0000","last_fullsized":"2024-03-24T12:36:31.133935+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:36:15.765929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:36:15.765929+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:39.552853+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.599589+0000","last_change":"2024-03-24T12:36:22.536047+0000","last_active":"2024-03-24T12:36:31.599589+0000","last_peered":"2024-03-24T12:36:31.599589+0000","last_clean":"2024-03-24T12:36:31.599589+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:36:31.599589+0000","last_undegraded":"2024-03-24T12:36:31.599589+0000","last_fullsized":"2024-03-24T12:36:31.599589+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:22.535905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:36:22.535905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:49.282847+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.134947+0000","last_change":"2024-03-24T12:35:02.723283+0000","last_active":"2024-03-24T12:36:31.134947+0000","last_peered":"2024-03-24T12:36:31.134947+0000","last_clean":"2024-03-24T12:36:31.134947+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:36:31.134947+0000","last_undegraded":"2024-03-24T12:36:31.134947+0000","last_fullsized":"2024-03-24T12:36:31.134947+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:02.723170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.723170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.036466+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"207'23","reported_seq":376,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.514057+0000","last_change":"2024-03-24T12:36:12.887496+0000","last_active":"2024-03-24T12:36:31.514057+0000","last_peered":"2024-03-24T12:36:31.514057+0000","last_clean":"2024-03-24T12:36:31.514057+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:36:31.514057+0000","last_undegraded":"2024-03-24T12:36:31.514057+0000","last_fullsized":"2024-03-24T12:36:31.514057+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-24T12:36:12.887336+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:36:12.887336+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:25.888676+0000","scrub_duration":92,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.133967+0000","last_change":"2024-03-24T12:35:11.997564+0000","last_active":"2024-03-24T12:36:31.133967+0000","last_peered":"2024-03-24T12:36:31.133967+0000","last_clean":"2024-03-24T12:36:31.133967+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:36:31.133967+0000","last_undegraded":"2024-03-24T12:36:31.133967+0000","last_fullsized":"2024-03-24T12:36:31.133967+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:11.997339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:11.997339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.100592+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":323,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.133580+0000","last_change":"2024-03-24T12:36:09.761939+0000","last_active":"2024-03-24T12:36:31.133580+0000","last_peered":"2024-03-24T12:36:31.133580+0000","last_clean":"2024-03-24T12:36:31.133580+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:36:31.133580+0000","last_undegraded":"2024-03-24T12:36:31.133580+0000","last_fullsized":"2024-03-24T12:36:31.133580+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:09.761812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:36:09.761812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:36.441392+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":330,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.134186+0000","last_change":"2024-03-24T12:35:58.711865+0000","last_active":"2024-03-24T12:36:31.134186+0000","last_peered":"2024-03-24T12:36:31.134186+0000","last_clean":"2024-03-24T12:35:54.681353+0000","last_became_active":"2024-03-24T12:35:58.711865+0000","last_became_peered":"2024-03-24T12:35:58.711865+0000","last_unstale":"2024-03-24T12:36:31.134186+0000","last_undegraded":"2024-03-24T12:36:31.134186+0000","last_fullsized":"2024-03-24T12:35:58.701950+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:16.019255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.019255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":330,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.135096+0000","last_change":"2024-03-24T12:35:58.751179+0000","last_active":"2024-03-24T12:36:31.135096+0000","last_peered":"2024-03-24T12:36:31.135096+0000","last_clean":"2024-03-24T12:35:54.689099+0000","last_became_active":"2024-03-24T12:35:58.751179+0000","last_became_peered":"2024-03-24T12:35:58.751179+0000","last_unstale":"2024-03-24T12:36:31.135096+0000","last_undegraded":"2024-03-24T12:36:31.135096+0000","last_fullsized":"2024-03-24T12:35:58.699964+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.204829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:32.023155+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"207'35","reported_seq":394,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.597252+0000","last_change":"2024-03-24T12:36:01.964714+0000","last_active":"2024-03-24T12:36:31.597252+0000","last_peered":"2024-03-24T12:36:31.597252+0000","last_clean":"2024-03-24T12:36:31.597252+0000","last_became_active":"2024-03-24T12:26:05.356864+0000","last_became_peered":"2024-03-24T12:26:05.356864+0000","last_unstale":"2024-03-24T12:36:31.597252+0000","last_undegraded":"2024-03-24T12:36:31.597252+0000","last_fullsized":"2024-03-24T12:36:31.597252+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-24T12:36:01.964549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:36:01.964549+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:09.692632+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":317,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.659504+0000","last_change":"2024-03-24T12:35:09.948925+0000","last_active":"2024-03-24T12:36:40.659504+0000","last_peered":"2024-03-24T12:36:40.659504+0000","last_clean":"2024-03-24T12:36:40.659504+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:36:40.659504+0000","last_undegraded":"2024-03-24T12:36:40.659504+0000","last_fullsized":"2024-03-24T12:36:40.659504+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:09.948402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:09.948402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":81,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.134969+0000","last_change":"2024-03-24T12:35:10.999806+0000","last_active":"2024-03-24T12:36:31.134969+0000","last_peered":"2024-03-24T12:36:31.134969+0000","last_clean":"2024-03-24T12:36:31.134969+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:36:31.134969+0000","last_undegraded":"2024-03-24T12:36:31.134969+0000","last_fullsized":"2024-03-24T12:36:31.134969+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.999487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.999487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:39.057856+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":79,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.598404+0000","last_change":"2024-03-24T12:35:58.716025+0000","last_active":"2024-03-24T12:36:31.598404+0000","last_peered":"2024-03-24T12:36:31.598404+0000","last_clean":"2024-03-24T12:27:09.233091+0000","last_became_active":"2024-03-24T12:35:58.716025+0000","last_became_peered":"2024-03-24T12:35:58.716025+0000","last_unstale":"2024-03-24T12:36:31.598404+0000","last_undegraded":"2024-03-24T12:35:58.704219+0000","last_fullsized":"2024-03-24T12:35:58.704098+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:35:15.994784+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.994784+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:45.262811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[{"shards":"3,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"207'15","reported_seq":366,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.660041+0000","last_change":"2024-03-24T12:35:59.198701+0000","last_active":"2024-03-24T12:36:40.660041+0000","last_peered":"2024-03-24T12:36:40.660041+0000","last_clean":"2024-03-24T12:36:40.660041+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:36:40.660041+0000","last_undegraded":"2024-03-24T12:36:40.660041+0000","last_fullsized":"2024-03-24T12:36:40.660041+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:35:59.198563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:59.198563+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:19.609298+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":212,"state":"creating+peering","last_fresh":"2024-03-24T12:36:40.657700+0000","last_change":"2024-03-24T12:36:40.657700+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.657700+0000","last_undegraded":"2024-03-24T12:36:40.657700+0000","last_fullsized":"2024-03-24T12:36:40.657700+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":315,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.597167+0000","last_change":"2024-03-24T12:35:07.993235+0000","last_active":"2024-03-24T12:36:31.597167+0000","last_peered":"2024-03-24T12:36:31.597167+0000","last_clean":"2024-03-24T12:36:31.597167+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:36:31.597167+0000","last_undegraded":"2024-03-24T12:36:31.597167+0000","last_fullsized":"2024-03-24T12:36:31.597167+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:07.993108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:35:07.993108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458793+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":47,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.138576+0000","last_change":"2024-03-24T12:35:58.702107+0000","last_active":"2024-03-24T12:36:31.138576+0000","last_peered":"2024-03-24T12:36:31.138576+0000","last_clean":"2024-03-24T12:26:07.318867+0000","last_became_active":"2024-03-24T12:35:58.702107+0000","last_became_peered":"2024-03-24T12:35:58.702107+0000","last_unstale":"2024-03-24T12:36:31.138576+0000","last_undegraded":"2024-03-24T12:35:58.696851+0000","last_fullsized":"2024-03-24T12:35:58.696723+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:35:56.337636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:56.337636+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:04.000013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.514298+0000","last_change":"2024-03-24T12:34:55.729278+0000","last_active":"2024-03-24T12:36:31.514298+0000","last_peered":"2024-03-24T12:36:31.514298+0000","last_clean":"2024-03-24T12:36:31.514298+0000","last_became_active":"2024-03-24T12:26:03.161499+0000","last_became_peered":"2024-03-24T12:26:03.161499+0000","last_unstale":"2024-03-24T12:36:31.514298+0000","last_undegraded":"2024-03-24T12:36:31.514298+0000","last_fullsized":"2024-03-24T12:36:31.514298+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:55.729129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:55.729129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561528+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"207'50","reported_seq":536,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:33.697865+0000","last_change":"2024-03-24T12:35:59.627748+0000","last_active":"2024-03-24T12:36:33.697865+0000","last_peered":"2024-03-24T12:36:33.697865+0000","last_clean":"2024-03-24T12:36:33.697865+0000","last_became_active":"2024-03-24T12:26:05.322210+0000","last_became_peered":"2024-03-24T12:26:05.322210+0000","last_unstale":"2024-03-24T12:36:33.697865+0000","last_undegraded":"2024-03-24T12:36:33.697865+0000","last_fullsized":"2024-03-24T12:36:33.697865+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'38","last_scrub_stamp":"2024-03-24T12:35:59.627597+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:35:59.627597+0000","objects_scrubbed":10,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:21.694429+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":322,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.659552+0000","last_change":"2024-03-24T12:34:43.416737+0000","last_active":"2024-03-24T12:36:40.659552+0000","last_peered":"2024-03-24T12:36:40.659552+0000","last_clean":"2024-03-24T12:36:40.659552+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:36:40.659552+0000","last_undegraded":"2024-03-24T12:36:40.659552+0000","last_fullsized":"2024-03-24T12:36:40.659552+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:43.416627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:43.416627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":332,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.135532+0000","last_change":"2024-03-24T12:36:27.040344+0000","last_active":"2024-03-24T12:36:31.135532+0000","last_peered":"2024-03-24T12:36:31.135532+0000","last_clean":"2024-03-24T12:36:31.135532+0000","last_became_active":"2024-03-24T12:26:09.645824+0000","last_became_peered":"2024-03-24T12:26:09.645824+0000","last_unstale":"2024-03-24T12:36:31.135532+0000","last_undegraded":"2024-03-24T12:36:31.135532+0000","last_fullsized":"2024-03-24T12:36:31.135532+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:27.040215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:36:27.040215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:34.613654+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":344,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.135039+0000","last_change":"2024-03-24T12:36:06.523229+0000","last_active":"2024-03-24T12:36:31.135039+0000","last_peered":"2024-03-24T12:36:31.135039+0000","last_clean":"2024-03-24T12:36:31.135039+0000","last_became_active":"2024-03-24T12:26:03.165496+0000","last_became_peered":"2024-03-24T12:26:03.165496+0000","last_unstale":"2024-03-24T12:36:31.135039+0000","last_undegraded":"2024-03-24T12:36:31.135039+0000","last_fullsized":"2024-03-24T12:36:31.135039+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:06.523108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:06.523108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:22.178048+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":351,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.135415+0000","last_change":"2024-03-24T12:36:03.272665+0000","last_active":"2024-03-24T12:36:31.135415+0000","last_peered":"2024-03-24T12:36:31.135415+0000","last_clean":"2024-03-24T12:36:31.135415+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:36:31.135415+0000","last_undegraded":"2024-03-24T12:36:31.135415+0000","last_fullsized":"2024-03-24T12:36:31.135415+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:03.272253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:03.272253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:14.708381+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"207'22","reported_seq":372,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.141028+0000","last_change":"2024-03-24T12:35:58.709644+0000","last_active":"2024-03-24T12:36:31.141028+0000","last_peered":"2024-03-24T12:36:31.141028+0000","last_clean":"2024-03-24T12:35:54.687814+0000","last_became_active":"2024-03-24T12:35:58.709644+0000","last_became_peered":"2024-03-24T12:35:58.709644+0000","last_unstale":"2024-03-24T12:36:31.141028+0000","last_undegraded":"2024-03-24T12:35:58.700232+0000","last_fullsized":"2024-03-24T12:35:58.699170+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:33:02.509218+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:31:08.237387+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.509218+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.828301+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":330,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.136842+0000","last_change":"2024-03-24T12:35:58.703902+0000","last_active":"2024-03-24T12:36:31.136842+0000","last_peered":"2024-03-24T12:36:31.136842+0000","last_clean":"2024-03-24T12:35:54.680479+0000","last_became_active":"2024-03-24T12:35:58.703902+0000","last_became_peered":"2024-03-24T12:35:58.703902+0000","last_unstale":"2024-03-24T12:36:31.136842+0000","last_undegraded":"2024-03-24T12:36:31.136842+0000","last_fullsized":"2024-03-24T12:35:58.701457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:13.031712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:13.031712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.601535+0000","last_change":"2024-03-24T12:35:58.713785+0000","last_active":"2024-03-24T12:36:31.601535+0000","last_peered":"2024-03-24T12:36:31.601535+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.713785+0000","last_became_peered":"2024-03-24T12:35:58.713785+0000","last_unstale":"2024-03-24T12:36:31.601535+0000","last_undegraded":"2024-03-24T12:36:31.601535+0000","last_fullsized":"2024-03-24T12:35:58.700902+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:18.057286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:18.057286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":589,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.133789+0000","last_change":"2024-03-24T12:35:16.741056+0000","last_active":"2024-03-24T12:36:31.133789+0000","last_peered":"2024-03-24T12:36:31.133789+0000","last_clean":"2024-03-24T12:36:31.133789+0000","last_became_active":"2024-03-24T12:25:52.687199+0000","last_became_peered":"2024-03-24T12:25:52.687199+0000","last_unstale":"2024-03-24T12:36:31.133789+0000","last_undegraded":"2024-03-24T12:36:31.133789+0000","last_fullsized":"2024-03-24T12:36:31.133789+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:35:16.740813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.740813+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:33.220003+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":331,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.596565+0000","last_change":"2024-03-24T12:35:58.710261+0000","last_active":"2024-03-24T12:36:31.596565+0000","last_peered":"2024-03-24T12:36:31.596565+0000","last_clean":"2024-03-24T12:35:54.688709+0000","last_became_active":"2024-03-24T12:35:58.710261+0000","last_became_peered":"2024-03-24T12:35:58.710261+0000","last_unstale":"2024-03-24T12:36:31.596565+0000","last_undegraded":"2024-03-24T12:36:31.596565+0000","last_fullsized":"2024-03-24T12:35:58.701753+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:24.345279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:24.345279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458572+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"207'28","reported_seq":507,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:32.567578+0000","last_change":"2024-03-24T12:36:19.012320+0000","last_active":"2024-03-24T12:36:32.567578+0000","last_peered":"2024-03-24T12:36:32.567578+0000","last_clean":"2024-03-24T12:36:32.567578+0000","last_became_active":"2024-03-24T12:26:05.358152+0000","last_became_peered":"2024-03-24T12:26:05.358152+0000","last_unstale":"2024-03-24T12:36:32.567578+0000","last_undegraded":"2024-03-24T12:36:32.567578+0000","last_fullsized":"2024-03-24T12:36:32.567578+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'28","last_scrub_stamp":"2024-03-24T12:36:19.012188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:36:19.012188+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:24.659484+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":323,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.134611+0000","last_change":"2024-03-24T12:35:58.711216+0000","last_active":"2024-03-24T12:36:31.134611+0000","last_peered":"2024-03-24T12:36:31.134611+0000","last_clean":"2024-03-24T12:35:54.688099+0000","last_became_active":"2024-03-24T12:35:58.711216+0000","last_became_peered":"2024-03-24T12:35:58.711216+0000","last_unstale":"2024-03-24T12:36:31.134611+0000","last_undegraded":"2024-03-24T12:36:31.134611+0000","last_fullsized":"2024-03-24T12:35:58.699124+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.802609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.802609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":212,"state":"creating+peering","last_fresh":"2024-03-24T12:36:40.657783+0000","last_change":"2024-03-24T12:36:40.657783+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.657783+0000","last_undegraded":"2024-03-24T12:36:40.657783+0000","last_fullsized":"2024-03-24T12:36:40.657783+0000","mapping_epoch":212,"log_start":"0'0","ondisk_log_start":"0'0","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.139262+0000","last_change":"2024-03-24T12:35:58.717519+0000","last_active":"2024-03-24T12:36:31.139262+0000","last_peered":"2024-03-24T12:36:31.139262+0000","last_clean":"2024-03-24T12:35:54.687071+0000","last_became_active":"2024-03-24T12:35:58.717519+0000","last_became_peered":"2024-03-24T12:35:58.717519+0000","last_unstale":"2024-03-24T12:36:31.139262+0000","last_undegraded":"2024-03-24T12:36:31.139262+0000","last_fullsized":"2024-03-24T12:35:58.696848+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:59.974279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:59.974279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:42.842002+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":338,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.659290+0000","last_change":"2024-03-24T12:36:13.630102+0000","last_active":"2024-03-24T12:36:40.659290+0000","last_peered":"2024-03-24T12:36:40.659290+0000","last_clean":"2024-03-24T12:36:40.659290+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:36:40.659290+0000","last_undegraded":"2024-03-24T12:36:40.659290+0000","last_fullsized":"2024-03-24T12:36:40.659290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:13.629968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:13.629968+0000","last_clean_scrub_stamp":"2024-03-24T12:36:13.629968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:21.143641+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":981,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:36.215098+0000","last_change":"2024-03-24T12:36:18.870565+0000","last_active":"2024-03-24T12:36:36.215098+0000","last_peered":"2024-03-24T12:36:36.215098+0000","last_clean":"2024-03-24T12:36:36.215098+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:36:36.215098+0000","last_undegraded":"2024-03-24T12:36:36.215098+0000","last_fullsized":"2024-03-24T12:36:36.215098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:36:18.870439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:18.870439+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:26.750950+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":627,"num_read_kb":622,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"207'24","reported_seq":361,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.660687+0000","last_change":"2024-03-24T12:34:35.069650+0000","last_active":"2024-03-24T12:36:40.660687+0000","last_peered":"2024-03-24T12:36:40.660687+0000","last_clean":"2024-03-24T12:36:40.660687+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:36:40.660687+0000","last_undegraded":"2024-03-24T12:36:40.660687+0000","last_fullsized":"2024-03-24T12:36:40.660687+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-24T12:34:35.069369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:35.069369+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":22,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.141360+0000","last_change":"2024-03-24T12:35:58.708969+0000","last_active":"2024-03-24T12:36:31.141360+0000","last_peered":"2024-03-24T12:36:31.141360+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.708969+0000","last_became_peered":"2024-03-24T12:35:58.708969+0000","last_unstale":"2024-03-24T12:36:31.141360+0000","last_undegraded":"2024-03-24T12:36:31.141360+0000","last_fullsized":"2024-03-24T12:35:58.699657+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:52.136918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:52.136918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:31.766767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":332,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.138756+0000","last_change":"2024-03-24T12:36:22.696662+0000","last_active":"2024-03-24T12:36:31.138756+0000","last_peered":"2024-03-24T12:36:31.138756+0000","last_clean":"2024-03-24T12:36:31.138756+0000","last_became_active":"2024-03-24T12:26:09.643728+0000","last_became_peered":"2024-03-24T12:26:09.643728+0000","last_unstale":"2024-03-24T12:36:31.138756+0000","last_undegraded":"2024-03-24T12:36:31.138756+0000","last_fullsized":"2024-03-24T12:36:31.138756+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:22.696538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:36:22.696538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:37.168103+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":339,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.139469+0000","last_change":"2024-03-24T12:35:58.713031+0000","last_active":"2024-03-24T12:36:31.139469+0000","last_peered":"2024-03-24T12:36:31.139469+0000","last_clean":"2024-03-24T12:35:54.687290+0000","last_became_active":"2024-03-24T12:35:58.713031+0000","last_became_peered":"2024-03-24T12:35:58.713031+0000","last_unstale":"2024-03-24T12:36:31.139469+0000","last_undegraded":"2024-03-24T12:36:31.139469+0000","last_fullsized":"2024-03-24T12:35:58.698188+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:54.953570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.953570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:31.766818+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"207'37","reported_seq":388,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:40.660153+0000","last_change":"2024-03-24T12:35:58.711927+0000","last_active":"2024-03-24T12:36:40.660153+0000","last_peered":"2024-03-24T12:36:40.660153+0000","last_clean":"2024-03-24T12:35:54.682985+0000","last_became_active":"2024-03-24T12:35:58.711927+0000","last_became_peered":"2024-03-24T12:35:58.711927+0000","last_unstale":"2024-03-24T12:36:40.660153+0000","last_undegraded":"2024-03-24T12:35:58.703907+0000","last_fullsized":"2024-03-24T12:35:58.703783+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'27","last_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_deep_scrub":"195'27","last_deep_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.981148+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:42.505141+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":331,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.597346+0000","last_change":"2024-03-24T12:35:58.709902+0000","last_active":"2024-03-24T12:36:31.597346+0000","last_peered":"2024-03-24T12:36:31.597346+0000","last_clean":"2024-03-24T12:35:54.684383+0000","last_became_active":"2024-03-24T12:35:58.709902+0000","last_became_peered":"2024-03-24T12:35:58.709902+0000","last_unstale":"2024-03-24T12:36:31.597346+0000","last_undegraded":"2024-03-24T12:36:31.597346+0000","last_fullsized":"2024-03-24T12:35:58.699174+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:44.164114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.164114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.017348+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":318,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.658516+0000","last_change":"2024-03-24T12:36:00.263886+0000","last_active":"2024-03-24T12:36:40.658516+0000","last_peered":"2024-03-24T12:36:40.658516+0000","last_clean":"2024-03-24T12:36:40.658516+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:36:40.658516+0000","last_undegraded":"2024-03-24T12:36:40.658516+0000","last_fullsized":"2024-03-24T12:36:40.658516+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:00.263629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:36:00.263629+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:01.367630+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.597661+0000","last_change":"2024-03-24T12:35:58.711061+0000","last_active":"2024-03-24T12:36:31.597661+0000","last_peered":"2024-03-24T12:36:31.597661+0000","last_clean":"2024-03-24T12:26:02.143860+0000","last_became_active":"2024-03-24T12:35:58.711061+0000","last_became_peered":"2024-03-24T12:35:58.711061+0000","last_unstale":"2024-03-24T12:36:31.597661+0000","last_undegraded":"2024-03-24T12:36:31.597661+0000","last_fullsized":"2024-03-24T12:35:58.704709+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:00.924106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.924106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"207'25","reported_seq":370,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.139363+0000","last_change":"2024-03-24T12:35:58.704200+0000","last_active":"2024-03-24T12:36:31.139363+0000","last_peered":"2024-03-24T12:36:31.139363+0000","last_clean":"2024-03-24T12:35:54.687582+0000","last_became_active":"2024-03-24T12:35:58.704200+0000","last_became_peered":"2024-03-24T12:35:58.704200+0000","last_unstale":"2024-03-24T12:36:31.139363+0000","last_undegraded":"2024-03-24T12:35:58.697842+0000","last_fullsized":"2024-03-24T12:35:58.697725+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:33:07.518368+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.518368+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.828282+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"205'9","reported_seq":462,"reported_epoch":212,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:40.660111+0000","last_change":"2024-03-24T12:35:58.709852+0000","last_active":"2024-03-24T12:36:40.660111+0000","last_peered":"2024-03-24T12:36:40.660111+0000","last_clean":"2024-03-24T12:35:54.682745+0000","last_became_active":"2024-03-24T12:35:58.709852+0000","last_became_peered":"2024-03-24T12:35:58.709852+0000","last_unstale":"2024-03-24T12:36:40.660111+0000","last_undegraded":"2024-03-24T12:35:58.706093+0000","last_fullsized":"2024-03-24T12:35:58.705326+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:34:55.749381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:55.749381+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":["0","6"],"object_location_counts":[{"shards":"0,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.134409+0000","last_change":"2024-03-24T12:36:03.769297+0000","last_active":"2024-03-24T12:36:31.134409+0000","last_peered":"2024-03-24T12:36:31.134409+0000","last_clean":"2024-03-24T12:36:31.134409+0000","last_became_active":"2024-03-24T12:26:09.645615+0000","last_became_peered":"2024-03-24T12:26:09.645615+0000","last_unstale":"2024-03-24T12:36:31.134409+0000","last_undegraded":"2024-03-24T12:36:31.134409+0000","last_fullsized":"2024-03-24T12:36:31.134409+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:03.769158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:36:03.769158+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:26.365465+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":338,"reported_epoch":212,"state":"active+clean","last_fresh":"2024-03-24T12:36:40.660731+0000","last_change":"2024-03-24T12:36:12.676345+0000","last_active":"2024-03-24T12:36:40.660731+0000","last_peered":"2024-03-24T12:36:40.660731+0000","last_clean":"2024-03-24T12:36:40.660731+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:36:40.660731+0000","last_undegraded":"2024-03-24T12:36:40.660731+0000","last_fullsized":"2024-03-24T12:36:40.660731+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:36:12.676228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:12.676228+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:42.162089+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"207'35","reported_seq":384,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.514022+0000","last_change":"2024-03-24T12:35:58.704494+0000","last_active":"2024-03-24T12:36:31.514022+0000","last_peered":"2024-03-24T12:36:31.514022+0000","last_clean":"2024-03-24T12:35:54.676736+0000","last_became_active":"2024-03-24T12:35:58.704494+0000","last_became_peered":"2024-03-24T12:35:58.704494+0000","last_unstale":"2024-03-24T12:36:31.514022+0000","last_undegraded":"2024-03-24T12:35:58.695857+0000","last_fullsized":"2024-03-24T12:35:58.695729+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'21","last_scrub_stamp":"2024-03-24T12:35:14.829466+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:35:14.829466+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561485+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":456,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:33.335601+0000","last_change":"2024-03-24T12:33:35.289329+0000","last_active":"2024-03-24T12:36:33.335601+0000","last_peered":"2024-03-24T12:36:33.335601+0000","last_clean":"2024-03-24T12:36:33.335601+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:36:33.335601+0000","last_undegraded":"2024-03-24T12:36:33.335601+0000","last_fullsized":"2024-03-24T12:36:33.335601+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:33:35.288932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:33:35.288932+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":324,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.514155+0000","last_change":"2024-03-24T12:36:01.762056+0000","last_active":"2024-03-24T12:36:31.514155+0000","last_peered":"2024-03-24T12:36:31.514155+0000","last_clean":"2024-03-24T12:36:31.514155+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:36:31.514155+0000","last_undegraded":"2024-03-24T12:36:31.514155+0000","last_fullsized":"2024-03-24T12:36:31.514155+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:01.761970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:36:01.761970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:23.217926+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.599172+0000","last_change":"2024-03-24T12:35:58.349567+0000","last_active":"2024-03-24T12:36:31.599172+0000","last_peered":"2024-03-24T12:36:31.599172+0000","last_clean":"2024-03-24T12:36:31.599172+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:36:31.599172+0000","last_undegraded":"2024-03-24T12:36:31.599172+0000","last_fullsized":"2024-03-24T12:36:31.599172+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:58.349481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:58.349481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:05.016663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"207'30","reported_seq":382,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.599336+0000","last_change":"2024-03-24T12:36:25.420249+0000","last_active":"2024-03-24T12:36:31.599336+0000","last_peered":"2024-03-24T12:36:31.599336+0000","last_clean":"2024-03-24T12:36:31.599336+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:36:31.599336+0000","last_undegraded":"2024-03-24T12:36:31.599336+0000","last_fullsized":"2024-03-24T12:36:31.599336+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'30","last_scrub_stamp":"2024-03-24T12:36:25.420105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:36:25.420105+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:54.732851+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":336,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.513639+0000","last_change":"2024-03-24T12:36:23.674111+0000","last_active":"2024-03-24T12:36:31.513639+0000","last_peered":"2024-03-24T12:36:31.513639+0000","last_clean":"2024-03-24T12:36:31.513639+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:36:31.513639+0000","last_undegraded":"2024-03-24T12:36:31.513639+0000","last_fullsized":"2024-03-24T12:36:31.513639+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:23.674016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:36:23.674016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:48.074217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.139962+0000","last_change":"2024-03-24T12:35:58.711067+0000","last_active":"2024-03-24T12:36:31.139962+0000","last_peered":"2024-03-24T12:36:31.139962+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.711067+0000","last_became_peered":"2024-03-24T12:35:58.711067+0000","last_unstale":"2024-03-24T12:36:31.139962+0000","last_undegraded":"2024-03-24T12:36:31.139962+0000","last_fullsized":"2024-03-24T12:35:58.696416+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:36.210440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.210440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.828324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"205'10","reported_seq":467,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:31.515005+0000","last_change":"2024-03-24T12:35:58.712862+0000","last_active":"2024-03-24T12:36:31.515005+0000","last_peered":"2024-03-24T12:36:31.515005+0000","last_clean":"2024-03-24T12:35:54.678342+0000","last_became_active":"2024-03-24T12:35:58.712862+0000","last_became_peered":"2024-03-24T12:35:58.712862+0000","last_unstale":"2024-03-24T12:36:31.515005+0000","last_undegraded":"2024-03-24T12:35:58.700712+0000","last_fullsized":"2024-03-24T12:35:58.700587+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'9","last_scrub_stamp":"2024-03-24T12:34:44.703379+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.703379+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.599296+0000","last_change":"2024-03-24T12:36:09.267653+0000","last_active":"2024-03-24T12:36:31.599296+0000","last_peered":"2024-03-24T12:36:31.599296+0000","last_clean":"2024-03-24T12:36:31.599296+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:36:31.599296+0000","last_undegraded":"2024-03-24T12:36:31.599296+0000","last_fullsized":"2024-03-24T12:36:31.599296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:09.267517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:09.267517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:16.394817+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"207'34","reported_seq":364,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.135545+0000","last_change":"2024-03-24T12:36:01.230671+0000","last_active":"2024-03-24T12:36:31.135545+0000","last_peered":"2024-03-24T12:36:31.135545+0000","last_clean":"2024-03-24T12:36:31.135545+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:36:31.135545+0000","last_undegraded":"2024-03-24T12:36:31.135545+0000","last_fullsized":"2024-03-24T12:36:31.135545+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"92'22","last_scrub_stamp":"2024-03-24T12:36:01.230424+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:36:01.230424+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:30.615318+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":310,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.598191+0000","last_change":"2024-03-24T12:35:58.715617+0000","last_active":"2024-03-24T12:36:31.598191+0000","last_peered":"2024-03-24T12:36:31.598191+0000","last_clean":"2024-03-24T12:35:54.692110+0000","last_became_active":"2024-03-24T12:35:58.715617+0000","last_became_peered":"2024-03-24T12:35:58.715617+0000","last_unstale":"2024-03-24T12:36:31.598191+0000","last_undegraded":"2024-03-24T12:36:31.598191+0000","last_fullsized":"2024-03-24T12:35:58.703670+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:19.376894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.376894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:36.458583+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":22,"reported_epoch":211,"state":"active+undersized","last_fresh":"2024-03-24T12:36:31.514700+0000","last_change":"2024-03-24T12:35:58.707619+0000","last_active":"2024-03-24T12:36:31.514700+0000","last_peered":"2024-03-24T12:36:31.514700+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.707619+0000","last_became_peered":"2024-03-24T12:35:58.707619+0000","last_unstale":"2024-03-24T12:36:31.514700+0000","last_undegraded":"2024-03-24T12:36:31.514700+0000","last_fullsized":"2024-03-24T12:35:58.700312+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:15.004831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.004831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.513803+0000","last_change":"2024-03-24T12:36:00.774645+0000","last_active":"2024-03-24T12:36:31.513803+0000","last_peered":"2024-03-24T12:36:31.513803+0000","last_clean":"2024-03-24T12:36:31.513803+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:36:31.513803+0000","last_undegraded":"2024-03-24T12:36:31.513803+0000","last_fullsized":"2024-03-24T12:36:31.513803+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:00.774537+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:00.774537+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:04.183652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"207'19","reported_seq":362,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.514578+0000","last_change":"2024-03-24T12:34:53.872190+0000","last_active":"2024-03-24T12:36:31.514578+0000","last_peered":"2024-03-24T12:36:31.514578+0000","last_clean":"2024-03-24T12:36:31.514578+0000","last_became_active":"2024-03-24T12:26:05.315510+0000","last_became_peered":"2024-03-24T12:26:05.315510+0000","last_unstale":"2024-03-24T12:36:31.514578+0000","last_undegraded":"2024-03-24T12:36:31.514578+0000","last_fullsized":"2024-03-24T12:36:31.514578+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:34:53.872101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.872101+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.561547+0000","scrub_duration":223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":331,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.135716+0000","last_change":"2024-03-24T12:36:16.268988+0000","last_active":"2024-03-24T12:36:31.135716+0000","last_peered":"2024-03-24T12:36:31.135716+0000","last_clean":"2024-03-24T12:36:31.135716+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:36:31.135716+0000","last_undegraded":"2024-03-24T12:36:31.135716+0000","last_fullsized":"2024-03-24T12:36:31.135716+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:16.268695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:36:16.268695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:33.530946+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":335,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.139811+0000","last_change":"2024-03-24T12:35:04.121930+0000","last_active":"2024-03-24T12:36:31.139811+0000","last_peered":"2024-03-24T12:36:31.139811+0000","last_clean":"2024-03-24T12:36:31.139811+0000","last_became_active":"2024-03-24T12:26:07.329740+0000","last_became_peered":"2024-03-24T12:26:07.329740+0000","last_unstale":"2024-03-24T12:36:31.139811+0000","last_undegraded":"2024-03-24T12:36:31.139811+0000","last_fullsized":"2024-03-24T12:36:31.139811+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.121802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.121802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:34.828368+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":343,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.597087+0000","last_change":"2024-03-24T12:36:03.979071+0000","last_active":"2024-03-24T12:36:31.597087+0000","last_peered":"2024-03-24T12:36:31.597087+0000","last_clean":"2024-03-24T12:36:31.597087+0000","last_became_active":"2024-03-24T12:26:03.162163+0000","last_became_peered":"2024-03-24T12:26:03.162163+0000","last_unstale":"2024-03-24T12:36:31.597087+0000","last_undegraded":"2024-03-24T12:36:31.597087+0000","last_fullsized":"2024-03-24T12:36:31.597087+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:03.978953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:03.978953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:14.036849+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"207'39","reported_seq":388,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.515505+0000","last_change":"2024-03-24T12:36:22.669801+0000","last_active":"2024-03-24T12:36:31.515505+0000","last_peered":"2024-03-24T12:36:31.515505+0000","last_clean":"2024-03-24T12:36:31.515505+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:36:31.515505+0000","last_undegraded":"2024-03-24T12:36:31.515505+0000","last_fullsized":"2024-03-24T12:36:31.515505+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-24T12:36:22.669686+0000","last_deep_scrub":"207'39","last_deep_scrub_stamp":"2024-03-24T12:36:22.669686+0000","last_clean_scrub_stamp":"2024-03-24T12:36:22.669686+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:33.665884+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-24T12:36:40.645298+0000","last_change":"2024-03-24T12:36:40.645298+0000","last_active":"2024-03-24T12:36:40.645298+0000","last_peered":"2024-03-24T12:36:40.645298+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-24T12:36:40.645298+0000","last_undegraded":"2024-03-24T12:36:40.645298+0000","last_fullsized":"2024-03-24T12:36:40.645298+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":331,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.515116+0000","last_change":"2024-03-24T12:35:55.870453+0000","last_active":"2024-03-24T12:36:31.515116+0000","last_peered":"2024-03-24T12:36:31.515116+0000","last_clean":"2024-03-24T12:36:31.515116+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:36:31.515116+0000","last_undegraded":"2024-03-24T12:36:31.515116+0000","last_fullsized":"2024-03-24T12:36:31.515116+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:55.870324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:55.870324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:16.357978+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"205'9","reported_seq":467,"reported_epoch":211,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:36:33.335403+0000","last_change":"2024-03-24T12:35:58.752262+0000","last_active":"2024-03-24T12:36:33.335403+0000","last_peered":"2024-03-24T12:36:33.335403+0000","last_clean":"2024-03-24T12:35:54.684037+0000","last_became_active":"2024-03-24T12:35:58.752262+0000","last_became_peered":"2024-03-24T12:35:58.752262+0000","last_unstale":"2024-03-24T12:36:33.335403+0000","last_undegraded":"2024-03-24T12:35:58.701740+0000","last_fullsized":"2024-03-24T12:35:58.701602+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:35:00.998713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.998713+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:36:41.149941+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":327,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.599331+0000","last_change":"2024-03-24T12:36:21.502153+0000","last_active":"2024-03-24T12:36:31.599331+0000","last_peered":"2024-03-24T12:36:31.599331+0000","last_clean":"2024-03-24T12:36:31.599331+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:36:31.599331+0000","last_undegraded":"2024-03-24T12:36:31.599331+0000","last_fullsized":"2024-03-24T12:36:31.599331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:21.502060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:21.502060+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:28.221556+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"207'20","reported_seq":369,"reported_epoch":211,"state":"active+clean","last_fresh":"2024-03-24T12:36:31.515717+0000","last_change":"2024-03-24T12:36:25.652455+0000","last_active":"2024-03-24T12:36:31.515717+0000","last_peered":"2024-03-24T12:36:31.515717+0000","last_clean":"2024-03-24T12:36:31.515717+0000","last_became_active":"2024-03-24T12:26:05.322262+0000","last_became_peered":"2024-03-24T12:26:05.322262+0000","last_unstale":"2024-03-24T12:36:31.515717+0000","last_undegraded":"2024-03-24T12:36:31.515717+0000","last_fullsized":"2024-03-24T12:36:31.515717+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:36:25.652296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:36:25.652296+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:37:29.059194+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":6,"acting":6,"num_store_stats":1},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":627,"num_read_kb":622,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":84,"acting":84,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":45,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1125,"num_read_kb":882,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":750,"ondisk_log_size":750,"up":88,"acting":88,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70,"ondisk_log_size":70,"up":82,"acting":82,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":82,"acting":82,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037891,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":285152,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93460000,"statfs":{"total":95995035648,"available":95703040000,"internally_reserved":0,"allocated":1331200,"data_stored":962432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":290650570},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332233,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":303176,"kb_used_data":1276,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":93441976,"statfs":{"total":95995035648,"available":95684583424,"internally_reserved":0,"allocated":1306624,"data_stored":961844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":309131719},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575012,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":520180,"kb_used_data":236332,"kb_used_omap":1,"kb_used_meta":283838,"kb_avail":93224972,"statfs":{"total":95995035648,"available":95462371328,"internally_reserved":0,"allocated":242003968,"data_stored":241610470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":290650565},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738537,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":371232,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":93373920,"statfs":{"total":95995035648,"available":95614894080,"internally_reserved":0,"allocated":1331200,"data_stored":961776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411486,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392836,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93352316,"statfs":{"total":95995035648,"available":95592771584,"internally_reserved":0,"allocated":712704,"data_stored":370936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":401537477},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117143,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":338732,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":337982,"kb_avail":93406420,"statfs":{"total":95995035648,"available":95648174080,"internally_reserved":0,"allocated":753664,"data_stored":372380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":346094023},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920915,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":212380,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":93532772,"statfs":{"total":95995035648,"available":95777558528,"internally_reserved":0,"allocated":737280,"data_stored":371206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":216725957},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626566,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957988,"kb_used_data":688028,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92787164,"statfs":{"total":95995035648,"available":95014055936,"internally_reserved":0,"allocated":704540672,"data_stored":704173829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:36:43.547 INFO:tasks.thrashosds.thrasher:Setting 5.15 to [5, 7, 2, 3, 6, 0] 2024-03-24T12:36:43.547 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.15', '5', '7', '2', '3', '6', '0'] 2024-03-24T12:36:43.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pg-upmap-items 5.15 5 7 2 3 6 0 2024-03-24T12:36:43.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:43.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:43.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:43.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:43.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:43.725 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:43.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:43.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:43.769 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[41500]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:36:43.769 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[41500]: pgmap v431: 148 pgs: 3 creating+peering, 9 unknown, 16 active+undersized+degraded, 35 active+undersized, 85 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:43.769 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[41500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:36:43.769 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:36:43.769 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1204776765' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:36:43.769 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[41500]: osdmap e214: 8 total, 7 up, 8 in 2024-03-24T12:36:43.769 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[29919]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:36:43.769 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[29919]: pgmap v431: 148 pgs: 3 creating+peering, 9 unknown, 16 active+undersized+degraded, 35 active+undersized, 85 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:43.770 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[29919]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:36:43.770 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:36:43.770 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1204776765' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:36:43.770 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:43 smithi029 ceph-mon[29919]: osdmap e214: 8 total, 7 up, 8 in 2024-03-24T12:36:43.782 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:36:43.782 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:36:43.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:43.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:43.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:43.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:43.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:43.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:43.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:44.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:44.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:44.103 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:44.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:44.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:43 smithi190 ceph-mon[33748]: from='client.24962 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:36:44.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:43 smithi190 ceph-mon[33748]: pgmap v431: 148 pgs: 3 creating+peering, 9 unknown, 16 active+undersized+degraded, 35 active+undersized, 85 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:44.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:43 smithi190 ceph-mon[33748]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:36:44.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:43 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:36:44.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:43 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1204776765' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:36:44.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:43 smithi190 ceph-mon[33748]: osdmap e214: 8 total, 7 up, 8 in 2024-03-24T12:36:44.204 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:44.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:44.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:44.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:44.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:44.357 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:44.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:44.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:44.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:44.482 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:44.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:44.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:44.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:44.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:44.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:44.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:44.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:44.736 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:44.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:44.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:44.860 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:44.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:44.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:44.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:44.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:44.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:45.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:45.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:45.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:45.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:45.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T12:36:45.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:45.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:45.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:45.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:45.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:45.373 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:45.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:45.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:45.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:45.501 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:45.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:45.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:45.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:45.626 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:45.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:45.690 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:45.692 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:45.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:45.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:45.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:45.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:45.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:45.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:45.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:45.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:46.014 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:46.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:46.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:46.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:46.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:46 smithi029 ceph-mon[29919]: pgmap v433: 148 pgs: 3 creating+peering, 16 active+undersized+degraded, 39 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:46.132 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:46 smithi029 ceph-mon[41500]: pgmap v433: 148 pgs: 3 creating+peering, 16 active+undersized+degraded, 39 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:46.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:46.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:46.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:46.270 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:46.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:46.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:46.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:46 smithi190 ceph-mon[33748]: pgmap v433: 148 pgs: 3 creating+peering, 16 active+undersized+degraded, 39 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:46.398 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:46.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:46.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:46.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:46.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:46.523 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:46.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:46.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:46.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:46.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:46.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:46.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:36:46.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:46.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:46.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:46.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:46.898 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:46.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:46.999 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:47.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:47.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:47.129 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:47.157 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:47.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:47.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:47.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:47.283 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:47.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:47.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:47.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:47.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:47.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:47.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:47.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:47.671 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:47.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:47.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:47.796 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:47.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:47.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:47.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:36:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:36:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:36:47.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:47.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:48.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:48.048 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:48.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:48.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:48.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:48.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:48.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:48.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:48.304 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:48.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:48.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:48 smithi190 ceph-mon[33748]: pgmap v434: 148 pgs: 16 active+undersized+degraded, 42 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:48.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:48.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:48.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:36:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:36:48.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:48 smithi029 ceph-mon[41500]: pgmap v434: 148 pgs: 16 active+undersized+degraded, 42 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:48 smithi029 ceph-mon[29919]: pgmap v434: 148 pgs: 16 active+undersized+degraded, 42 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:36:48.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:36:48.429 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:48.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:48.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:48.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:48.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:48.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:48.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:48.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:48.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:48.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:48.813 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:48.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:48.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:48.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:48.941 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:48.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:48.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:49.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:49.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:49.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:49.071 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:49.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:49.194 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:49.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:49.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:49.268 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:36:49.268 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:49.268 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:49.288 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:36:49.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:49.308 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:36:49.319 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:49.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:49.329 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:36:49.349 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:36:49.371 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:36:49.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:49.392 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:36:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:36:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:36:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:36:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:36:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:36:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:36:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:36:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:36:49.405 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:49.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:49.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:49.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:49.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:49.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:49.577 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:49.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:49.679 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:49.703 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:49.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:49.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:49.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:49.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:49.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:49.956 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:49.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:49.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:50.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:50.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:50.082 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:50.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:50.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:50.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:50.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:50.279 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:50 smithi029 ceph-mon[29919]: pgmap v435: 148 pgs: 16 active+undersized+degraded, 42 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:50.279 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:50 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3017017158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:50.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:50 smithi029 ceph-mon[41500]: pgmap v435: 148 pgs: 16 active+undersized+degraded, 42 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:50.279 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:50 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3017017158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:50.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:50.336 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:50.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:50.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:50 smithi190 ceph-mon[33748]: pgmap v435: 148 pgs: 16 active+undersized+degraded, 42 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:50 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3017017158' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:36:50.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:50.466 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:50.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:50.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:50.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:50.591 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:50.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:50.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:50.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:50.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:50.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:50.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:50.845 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:50.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:50.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:50.947 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:50.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:50.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:50.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:51.054 INFO:teuthology.orchestra.run.smithi029.stderr:set 5.15 pg_upmap_items mapping to [5->7,2->3,6->0] 2024-03-24T12:36:51.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:51.101 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:51.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:51.103 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:51 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3342144447' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.15", "id": [5, 7, 2, 3, 6, 0]}]: dispatch 2024-03-24T12:36:51.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:51 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3342144447' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.15", "id": [5, 7, 2, 3, 6, 0]}]: dispatch 2024-03-24T12:36:51.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:51.226 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:51.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:51.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:51.352 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:51.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:51.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:51 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3342144447' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.15", "id": [5, 7, 2, 3, 6, 0]}]: dispatch 2024-03-24T12:36:51.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:51.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:51.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:51.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:51.610 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:51.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:51.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:51.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:51.739 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:51.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:51.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:51.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:51.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:51.867 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:51.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:51.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:51.992 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:51.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-24T12:36:52.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:52.118 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:52.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:52.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:52.242 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:52.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:52.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:52.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:52.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:52.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:52 smithi190 ceph-mon[33748]: pgmap v436: 148 pgs: 16 active+undersized+degraded, 42 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:52.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:52 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3342144447' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.15", "id": [5, 7, 2, 3, 6, 0]}]': finished 2024-03-24T12:36:52.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:52 smithi190 ceph-mon[33748]: osdmap e215: 8 total, 7 up, 8 in 2024-03-24T12:36:52.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:52 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2700144128' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:36:52.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:52 smithi029 ceph-mon[29919]: pgmap v436: 148 pgs: 16 active+undersized+degraded, 42 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:52.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:52 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3342144447' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.15", "id": [5, 7, 2, 3, 6, 0]}]': finished 2024-03-24T12:36:52.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:52 smithi029 ceph-mon[29919]: osdmap e215: 8 total, 7 up, 8 in 2024-03-24T12:36:52.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:52 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2700144128' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:36:52.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:52 smithi029 ceph-mon[41500]: pgmap v436: 148 pgs: 16 active+undersized+degraded, 42 active+undersized, 90 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 53/594 objects degraded (8.923%) 2024-03-24T12:36:52.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:52 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3342144447' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.15", "id": [5, 7, 2, 3, 6, 0]}]': finished 2024-03-24T12:36:52.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:52 smithi029 ceph-mon[41500]: osdmap e215: 8 total, 7 up, 8 in 2024-03-24T12:36:52.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:52 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2700144128' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:36:52.469 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:52.497 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:52.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:52.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:52.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:52.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:52.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:52.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:52.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:52.753 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:52.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:52.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:52.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:52.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:52.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:52.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:53.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:53.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:53.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:53.134 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:53.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:53.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:53 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:36:53.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:53 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:36:53.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:53 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2700144128' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:36:53.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:53 smithi029 ceph-mon[29919]: osdmap e216: 8 total, 7 up, 8 in 2024-03-24T12:36:53.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:53 smithi029 ceph-mon[29919]: pgmap v439: 148 pgs: 15 active+undersized+degraded, 42 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 52/594 objects degraded (8.754%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:53.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:53 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:36:53.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:53 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:36:53.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:53 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2700144128' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:36:53.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:53 smithi029 ceph-mon[41500]: osdmap e216: 8 total, 7 up, 8 in 2024-03-24T12:36:53.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:53 smithi029 ceph-mon[41500]: pgmap v439: 148 pgs: 15 active+undersized+degraded, 42 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 52/594 objects degraded (8.754%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:53.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:53.260 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:53.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:53.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:53 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:36:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:53 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:36:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:53 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2700144128' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:36:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:53 smithi190 ceph-mon[33748]: osdmap e216: 8 total, 7 up, 8 in 2024-03-24T12:36:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:53 smithi190 ceph-mon[33748]: pgmap v439: 148 pgs: 15 active+undersized+degraded, 42 active+undersized, 91 active+clean; 582 KiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 52/594 objects degraded (8.754%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:53.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:53.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:53.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:53.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:53.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:53.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:53.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:53.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:53.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:53.647 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:53.647 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:53.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:53.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:53.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:53.865 DEBUG:teuthology.orchestra.run.smithi029:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-24T12:36:53.867 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T12:36:53.867 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T12:36:53.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:53.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:53.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:53.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:53.914 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:hints = 1 2024-03-24T12:36:53.915 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T12:36:53.915 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Object prefix: benchmark_data_smithi029_134815 2024-03-24T12:36:53.915 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:36:53.915 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 0 0 0 0 0 0 - 0 2024-03-24T12:36:54.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:36:54.015 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:36:54.044 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:54.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:54.045 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:54.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:54.174 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:54.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:54.208 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:54 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 52/594 objects degraded (8.754%), 15 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:54.209 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:54 smithi029 ceph-mon[29919]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:36:54.209 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:54 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 52/594 objects degraded (8.754%), 15 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:54.209 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:54 smithi029 ceph-mon[41500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:36:54.221 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:54.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:54.322 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:54.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:54.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:54 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 52/594 objects degraded (8.754%), 15 pgs degraded (PG_DEGRADED) 2024-03-24T12:36:54.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:54 smithi190 ceph-mon[33748]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:36:54.423 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:54.464 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:54.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:54.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:54.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:54.590 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:54.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:54.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:54.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:54.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:54.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:54.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:54.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:54.868 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:54.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:54.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:54.915 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 1 16 2788 2772 173.199 173.25 0.00343802 0.0057486 2024-03-24T12:36:54.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:55.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:55.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:55.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:55.084 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:36:55.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:55.144 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:55.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:55.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:55.246 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:55.286 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:55.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:55.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:55 smithi190 ceph-mon[33748]: pgmap v440: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 26 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.1 MiB/s wr, 51 op/s; 154/1791 objects degraded (8.599%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:55.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:55 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4029003643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:36:55.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:55 smithi029 ceph-mon[41500]: pgmap v440: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 26 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.1 MiB/s wr, 51 op/s; 154/1791 objects degraded (8.599%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:55 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4029003643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:36:55.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:55 smithi029 ceph-mon[29919]: pgmap v440: 148 pgs: 16 active+undersized+degraded, 41 active+undersized, 91 active+clean; 26 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.1 MiB/s wr, 51 op/s; 154/1791 objects degraded (8.599%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:55 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4029003643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:36:55.418 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:55.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:55.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:55.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:55.561 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:55.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:55.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:55.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:55.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:55.848 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:55.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:55.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:55.915 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 2 16 6140 6124 191.337 209.5 0.00230988 0.00521476 2024-03-24T12:36:55.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:55.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:55.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:55.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:56.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:36:56.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:56.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:56.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:56.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:56.277 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:56.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:56.318 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:56 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:36:56.318 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:56 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4029003643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:36:56.318 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:56 smithi029 ceph-mon[41500]: osdmap e217: 8 total, 7 up, 8 in 2024-03-24T12:36:56.318 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:56 smithi029 ceph-mon[41500]: 4.6 scrub starts 2024-03-24T12:36:56.318 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:56 smithi029 ceph-mon[41500]: 4.6 scrub ok 2024-03-24T12:36:56.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:56 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:36:56.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:56 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4029003643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:36:56.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:56 smithi029 ceph-mon[29919]: osdmap e217: 8 total, 7 up, 8 in 2024-03-24T12:36:56.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:56 smithi029 ceph-mon[29919]: 4.6 scrub starts 2024-03-24T12:36:56.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:56 smithi029 ceph-mon[29919]: 4.6 scrub ok 2024-03-24T12:36:56.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:56.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:56 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:36:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:56 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4029003643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:36:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:56 smithi190 ceph-mon[33748]: osdmap e217: 8 total, 7 up, 8 in 2024-03-24T12:36:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:56 smithi190 ceph-mon[33748]: 4.6 scrub starts 2024-03-24T12:36:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:56 smithi190 ceph-mon[33748]: 4.6 scrub ok 2024-03-24T12:36:56.419 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:56.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:56.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:56.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:56.560 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:56.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:56.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:56.722 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:56.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:56.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:56.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:56.865 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:56.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:56.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:56.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 3 15 9210 9195 191.532 191.938 0.00240349 0.00521085 2024-03-24T12:36:56.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:36:57.007 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:57.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:57.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:36:57.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:57.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:57.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:57.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:57.270 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:57.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:57.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:57.327 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:36:57.327 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:36:57.327 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-mon[29919]: pgmap v442: 148 pgs: 19 active+undersized+degraded, 38 active+undersized, 91 active+clean; 103 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 17 MiB/s wr, 274 op/s; 1400/5529 objects degraded (25.321%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:57.327 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:36:57.327 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:36:57.327 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:36:57.327 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:36:57.327 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-mon[41500]: pgmap v442: 148 pgs: 19 active+undersized+degraded, 38 active+undersized, 91 active+clean; 103 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 17 MiB/s wr, 274 op/s; 1400/5529 objects degraded (25.321%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:57.328 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:36:57.328 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:36:57.366 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:36:57.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:57.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:57 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:36:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:57 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:36:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:57 smithi190 ceph-mon[33748]: pgmap v442: 148 pgs: 19 active+undersized+degraded, 38 active+undersized, 91 active+clean; 103 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 17 MiB/s wr, 274 op/s; 1400/5529 objects degraded (25.321%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:57 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:36:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:57 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:36:57.429 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:57.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:57.443 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:36:57.443 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:36:57.443 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-03-24T12:36:57.443 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-03-24T12:36:57.443 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:36:57.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:57.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:57.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:57.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:57.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:57.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:57.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:57.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:57.821 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:57.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:57.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:57.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 4 16 12116 12100 189.036 181.562 0.00457917 0.00528248 2024-03-24T12:36:57.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:36:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:36:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:36:57.924 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:57.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:57.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:58.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:58.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:58.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:58.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:58.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:58.283 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:36:57 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:36:58.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:58 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:36:58.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:58 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:36:58.284 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:58.295 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:36:58.295 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:36:58.295 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-24T12:36:58.295 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-03-24T12:36:58.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-24T12:36:58.334 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:58 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:36:58.335 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:58 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:36:58.335 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:58 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:58.335 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:58 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:36:58.335 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:58 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:36:58.335 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:58 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:36:58.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:58.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:58.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:58.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:58.471 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:58.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:58.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:58.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:58.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T12:36:58.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:58.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:58.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:58.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:36:58.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:36:58.773 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:58.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:58.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:36:58.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:58.906 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:58.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:58.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:58.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 5 15 14710 14695 183.664 162.188 0.00437461 0.00543685 2024-03-24T12:36:59.007 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:59.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:59.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:59.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:59.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:36:59.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:59.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:36:59.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:36:59.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:59.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:59.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:36:59.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:59 smithi190 ceph-mon[33748]: 3.e scrub starts 2024-03-24T12:36:59.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:59 smithi190 ceph-mon[33748]: 3.e scrub ok 2024-03-24T12:36:59.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:36:59 smithi190 ceph-mon[33748]: pgmap v443: 148 pgs: 22 active+undersized+degraded, 35 active+undersized, 91 active+clean; 331 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 702 op/s; 3628/16452 objects degraded (22.052%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:59.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:36:59.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:59 smithi029 ceph-mon[29919]: 3.e scrub starts 2024-03-24T12:36:59.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:59 smithi029 ceph-mon[29919]: 3.e scrub ok 2024-03-24T12:36:59.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:36:59 smithi029 ceph-mon[29919]: pgmap v443: 148 pgs: 22 active+undersized+degraded, 35 active+undersized, 91 active+clean; 331 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 702 op/s; 3628/16452 objects degraded (22.052%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:59.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:59 smithi029 ceph-mon[41500]: 3.e scrub starts 2024-03-24T12:36:59.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:59 smithi029 ceph-mon[41500]: 3.e scrub ok 2024-03-24T12:36:59.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:36:59 smithi029 ceph-mon[41500]: pgmap v443: 148 pgs: 22 active+undersized+degraded, 35 active+undersized, 91 active+clean; 331 MiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 44 MiB/s wr, 702 op/s; 3628/16452 objects degraded (22.052%); 0 B/s, 0 objects/s recovering 2024-03-24T12:36:59.443 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:59.444 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:36:59.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:59.574 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:59.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:59.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:36:59.717 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:36:59.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:59.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:36:59.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:36:59.847 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:36:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:36:59.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:36:59.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 6 16 17425 17409 181.322 169.625 0.00594466 0.00550855 2024-03-24T12:36:59.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:00.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:00.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:00.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:00.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:00.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:00.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:00.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:00.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:00.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:00.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:00 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 3628/16452 objects degraded (22.052%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T12:37:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:00 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:37:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:00 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:37:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:00 smithi190 ceph-mon[33748]: 6.18 scrub starts 2024-03-24T12:37:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:00 smithi190 ceph-mon[33748]: 6.18 scrub ok 2024-03-24T12:37:00.407 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:00.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:00.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:00 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 3628/16452 objects degraded (22.052%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T12:37:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:00 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:37:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:00 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:37:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:00 smithi029 ceph-mon[41500]: 6.18 scrub starts 2024-03-24T12:37:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:00 smithi029 ceph-mon[41500]: 6.18 scrub ok 2024-03-24T12:37:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:00 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 3628/16452 objects degraded (22.052%), 22 pgs degraded (PG_DEGRADED) 2024-03-24T12:37:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:00 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:37:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:00 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:37:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:00 smithi029 ceph-mon[29919]: 6.18 scrub starts 2024-03-24T12:37:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:00 smithi029 ceph-mon[29919]: 6.18 scrub ok 2024-03-24T12:37:00.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:00.538 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:00.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:00.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:00.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:00.679 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:00.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:00.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:00.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:00.819 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:00.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:37:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:37:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:37:00.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 7 16 19839 19823 176.965 150.875 0.00678318 0.00564471 2024-03-24T12:37:00.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:00.965 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:00.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:00.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:01.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:01.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:01.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:01.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:01.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:01.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:01 smithi029 ceph-mon[29919]: osdmap e218: 8 total, 7 up, 8 in 2024-03-24T12:37:01.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:01 smithi029 ceph-mon[29919]: pgmap v445: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 775 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s; 6349/37782 objects degraded (16.804%) 2024-03-24T12:37:01.250 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:01 smithi029 ceph-mon[41500]: osdmap e218: 8 total, 7 up, 8 in 2024-03-24T12:37:01.250 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:01 smithi029 ceph-mon[41500]: pgmap v445: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 775 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s; 6349/37782 objects degraded (16.804%) 2024-03-24T12:37:01.254 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:01.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:01.260 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:37:01.308 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:37:01.348 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:37:01.356 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:01 smithi190 ceph-mon[33748]: osdmap e218: 8 total, 7 up, 8 in 2024-03-24T12:37:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:01 smithi190 ceph-mon[33748]: pgmap v445: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 775 MiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 97 MiB/s wr, 1.55k op/s; 6349/37782 objects degraded (16.804%) 2024-03-24T12:37:01.394 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:37:01.397 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:01.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:01.444 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:37:01.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:01.500 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:37:01.544 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:37:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:37:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:37:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:37:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:37:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:37:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:37:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:37:01.545 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:37:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:01.548 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:01.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:01.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:01.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:01.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:01.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:01.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:01.820 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:01.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:01.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:01.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 8 16 22201 22185 173.296 147.625 0.00334389 0.00576375 2024-03-24T12:37:01.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:01.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:01.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:02.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:02.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:02.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:02.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:02.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:02.204 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:02.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:02.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3400227294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:37:02.258 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[41500]: 4.10 scrub starts 2024-03-24T12:37:02.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3400227294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:37:02.258 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[29919]: 4.10 scrub starts 2024-03-24T12:37:02.307 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:02.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:02.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:02.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:02 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3400227294' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:37:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:02 smithi190 ceph-mon[33748]: 4.10 scrub starts 2024-03-24T12:37:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:02 smithi190 ceph-mon[33748]: 4.10 scrub ok 2024-03-24T12:37:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:02 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:37:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:02 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:37:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:02 smithi190 ceph-mon[33748]: 5.15 scrub starts 2024-03-24T12:37:02.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:02.477 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:02.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:02.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:02.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:02.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:02.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:02.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:02.623 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[41500]: 4.10 scrub ok 2024-03-24T12:37:02.624 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:37:02.624 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:37:02.624 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[41500]: 5.15 scrub starts 2024-03-24T12:37:02.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[29919]: 4.10 scrub ok 2024-03-24T12:37:02.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:37:02.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:37:02.624 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:02 smithi029 ceph-mon[29919]: 5.15 scrub starts 2024-03-24T12:37:02.672 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:37:02.672 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:37:02.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:02.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:02.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:02.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:02.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:02.868 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:02.869 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:02.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 9 16 24164 24148 167.672 122.688 0.0056934 0.00595773 2024-03-24T12:37:02.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:03.000 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:03.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:03.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:03.131 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:03.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:03.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:03.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:03.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:03.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:03.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:03.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:03.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:03.552 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:03.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:03.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:03 smithi190 ceph-mon[33748]: 5.15 scrub ok 2024-03-24T12:37:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:03 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:37:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:03 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:37:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:03 smithi190 ceph-mon[33748]: osdmap e219: 8 total, 7 up, 8 in 2024-03-24T12:37:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:03 smithi190 ceph-mon[33748]: 6.9 scrub starts 2024-03-24T12:37:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:03 smithi190 ceph-mon[33748]: pgmap v447: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 146 MiB/s wr, 2.33k op/s; 10404/54387 objects degraded (19.130%) 2024-03-24T12:37:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:03 smithi190 ceph-mon[33748]: 6.9 scrub ok 2024-03-24T12:37:03.656 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[29919]: 5.15 scrub ok 2024-03-24T12:37:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:37:03.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:37:03.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[29919]: osdmap e219: 8 total, 7 up, 8 in 2024-03-24T12:37:03.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[29919]: 6.9 scrub starts 2024-03-24T12:37:03.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[29919]: pgmap v447: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 146 MiB/s wr, 2.33k op/s; 10404/54387 objects degraded (19.130%) 2024-03-24T12:37:03.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:03.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:03.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[29919]: 6.9 scrub ok 2024-03-24T12:37:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[41500]: 5.15 scrub ok 2024-03-24T12:37:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:37:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:37:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[41500]: osdmap e219: 8 total, 7 up, 8 in 2024-03-24T12:37:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[41500]: 6.9 scrub starts 2024-03-24T12:37:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[41500]: pgmap v447: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.1 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 146 MiB/s wr, 2.33k op/s; 10404/54387 objects degraded (19.130%) 2024-03-24T12:37:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:03.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:03 smithi029 ceph-mon[41500]: 6.9 scrub ok 2024-03-24T12:37:03.695 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:03.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:03.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:03.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:03.825 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:03.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:03.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:03.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 10 16 26613 26597 166.21 153.062 0.00612266 0.00601079 2024-03-24T12:37:03.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:03.964 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:03.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:03.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:04.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:04.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:04.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:04.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:04.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:04.243 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:04.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:04.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:04.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:04.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:04.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:04.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:04.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:04.506 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:04.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:04.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:04.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:04.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:04 smithi190 ceph-mon[33748]: 3.1b scrub starts 2024-03-24T12:37:04.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:04 smithi190 ceph-mon[33748]: 3.1b scrub ok 2024-03-24T12:37:04.647 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:04.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:04.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:04.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:04 smithi029 ceph-mon[29919]: 3.1b scrub starts 2024-03-24T12:37:04.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:04 smithi029 ceph-mon[29919]: 3.1b scrub ok 2024-03-24T12:37:04.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:04 smithi029 ceph-mon[41500]: 3.1b scrub starts 2024-03-24T12:37:04.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:04 smithi029 ceph-mon[41500]: 3.1b scrub ok 2024-03-24T12:37:04.674 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:37:04.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:04.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:04.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:04.917 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 11 16 28813 28797 163.591 137.5 0.0148038 0.00610604 2024-03-24T12:37:04.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:04.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:04.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:05.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:05.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:05.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:05.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:05.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:05.187 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:05.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:05.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:05.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:05.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:05.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:05.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:05.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:05.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:05.598 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:05.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:05 smithi190 ceph-mon[33748]: 5.13 scrub starts 2024-03-24T12:37:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:05 smithi190 ceph-mon[33748]: 5.13 scrub ok 2024-03-24T12:37:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:05 smithi190 ceph-mon[33748]: pgmap v448: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 180 MiB/s wr, 2.87k op/s; 13400/74499 objects degraded (17.987%) 2024-03-24T12:37:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:05 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 10404/54387 objects degraded (19.130%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:05 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:37:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:05 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:37:05.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[41500]: 5.13 scrub starts 2024-03-24T12:37:05.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[41500]: 5.13 scrub ok 2024-03-24T12:37:05.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[41500]: pgmap v448: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 180 MiB/s wr, 2.87k op/s; 13400/74499 objects degraded (17.987%) 2024-03-24T12:37:05.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 10404/54387 objects degraded (19.130%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:05.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:37:05.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:37:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[29919]: 5.13 scrub starts 2024-03-24T12:37:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[29919]: 5.13 scrub ok 2024-03-24T12:37:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[29919]: pgmap v448: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.5 GiB data, 6.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 180 MiB/s wr, 2.87k op/s; 13400/74499 objects degraded (17.987%) 2024-03-24T12:37:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 10404/54387 objects degraded (19.130%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:37:05.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:05 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:37:05.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:05.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:05.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:05.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:05.862 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:05.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:05.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:05.917 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 12 15 30656 30641 159.562 115.25 0.00255535 0.0062617 2024-03-24T12:37:05.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:06.004 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:06.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:06.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:06.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:06.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:06.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:06.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:06.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:06.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:06.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:06.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:06.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:06.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:06.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:06.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:06.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:06.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:06.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:06 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:37:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:06 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:37:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:06 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:37:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:06 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:37:06.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:06.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:06.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:06.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:06 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:37:06.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:06 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:37:06.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:06 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:37:06.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:06 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:37:06.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:06 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:37:06.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:06 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:37:06.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:06 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:37:06.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:06 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:37:06.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:06.794 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:06.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:06.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:06.917 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 13 16 32299 32283 155.182 102.625 0.0023331 0.00640675 2024-03-24T12:37:06.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:06.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:06.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:07.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:07.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:07.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:07.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:07.171 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:07.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:07.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:07.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:07.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:07.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:07.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:07.431 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:07.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:07.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:07.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:07.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:07.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:07.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:07.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:07 smithi029 ceph-mon[41500]: pgmap v449: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 167 MiB/s wr, 2.67k op/s; 15461/80682 objects degraded (19.163%) 2024-03-24T12:37:07.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:07 smithi029 ceph-mon[29919]: pgmap v449: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 167 MiB/s wr, 2.67k op/s; 15461/80682 objects degraded (19.163%) 2024-03-24T12:37:07.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:07 smithi190 ceph-mon[33748]: pgmap v449: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.6 GiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 167 MiB/s wr, 2.67k op/s; 15461/80682 objects degraded (19.163%) 2024-03-24T12:37:07.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:07.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:07.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:07.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:07.827 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:07.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:07.917 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 14 15 33527 33512 149.584 76.8125 0.00889777 0.0066797 2024-03-24T12:37:07.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:37:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:37:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:37:07.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:07.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:07.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:08.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:08.106 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:08.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:08.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:08.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:08.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:08.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:08.311 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:37:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:37:08.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:08.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:08.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:08.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:08.501 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:08.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:08.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:08.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:08.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:08 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:37:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:08 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:37:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:08 smithi190 ceph-mon[33748]: 5.a scrub starts 2024-03-24T12:37:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:08 smithi190 ceph-mon[33748]: 5.a scrub ok 2024-03-24T12:37:08.644 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:08.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:08 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:37:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:08 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:37:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:08 smithi029 ceph-mon[41500]: 5.a scrub starts 2024-03-24T12:37:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:08 smithi029 ceph-mon[41500]: 5.a scrub ok 2024-03-24T12:37:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:08 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:37:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:08 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:37:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:08 smithi029 ceph-mon[29919]: 5.a scrub starts 2024-03-24T12:37:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:08 smithi029 ceph-mon[29919]: 5.a scrub ok 2024-03-24T12:37:08.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:08.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:08.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:08.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:08.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:08.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:08.917 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 15 16 35163 35147 146.424 102.188 0.00411362 0.00682461 2024-03-24T12:37:09.010 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:09.033 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:09.035 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:09.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:09.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:09.160 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:09.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:09.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:09.288 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:09.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:09.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:09.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:09.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:09.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:09.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:09.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:09.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:09 smithi190 ceph-mon[33748]: 3.18 scrub starts 2024-03-24T12:37:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:09 smithi190 ceph-mon[33748]: 3.18 scrub ok 2024-03-24T12:37:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:09 smithi190 ceph-mon[33748]: pgmap v450: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s; 17221/89547 objects degraded (19.231%) 2024-03-24T12:37:09.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:09.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:09 smithi029 ceph-mon[29919]: 3.18 scrub starts 2024-03-24T12:37:09.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:09 smithi029 ceph-mon[29919]: 3.18 scrub ok 2024-03-24T12:37:09.671 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:09 smithi029 ceph-mon[29919]: pgmap v450: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s; 17221/89547 objects degraded (19.231%) 2024-03-24T12:37:09.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:09 smithi029 ceph-mon[41500]: 3.18 scrub starts 2024-03-24T12:37:09.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:09 smithi029 ceph-mon[41500]: 3.18 scrub ok 2024-03-24T12:37:09.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:09 smithi029 ceph-mon[41500]: pgmap v450: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 1.8 GiB data, 7.6 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s; 17221/89547 objects degraded (19.231%) 2024-03-24T12:37:09.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:09.722 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:09.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:09.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:09.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:09.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:09.917 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 16 16 36336 36320 141.854 73.3125 0.0122214 0.00704387 2024-03-24T12:37:09.963 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:10.012 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:10.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:10.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:10.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:10.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:10.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:10.267 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:10.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:10.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:10.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:10.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:10.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:10.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:10.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:10.522 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:10.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:10.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:10.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:10 smithi190 ceph-mon[33748]: 4.19 scrub starts 2024-03-24T12:37:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:10 smithi190 ceph-mon[33748]: 4.19 scrub ok 2024-03-24T12:37:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:10 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 17221/89547 objects degraded (19.231%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:10.649 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:10.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:10.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:10.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:10 smithi029 ceph-mon[29919]: 4.19 scrub starts 2024-03-24T12:37:10.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:10 smithi029 ceph-mon[29919]: 4.19 scrub ok 2024-03-24T12:37:10.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:10 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 17221/89547 objects degraded (19.231%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:10 smithi029 ceph-mon[41500]: 4.19 scrub starts 2024-03-24T12:37:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:10 smithi029 ceph-mon[41500]: 4.19 scrub ok 2024-03-24T12:37:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:10 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 17221/89547 objects degraded (19.231%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:10.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:10.776 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:10.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:10.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:10.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:10.907 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:10.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:10.917 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 17 16 37298 37282 137.046 60.125 0.00206758 0.0072835 2024-03-24T12:37:11.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:11.032 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:11.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:11.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:11.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:11.157 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:11.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:11.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:11.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:11.299 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:11.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:11.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:11.440 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:11.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:11.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:11.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:11.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:11.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:11 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:37:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:11 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:37:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:11 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:37:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:11 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:37:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:11 smithi190 ceph-mon[33748]: pgmap v451: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.1 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 133 MiB/s wr, 2.13k op/s; 18727/101598 objects degraded (18.432%) 2024-03-24T12:37:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:11 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:37:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:11 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:37:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:11 smithi190 ceph-mon[33748]: 5.0 scrub starts 2024-03-24T12:37:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:11 smithi190 ceph-mon[33748]: 5.0 scrub ok 2024-03-24T12:37:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:11 smithi190 ceph-mon[33748]: osdmap e220: 8 total, 7 up, 8 in 2024-03-24T12:37:11.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[29919]: pgmap v451: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.1 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 133 MiB/s wr, 2.13k op/s; 18727/101598 objects degraded (18.432%) 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[29919]: 5.0 scrub starts 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[29919]: 5.0 scrub ok 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[29919]: osdmap e220: 8 total, 7 up, 8 in 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:37:11.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:37:11.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:37:11.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:37:11.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[41500]: pgmap v451: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.1 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 133 MiB/s wr, 2.13k op/s; 18727/101598 objects degraded (18.432%) 2024-03-24T12:37:11.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:37:11.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:37:11.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[41500]: 5.0 scrub starts 2024-03-24T12:37:11.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[41500]: 5.0 scrub ok 2024-03-24T12:37:11.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:11 smithi029 ceph-mon[41500]: osdmap e220: 8 total, 7 up, 8 in 2024-03-24T12:37:11.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:11.707 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:11.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:11.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:11.834 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:11.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:11.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 18 16 38395 38379 133.241 68.5625 0.0030252 0.00748874 2024-03-24T12:37:11.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:11.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:11.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:12.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:12.085 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:12.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:12.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:12.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:12.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:12.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:12.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:12.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:12.356 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:12.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:12 smithi029 ceph-mon[29919]: 5.1c scrub starts 2024-03-24T12:37:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:12 smithi029 ceph-mon[29919]: 5.1c scrub ok 2024-03-24T12:37:12.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:12 smithi029 ceph-mon[41500]: 5.1c scrub starts 2024-03-24T12:37:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:12 smithi029 ceph-mon[41500]: 5.1c scrub ok 2024-03-24T12:37:12.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:12.497 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:12.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:12.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:12.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:12.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:12.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:12.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:12.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:12 smithi190 ceph-mon[33748]: 5.1c scrub starts 2024-03-24T12:37:12.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:12 smithi190 ceph-mon[33748]: 5.1c scrub ok 2024-03-24T12:37:12.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:12.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:12.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:12.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:12.895 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:12.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:12.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:12.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 19 16 39984 39968 131.455 99.3125 0.0100607 0.00760193 2024-03-24T12:37:12.996 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:13.036 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:13.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:13.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:13.177 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:13.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:13.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:13.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:13.316 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:13.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:13.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:13.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:13 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:37:13.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:13 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:37:13.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:13.441 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:13.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:13.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:13.542 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:13.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:13.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:13 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:37:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:13 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:37:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:13 smithi190 ceph-mon[33748]: pgmap v453: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.85k op/s; 20731/109956 objects degraded (18.854%) 2024-03-24T12:37:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:13 smithi190 ceph-mon[33748]: 3.1e scrub starts 2024-03-24T12:37:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:13 smithi190 ceph-mon[33748]: 3.1e scrub ok 2024-03-24T12:37:13.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:13.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:13.698 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:13.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:13.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:13.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:13.823 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:13.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:13.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:13 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:37:13.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:13 smithi029 ceph-mon[41500]: pgmap v453: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.85k op/s; 20731/109956 objects degraded (18.854%) 2024-03-24T12:37:13.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:13 smithi029 ceph-mon[41500]: 3.1e scrub starts 2024-03-24T12:37:13.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:13 smithi029 ceph-mon[41500]: 3.1e scrub ok 2024-03-24T12:37:13.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:13 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:37:13.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:13 smithi029 ceph-mon[29919]: pgmap v453: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.2 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.85k op/s; 20731/109956 objects degraded (18.854%) 2024-03-24T12:37:13.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:13 smithi029 ceph-mon[29919]: 3.1e scrub starts 2024-03-24T12:37:13.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:13 smithi029 ceph-mon[29919]: 3.1e scrub ok 2024-03-24T12:37:13.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:37:13.922838+0000 min lat: 0.00103404 max lat: 0.417648 avg lat: 0.00766059 2024-03-24T12:37:13.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:37:13.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 20 16 40973 40957 127.973 61.8125 0.00195925 0.00766059 2024-03-24T12:37:13.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:13.950 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:13.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:14.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:14.078 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:14.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:14.180 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:14.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:14.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:14.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:14.305 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:14.335 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:14.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:14.437 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:14.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:14.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:14.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:14.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:14.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:14.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:14.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:14.700 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:14.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:14.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:14.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:14.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:14.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:14.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:14.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 21 16 42302 42286 125.834 83.0625 0.00391451 0.00791775 2024-03-24T12:37:14.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:14.991 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:14.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:15.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:15.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:15.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:15.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:15.259 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:15.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:15.360 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:15.390 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:15.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:15.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:15.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:15.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:15.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:15.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:15.656 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:15.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:15.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:15.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:15.781 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:15.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:15.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:15.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[29919]: 6.15 scrub starts 2024-03-24T12:37:15.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[29919]: 6.15 scrub ok 2024-03-24T12:37:15.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:37:15.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:37:15.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[29919]: pgmap v454: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.53k op/s; 22321/120525 objects degraded (18.520%) 2024-03-24T12:37:15.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 22321/120525 objects degraded (18.520%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:15.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[41500]: 6.15 scrub starts 2024-03-24T12:37:15.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[41500]: 6.15 scrub ok 2024-03-24T12:37:15.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:37:15.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:37:15.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[41500]: pgmap v454: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.53k op/s; 22321/120525 objects degraded (18.520%) 2024-03-24T12:37:15.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:15 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 22321/120525 objects degraded (18.520%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:15.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 22 16 43526 43510 123.59 76.5 0.00993083 0.00808572 2024-03-24T12:37:15.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:15.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:16.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:16.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:16.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:15 smithi190 ceph-mon[33748]: 6.15 scrub starts 2024-03-24T12:37:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:15 smithi190 ceph-mon[33748]: 6.15 scrub ok 2024-03-24T12:37:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:15 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:37:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:15 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:37:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:15 smithi190 ceph-mon[33748]: pgmap v454: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.4 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 96 MiB/s wr, 1.53k op/s; 22321/120525 objects degraded (18.520%) 2024-03-24T12:37:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:15 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 22321/120525 objects degraded (18.520%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:16.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:16.183 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:16.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:16.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:16.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:16.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:16.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:16.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:16.450 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:16.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:16.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:16.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:16.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:16.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:16.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:16.843 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:16.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:16.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:16 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:37:16.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:16 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:37:16.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:16 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:37:16.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:16 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:37:16.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:16 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:37:16.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:16 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:37:16.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:16 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:37:16.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:16 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:37:16.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 23 15 44881 44866 121.902 84.75 0.00394827 0.00819806 2024-03-24T12:37:16.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:16.973 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:16.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:16.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:17.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:17.115 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:17.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:17.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:17.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:16 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:37:17.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:16 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:37:17.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:16 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:37:17.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:16 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:37:17.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:17.256 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:17.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:17.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:17.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:17.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:17.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:17.497 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:17.541 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:17.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:17.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:17.666 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:17.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:17.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:17.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:17.791 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:17.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:17.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:17.916 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:17.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:17.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:17.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:17 smithi029 ceph-mon[29919]: pgmap v455: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s; 23463/123951 objects degraded (18.929%) 2024-03-24T12:37:17.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:17 smithi029 ceph-mon[41500]: pgmap v455: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s; 23463/123951 objects degraded (18.929%) 2024-03-24T12:37:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:17.919 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:37:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:37:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:37:17.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 24 16 46063 46047 119.898 73.8125 0.00244244 0.00826292 2024-03-24T12:37:18.003 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:17 smithi190 ceph-mon[33748]: pgmap v455: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.5 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s; 23463/123951 objects degraded (18.929%) 2024-03-24T12:37:18.003 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:18.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:18.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:18.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:18.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:18.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:18.185 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:18.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:18.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:18.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:18.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:18.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:18.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:37:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:37:18.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:18.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:18.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:18.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:18.600 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:18.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:18.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:18.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:18.731 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:18.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:18.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:18.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:18.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:18.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:18.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:18 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:37:18.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:18 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:37:18.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:18.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:18 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:37:18.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:18 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:37:18.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:18.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 25 16 47727 47711 119.262 104 0.00167651 0.00834545 2024-03-24T12:37:18.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:18.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:18.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:18.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:19.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:19.129 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:19.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:19.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:19.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:18 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:37:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:18 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:37:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:19.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:19.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:19.271 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:19.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:19.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:19.410 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:19.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:19.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:19.535 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:19.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:19.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:19.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:19.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:19.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:19.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:19.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:19.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:19.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:19.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 26 16 48939 48923 117.577 75.75 0.00488402 0.00850024 2024-03-24T12:37:19.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:19.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:20.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:20.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:20.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:20.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:20.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:19 smithi190 ceph-mon[33748]: pgmap v456: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.6 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s; 24336/129066 objects degraded (18.855%) 2024-03-24T12:37:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:19 smithi029 ceph-mon[41500]: pgmap v456: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.6 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s; 24336/129066 objects degraded (18.855%) 2024-03-24T12:37:20.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:19 smithi029 ceph-mon[29919]: pgmap v456: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.6 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.32k op/s; 24336/129066 objects degraded (18.855%) 2024-03-24T12:37:20.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:20.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:20.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:20.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:20.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:20.339 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:20.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:20.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:20.470 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:20.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:20.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:20.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:20.613 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:20.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:20.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:20.752 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:20.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:20.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:20.891 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:20.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:20.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:20.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 27 15 50670 50655 117.231 108.25 0.00977256 0.0085254 2024-03-24T12:37:20.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:21.021 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:21.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:21.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:21.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:21.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:20 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:37:21.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:20 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:37:21.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:20 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:37:21.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:20 smithi190 ceph-mon[33748]: 4.11 scrub ok 2024-03-24T12:37:21.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:20 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 24336/129066 objects degraded (18.855%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:21.147 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:21.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:20 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:37:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:20 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:37:21.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:20 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:37:21.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:20 smithi029 ceph-mon[41500]: 4.11 scrub ok 2024-03-24T12:37:21.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:20 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 24336/129066 objects degraded (18.855%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:20 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:37:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:20 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:37:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:20 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:37:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:20 smithi029 ceph-mon[29919]: 4.11 scrub ok 2024-03-24T12:37:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:20 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 24336/129066 objects degraded (18.855%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:21.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:21.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:21.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:21.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:21.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:21.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:21.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:21.545 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:21.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:21.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:21.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:21.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:21.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:21.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:21.815 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:21.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:21.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:21.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 28 16 52009 51993 116.031 83.625 0.00251269 0.00859868 2024-03-24T12:37:21.939 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:21.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:22.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:22.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:22.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:22.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:21 smithi190 ceph-mon[33748]: pgmap v457: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.29k op/s; 25957/140439 objects degraded (18.483%) 2024-03-24T12:37:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:21 smithi029 ceph-mon[41500]: pgmap v457: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.29k op/s; 25957/140439 objects degraded (18.483%) 2024-03-24T12:37:22.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:21 smithi029 ceph-mon[29919]: pgmap v457: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 2.8 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 81 MiB/s wr, 1.29k op/s; 25957/140439 objects degraded (18.483%) 2024-03-24T12:37:22.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:22.202 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:22.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:22.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:22.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:22.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:22.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:22.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:22.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:22.472 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:22.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:22.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:22.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:22.616 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:22.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:22.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:22.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:22.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:22.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:22.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:22.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:22.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:22.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 29 16 53210 53194 114.618 75.0625 0.127127 0.00869023 2024-03-24T12:37:22.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:23.022 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:23.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:23.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:23.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:22 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:37:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:22 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:37:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:22 smithi190 ceph-mon[33748]: 5.1e scrub starts 2024-03-24T12:37:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:22 smithi190 ceph-mon[33748]: 5.1e scrub ok 2024-03-24T12:37:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:22 smithi190 ceph-mon[33748]: 6.13 scrub starts 2024-03-24T12:37:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:22 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:37:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:22 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:37:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:22 smithi190 ceph-mon[33748]: 6.13 scrub ok 2024-03-24T12:37:23.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:23.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:23.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:23.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:37:23.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:37:23.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[29919]: 5.1e scrub starts 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[29919]: 5.1e scrub ok 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[29919]: 6.13 scrub starts 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[29919]: 6.13 scrub ok 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[41500]: 5.1e scrub starts 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[41500]: 5.1e scrub ok 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[41500]: 6.13 scrub starts 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:37:23.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:37:23.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:22 smithi029 ceph-mon[41500]: 6.13 scrub ok 2024-03-24T12:37:23.254 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:23.278 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:23.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:23.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:23.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:23.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:23.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:23.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:23.537 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:23.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:23.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:23.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:23.706 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:23.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:23.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:23.841 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:23.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:23.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 30 15 54632 54617 113.762 88.9375 0.00700332 0.00878509 2024-03-24T12:37:23.942 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:23.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:24.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:24.126 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:24.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:24.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:24.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:23 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:37:24.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:23 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:37:24.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:23 smithi190 ceph-mon[33748]: pgmap v458: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 28149/149736 objects degraded (18.799%) 2024-03-24T12:37:24.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:23 smithi190 ceph-mon[33748]: 5.18 scrub starts 2024-03-24T12:37:24.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:23 smithi190 ceph-mon[33748]: 5.18 scrub ok 2024-03-24T12:37:24.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:23 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:37:24.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:23 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:37:24.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:23 smithi029 ceph-mon[29919]: pgmap v458: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 28149/149736 objects degraded (18.799%) 2024-03-24T12:37:24.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:23 smithi029 ceph-mon[29919]: 5.18 scrub starts 2024-03-24T12:37:24.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:23 smithi029 ceph-mon[29919]: 5.18 scrub ok 2024-03-24T12:37:24.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:23 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:37:24.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:23 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:37:24.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:23 smithi029 ceph-mon[41500]: pgmap v458: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.0 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.3 KiB/s rd, 80 MiB/s wr, 1.28k op/s; 28149/149736 objects degraded (18.799%) 2024-03-24T12:37:24.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:23 smithi029 ceph-mon[41500]: 5.18 scrub starts 2024-03-24T12:37:24.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:23 smithi029 ceph-mon[41500]: 5.18 scrub ok 2024-03-24T12:37:24.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:24.257 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:24.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:24.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:24.359 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:24.382 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:24.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:24.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:24.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:24.524 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:24.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:24.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:24.666 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:24.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:24.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:24.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:24.798 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:24.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:24.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:24.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:24.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 31 16 56049 56033 112.947 88.5 0.00305216 0.00884398 2024-03-24T12:37:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:24.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:24.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:25.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:25.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:25.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:25.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:24 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:37:25.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:24 smithi190 ceph-mon[33748]: 4.4 scrub ok 2024-03-24T12:37:25.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:24 smithi190 ceph-mon[33748]: 4.b scrub starts 2024-03-24T12:37:25.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:24 smithi190 ceph-mon[33748]: 4.b scrub ok 2024-03-24T12:37:25.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:24 smithi190 ceph-mon[33748]: 6.6 scrub starts 2024-03-24T12:37:25.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:24 smithi190 ceph-mon[33748]: 6.6 scrub ok 2024-03-24T12:37:25.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:24 smithi190 ceph-mon[33748]: 3.9 scrub starts 2024-03-24T12:37:25.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:24 smithi190 ceph-mon[33748]: 3.9 scrub ok 2024-03-24T12:37:25.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:25.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:37:25.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[29919]: 4.4 scrub ok 2024-03-24T12:37:25.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[29919]: 4.b scrub starts 2024-03-24T12:37:25.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[29919]: 4.b scrub ok 2024-03-24T12:37:25.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[29919]: 6.6 scrub starts 2024-03-24T12:37:25.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[29919]: 6.6 scrub ok 2024-03-24T12:37:25.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[29919]: 3.9 scrub starts 2024-03-24T12:37:25.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[29919]: 3.9 scrub ok 2024-03-24T12:37:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:37:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[41500]: 4.4 scrub ok 2024-03-24T12:37:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[41500]: 4.b scrub starts 2024-03-24T12:37:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[41500]: 4.b scrub ok 2024-03-24T12:37:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[41500]: 6.6 scrub starts 2024-03-24T12:37:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[41500]: 6.6 scrub ok 2024-03-24T12:37:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[41500]: 3.9 scrub starts 2024-03-24T12:37:25.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:24 smithi029 ceph-mon[41500]: 3.9 scrub ok 2024-03-24T12:37:25.199 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:25.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:25.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:25.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:25.329 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:25.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:25.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:25.459 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:25.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:25.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:25.583 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:25.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:25.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:25.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:25.707 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:25.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:25.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:25.848 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:25.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:25.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 32 16 57170 57154 111.607 70.0625 0.00714015 0.0089554 2024-03-24T12:37:25.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:25.971 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:37:25.971 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:37:25.971 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[41500]: pgmap v459: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.43k op/s; 29928/161346 objects degraded (18.549%) 2024-03-24T12:37:25.972 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[41500]: 6.1b scrub starts 2024-03-24T12:37:25.972 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[41500]: 6.1b scrub ok 2024-03-24T12:37:25.972 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 29928/161346 objects degraded (18.549%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:25.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:37:25.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:37:25.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[29919]: pgmap v459: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.43k op/s; 29928/161346 objects degraded (18.549%) 2024-03-24T12:37:25.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[29919]: 6.1b scrub starts 2024-03-24T12:37:25.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[29919]: 6.1b scrub ok 2024-03-24T12:37:25.972 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:25 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 29928/161346 objects degraded (18.549%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:25.986 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:25.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:26.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:26.129 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:26.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:26.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:26.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:25 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:37:26.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:25 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:37:26.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:25 smithi190 ceph-mon[33748]: pgmap v459: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.43k op/s; 29928/161346 objects degraded (18.549%) 2024-03-24T12:37:26.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:25 smithi190 ceph-mon[33748]: 6.1b scrub starts 2024-03-24T12:37:26.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:25 smithi190 ceph-mon[33748]: 6.1b scrub ok 2024-03-24T12:37:26.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:25 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 29928/161346 objects degraded (18.549%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:26.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:26.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:26.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:26.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:26.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:26.490 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:26.515 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:26.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:26.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:26.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:26.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:26.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:26.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:26.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:26.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:26.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:26.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:26.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 33 16 58157 58141 110.094 61.6875 0.00347577 0.00907693 2024-03-24T12:37:27.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:27.042 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:27.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:26 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:37:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:26 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:37:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:26 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:37:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:26 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:37:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:26 smithi190 ceph-mon[33748]: 3.1 scrub starts 2024-03-24T12:37:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:26 smithi190 ceph-mon[33748]: 3.1 scrub ok 2024-03-24T12:37:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:26 smithi190 ceph-mon[33748]: 4.17 scrub starts 2024-03-24T12:37:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:26 smithi190 ceph-mon[33748]: 4.17 scrub ok 2024-03-24T12:37:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:26 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:37:27.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:37:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:37:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:37:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[29919]: 3.1 scrub starts 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[29919]: 3.1 scrub ok 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[29919]: 4.17 scrub starts 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[29919]: 4.17 scrub ok 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[41500]: 3.1 scrub starts 2024-03-24T12:37:27.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[41500]: 3.1 scrub ok 2024-03-24T12:37:27.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[41500]: 4.17 scrub starts 2024-03-24T12:37:27.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[41500]: 4.17 scrub ok 2024-03-24T12:37:27.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:26 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:37:27.655 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:27.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:27.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:27.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:27.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:27.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:27.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:27.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:27.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:37:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:37:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[41500]: pgmap v460: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s; 31136/164970 objects degraded (18.874%) 2024-03-24T12:37:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[41500]: 3.1a scrub starts 2024-03-24T12:37:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[41500]: 3.1a scrub ok 2024-03-24T12:37:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi190 to 3248M 2024-03-24T12:37:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:27.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[29919]: pgmap v460: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s; 31136/164970 objects degraded (18.874%) 2024-03-24T12:37:27.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[29919]: 3.1a scrub starts 2024-03-24T12:37:27.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[29919]: 3.1a scrub ok 2024-03-24T12:37:27.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:27.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:27.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:27.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:27.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi190 to 3248M 2024-03-24T12:37:27.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:27.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 34 16 59191 59175 108.757 64.625 0.00679106 0.00918909 2024-03-24T12:37:27.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:27.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:27.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:27 smithi190 ceph-mon[33748]: pgmap v460: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.3 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 77 MiB/s wr, 1.23k op/s; 31136/164970 objects degraded (18.874%) 2024-03-24T12:37:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:27 smithi190 ceph-mon[33748]: 3.1a scrub starts 2024-03-24T12:37:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:27 smithi190 ceph-mon[33748]: 3.1a scrub ok 2024-03-24T12:37:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:27 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi190 to 3248M 2024-03-24T12:37:28.006 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:28.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:28.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:28.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:28.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:28.191 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:28.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:28.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:28.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:28.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:28.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:28.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:28.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:37:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:37:28.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:28.442 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:28.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:28.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:28.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:28.586 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:28.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:28.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:28.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:28.711 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:28.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:28.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:28.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:28.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:28.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:28.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:28.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 35 16 59833 59817 106.796 40.125 0.0494643 0.00935294 2024-03-24T12:37:28.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:28.968 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:28.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:28.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:29.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:29.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:29.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:28 smithi190 ceph-mon[33748]: 4.14 scrub starts 2024-03-24T12:37:29.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:28 smithi190 ceph-mon[33748]: 4.14 scrub ok 2024-03-24T12:37:29.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:28 smithi029 ceph-mon[29919]: 4.14 scrub starts 2024-03-24T12:37:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:28 smithi029 ceph-mon[29919]: 4.14 scrub ok 2024-03-24T12:37:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:29.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:28 smithi029 ceph-mon[41500]: 4.14 scrub starts 2024-03-24T12:37:29.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:28 smithi029 ceph-mon[41500]: 4.14 scrub ok 2024-03-24T12:37:29.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:29.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:29.237 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:29.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:29.339 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:29.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:29.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:29.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:29.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:29.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:29.613 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:29.636 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:29.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:29.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:29.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:29.761 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:29.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:29.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:29.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:29.902 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:29.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:29.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 36 16 61059 61043 105.958 76.625 0.00391027 0.00943327 2024-03-24T12:37:30.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:30.028 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:30.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:30.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:30.157 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:30.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:30.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:30.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:30.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:30.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:30.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:30.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:29 smithi190 ceph-mon[33748]: pgmap v461: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 31897/169365 objects degraded (18.833%) 2024-03-24T12:37:30.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:29 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:37:30.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:29 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:37:30.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:30.415 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:30.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:30.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:29 smithi029 ceph-mon[29919]: pgmap v461: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 31897/169365 objects degraded (18.833%) 2024-03-24T12:37:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:29 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:37:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:29 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:37:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:29 smithi029 ceph-mon[41500]: pgmap v461: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.4 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 31897/169365 objects degraded (18.833%) 2024-03-24T12:37:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:29 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:37:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:29 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:37:30.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:30.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:30.541 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:30.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:30.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:30.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:30.666 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:30.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:30.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:30.806 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:30.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:30.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:30.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:30.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 37 16 61804 61788 104.352 46.5625 0.00785343 0.00957552 2024-03-24T12:37:30.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:30.934 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:30.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:31.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:31.058 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:31.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:31.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:31.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:31.198 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:31.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:31.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:31.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:31.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:31.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:31.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:30 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:37:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:30 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:37:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:30 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 31897/169365 objects degraded (18.833%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:31.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:30 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:37:31.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:37:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 31897/169365 objects degraded (18.833%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:31.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:30 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:37:31.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:37:31.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 31897/169365 objects degraded (18.833%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:31.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:31.453 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:31.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:31.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:31.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:31.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:31.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:31.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:31.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:31.725 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:31.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:31.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:31.855 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:31.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:31.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:31.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 38 16 62636 62620 102.975 52 0.00860978 0.00970587 2024-03-24T12:37:31.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:31.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:31.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:32.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:32.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:32.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:32.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:32.250 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:32.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:32.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:32.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:32.375 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:32.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:32.376 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:31 smithi190 ceph-mon[33748]: 5.1a scrub starts 2024-03-24T12:37:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:31 smithi190 ceph-mon[33748]: 5.1a scrub ok 2024-03-24T12:37:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:31 smithi190 ceph-mon[33748]: pgmap v462: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 86 MiB/s wr, 1.37k op/s; 33284/178551 objects degraded (18.641%) 2024-03-24T12:37:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:31 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:37:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:31 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:37:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:31 smithi190 ceph-mon[33748]: 3.7 scrub starts 2024-03-24T12:37:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:31 smithi190 ceph-mon[33748]: 3.7 scrub ok 2024-03-24T12:37:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:31 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:37:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:31 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:37:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: 5.1a scrub starts 2024-03-24T12:37:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: 5.1a scrub ok 2024-03-24T12:37:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: pgmap v462: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 86 MiB/s wr, 1.37k op/s; 33284/178551 objects degraded (18.641%) 2024-03-24T12:37:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:37:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:37:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: 3.7 scrub starts 2024-03-24T12:37:32.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: 3.7 scrub ok 2024-03-24T12:37:32.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:37:32.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:37:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: 5.1a scrub starts 2024-03-24T12:37:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: 5.1a scrub ok 2024-03-24T12:37:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: pgmap v462: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 86 MiB/s wr, 1.37k op/s; 33284/178551 objects degraded (18.641%) 2024-03-24T12:37:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:37:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:37:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: 3.7 scrub starts 2024-03-24T12:37:32.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: 3.7 scrub ok 2024-03-24T12:37:32.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:37:32.421 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:31 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:37:32.477 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:32.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:32.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:32.647 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:32.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:32.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:32.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:32.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:32.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:32.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:32.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:32.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:32.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 39 16 63828 63812 102.245 74.5 0.0216156 0.00977556 2024-03-24T12:37:33.010 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:33.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:33.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:33.101 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:37:33.102 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[41500]: 3.13 scrub ok 2024-03-24T12:37:33.102 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[41500]: 3.12 scrub starts 2024-03-24T12:37:33.102 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[41500]: 3.12 scrub ok 2024-03-24T12:37:33.102 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:37:33.102 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:37:33.102 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:37:33.102 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:37:33.102 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[41500]: osdmap e221: 8 total, 7 up, 8 in 2024-03-24T12:37:33.102 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:33.102 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:37:33.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[29919]: 3.13 scrub ok 2024-03-24T12:37:33.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[29919]: 3.12 scrub starts 2024-03-24T12:37:33.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[29919]: 3.12 scrub ok 2024-03-24T12:37:33.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:37:33.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:37:33.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:37:33.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:37:33.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[29919]: osdmap e221: 8 total, 7 up, 8 in 2024-03-24T12:37:33.103 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:32 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:33.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:32 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:37:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:32 smithi190 ceph-mon[33748]: 3.13 scrub ok 2024-03-24T12:37:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:32 smithi190 ceph-mon[33748]: 3.12 scrub starts 2024-03-24T12:37:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:32 smithi190 ceph-mon[33748]: 3.12 scrub ok 2024-03-24T12:37:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:32 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:37:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:32 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:37:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:32 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:37:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:32 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:37:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:32 smithi190 ceph-mon[33748]: osdmap e221: 8 total, 7 up, 8 in 2024-03-24T12:37:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:32 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:33.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:37:33.927100+0000 min lat: 0.00103404 max lat: 0.488958 avg lat: 0.00983498 2024-03-24T12:37:33.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:37:33.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 40 16 65011 64995 101.537 73.9375 0.0069998 0.00983498 2024-03-24T12:37:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:34 smithi190 ceph-mon[33748]: pgmap v464: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s; 34692/184500 objects degraded (18.803%) 2024-03-24T12:37:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:34 smithi190 ceph-mon[33748]: 5.14 scrub starts 2024-03-24T12:37:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:34 smithi190 ceph-mon[33748]: 5.14 scrub ok 2024-03-24T12:37:34.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:34 smithi029 ceph-mon[41500]: pgmap v464: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s; 34692/184500 objects degraded (18.803%) 2024-03-24T12:37:34.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:34 smithi029 ceph-mon[41500]: 5.14 scrub starts 2024-03-24T12:37:34.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:34 smithi029 ceph-mon[41500]: 5.14 scrub ok 2024-03-24T12:37:34.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:34 smithi029 ceph-mon[29919]: pgmap v464: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 72 MiB/s wr, 1.16k op/s; 34692/184500 objects degraded (18.803%) 2024-03-24T12:37:34.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:34 smithi029 ceph-mon[29919]: 5.14 scrub starts 2024-03-24T12:37:34.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:34 smithi029 ceph-mon[29919]: 5.14 scrub ok 2024-03-24T12:37:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:34.662 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:34.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:34.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:34.790 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:34.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:34.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:34.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 41 16 66068 66052 100.671 66.0625 0.0060598 0.00992817 2024-03-24T12:37:34.949 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:34.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:34.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:35.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:35.101 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:35.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:35.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: osdmap e222: 8 total, 7 up, 8 in 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: 6.3 scrub starts 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: 6.3 scrub ok 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 34692/184500 objects degraded (18.803%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: osdmap e222: 8 total, 7 up, 8 in 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: 6.3 scrub starts 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: 6.3 scrub ok 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:37:35.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:37:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 34692/184500 objects degraded (18.803%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:35 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:35.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:35.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:35.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:35.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:35.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:35.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: osdmap e222: 8 total, 7 up, 8 in 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: 6.3 scrub starts 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: 6.3 scrub ok 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 34692/184500 objects degraded (18.803%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T12:37:35.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:35 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:35.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:35.488 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:35.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:35.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:35.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:35.633 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:35.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:35.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:35.769 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:35.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:35.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:35.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:35.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:35.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:35.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:35.923 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 42 16 67236 67220 100.012 73 0.0136916 0.00999387 2024-03-24T12:37:36.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:36.038 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:36.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:36.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:36.179 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:36.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:36.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:36.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:36.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: pgmap v466: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 35964/193062 objects degraded (18.628%) 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: 4.1d scrub starts 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: 4.1d scrub ok 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: Adjusting osd_memory_target on smithi029 to 3249M 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:37:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2648948511' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:37:36.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:36 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:37:36.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:36.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: pgmap v466: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 35964/193062 objects degraded (18.628%) 2024-03-24T12:37:36.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: 4.1d scrub starts 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: 4.1d scrub ok 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: Adjusting osd_memory_target on smithi029 to 3249M 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2648948511' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:37:36.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: pgmap v466: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 3.9 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 35964/193062 objects degraded (18.628%) 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: 4.1d scrub starts 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: 4.1d scrub ok 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: Adjusting osd_memory_target on smithi029 to 3249M 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2648948511' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:37:36.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:36 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:37:36.429 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:36.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:36.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:36.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:36.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:36.573 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:36.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:36.617 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:37:36.675 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:36.706 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:36.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:36.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:36.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:36.840 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:36.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:36.923 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 43 15 68362 68347 99.3238 70.4375 0.0073551 0.010063 2024-03-24T12:37:36.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:36.979 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:36.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:37.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:37.109 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:37.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:37.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:37.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:37.249 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:37.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:37.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:37:37.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:37:37.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:37:37.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:37:37.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[29919]: osdmap e223: 8 total, 7 up, 8 in 2024-03-24T12:37:37.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[29919]: pgmap v468: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 990 op/s; 37067/196371 objects degraded (18.876%) 2024-03-24T12:37:37.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:37:37.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:37:37.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:37:37.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:37:37.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[41500]: osdmap e223: 8 total, 7 up, 8 in 2024-03-24T12:37:37.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-mon[41500]: pgmap v468: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 990 op/s; 37067/196371 objects degraded (18.876%) 2024-03-24T12:37:37.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:37 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:37:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:37 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:37:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:37 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:37:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:37 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:37:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:37 smithi190 ceph-mon[33748]: osdmap e223: 8 total, 7 up, 8 in 2024-03-24T12:37:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:37 smithi190 ceph-mon[33748]: pgmap v468: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 990 op/s; 37067/196371 objects degraded (18.876%) 2024-03-24T12:37:37.389 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:37.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:37.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:37.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:37.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:37.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:37.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:37.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:37.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:37.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:37.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:37.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:37.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:37.799 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:37.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:37.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:37.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:37:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:37:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:37:37.923 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 44 16 69702 69686 98.9683 83.6875 0.00968183 0.0100989 2024-03-24T12:37:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:37.940 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:37.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:38.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:38.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:38.092 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:38.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:38.113 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-24T12:37:38.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:38.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:38.222 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:38.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:38.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:38.364 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:38.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:38.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:38.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:37:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:37:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:38 smithi190 ceph-mon[33748]: 3.14 scrub starts 2024-03-24T12:37:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:38 smithi190 ceph-mon[33748]: 3.14 scrub ok 2024-03-24T12:37:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:38 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2596302' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:37:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:38 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:37:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:38 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:38.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:38 smithi029 ceph-mon[29919]: 3.14 scrub starts 2024-03-24T12:37:38.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:38 smithi029 ceph-mon[29919]: 3.14 scrub ok 2024-03-24T12:37:38.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:38 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2596302' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:37:38.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:38 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:37:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:38 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:38.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:38 smithi029 ceph-mon[41500]: 3.14 scrub starts 2024-03-24T12:37:38.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:38 smithi029 ceph-mon[41500]: 3.14 scrub ok 2024-03-24T12:37:38.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:38 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2596302' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:37:38.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:38 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:37:38.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:38 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:38.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:38.510 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:38.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:38.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:38.616 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:37:38.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:38.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:38.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:38.720 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:37:38.720 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:37:38.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:38.794 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:38.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:38.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:38.895 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:38.923 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 45 16 71042 71026 98.6299 83.75 0.0130764 0.0101344 2024-03-24T12:37:38.934 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:38.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:39.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:39.072 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:39.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:39.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:39.212 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:39.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:39.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:39.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:39.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:39.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:39.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:39.381 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:39 smithi029 ceph-mon[41500]: 3.8 scrub starts 2024-03-24T12:37:39.381 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:39 smithi029 ceph-mon[41500]: 3.8 scrub ok 2024-03-24T12:37:39.381 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:39 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:37:39.381 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:39 smithi029 ceph-mon[41500]: osdmap e224: 8 total, 7 up, 8 in 2024-03-24T12:37:39.381 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:39 smithi029 ceph-mon[41500]: pgmap v470: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 56 MiB/s wr, 896 op/s; 37747/200637 objects degraded (18.814%) 2024-03-24T12:37:39.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:39 smithi029 ceph-mon[29919]: 3.8 scrub starts 2024-03-24T12:37:39.381 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:39 smithi029 ceph-mon[29919]: 3.8 scrub ok 2024-03-24T12:37:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:39 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:37:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:39 smithi029 ceph-mon[29919]: osdmap e224: 8 total, 7 up, 8 in 2024-03-24T12:37:39.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:39 smithi029 ceph-mon[29919]: pgmap v470: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 56 MiB/s wr, 896 op/s; 37747/200637 objects degraded (18.814%) 2024-03-24T12:37:39.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:39 smithi190 ceph-mon[33748]: 3.8 scrub starts 2024-03-24T12:37:39.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:39 smithi190 ceph-mon[33748]: 3.8 scrub ok 2024-03-24T12:37:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:39 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:37:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:39 smithi190 ceph-mon[33748]: osdmap e224: 8 total, 7 up, 8 in 2024-03-24T12:37:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:39 smithi190 ceph-mon[33748]: pgmap v470: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 56 MiB/s wr, 896 op/s; 37747/200637 objects degraded (18.814%) 2024-03-24T12:37:39.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:39.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:39.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:39.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:39.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:39.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:39.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:39.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:39.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:39.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:39.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:39.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:39.874 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:39.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:39.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:39.923 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 46 16 72442 72426 98.3878 87.5 0.00834073 0.0101592 2024-03-24T12:37:39.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:40.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:40.021 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:40.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:40.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:37:40.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:40.193 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:40.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:40.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:40.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:40.323 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:40.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:40.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:40 smithi190 ceph-mon[33748]: 4.d scrub starts 2024-03-24T12:37:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:40 smithi190 ceph-mon[33748]: 4.d scrub ok 2024-03-24T12:37:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:40 smithi190 ceph-mon[33748]: 3.b scrub starts 2024-03-24T12:37:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:40 smithi190 ceph-mon[33748]: 3.b scrub ok 2024-03-24T12:37:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:40 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 37747/200637 objects degraded (18.814%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:40 smithi190 ceph-mon[33748]: osdmap e225: 8 total, 7 up, 8 in 2024-03-24T12:37:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[29919]: 4.d scrub starts 2024-03-24T12:37:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[29919]: 4.d scrub ok 2024-03-24T12:37:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[29919]: 3.b scrub starts 2024-03-24T12:37:40.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[29919]: 3.b scrub ok 2024-03-24T12:37:40.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 37747/200637 objects degraded (18.814%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:40.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[29919]: osdmap e225: 8 total, 7 up, 8 in 2024-03-24T12:37:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[41500]: 4.d scrub starts 2024-03-24T12:37:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[41500]: 4.d scrub ok 2024-03-24T12:37:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[41500]: 3.b scrub starts 2024-03-24T12:37:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[41500]: 3.b scrub ok 2024-03-24T12:37:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 37747/200637 objects degraded (18.814%), 24 pgs degraded, 48 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:40 smithi029 ceph-mon[41500]: osdmap e225: 8 total, 7 up, 8 in 2024-03-24T12:37:40.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:40.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:40.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:40.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:40.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:40.596 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:40.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:40.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:40.725 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:40.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:40.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:40.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:40.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:40.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:40.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:40.923 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 47 16 73663 73647 97.9181 76.3125 0.0112021 0.010208 2024-03-24T12:37:40.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:40.982 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:40.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:41.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:41.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:41.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:41.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:41.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:41.242 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:41.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:41.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:41.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:41.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:41.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:41.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:41 smithi029 ceph-mon[29919]: pgmap v472: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 989 op/s; 39280/210858 objects degraded (18.629%) 2024-03-24T12:37:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:41 smithi029 ceph-mon[41500]: pgmap v472: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 989 op/s; 39280/210858 objects degraded (18.629%) 2024-03-24T12:37:41.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:41.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:41.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:41.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:41.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:41.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:41.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:41.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:41 smithi190 ceph-mon[33748]: pgmap v472: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 62 MiB/s wr, 989 op/s; 39280/210858 objects degraded (18.629%) 2024-03-24T12:37:41.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:41.771 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:41.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:41.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:41.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:41.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:41.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:41.923 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 48 16 74837 74821 97.4067 73.375 0.0113023 0.0102613 2024-03-24T12:37:42.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:42.035 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:42.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:42.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:42.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:42.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:42.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:42.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:42.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:42.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:42.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:42.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:42.447 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:42.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:42.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:42.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T12:37:42.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:42.602 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:42.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:42.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:42.754 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:42.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:42.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:42.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:42.901 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:42.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:42.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 49 16 75945 75929 96.8308 69.25 0.0142193 0.0103224 2024-03-24T12:37:43.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:43.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:43.044 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:43.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:43.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:43.184 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:43.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:43.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:43.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:43.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:43.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:43.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:43.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:43.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:43.602 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:43.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:43.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:43.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:43.747 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:43.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:43.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:43.889 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:43.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:43.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:43 smithi029 ceph-mon[41500]: pgmap v473: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 41308/219201 objects degraded (18.845%) 2024-03-24T12:37:43.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:43 smithi029 ceph-mon[29919]: pgmap v473: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 41308/219201 objects degraded (18.845%) 2024-03-24T12:37:43.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 50 15 77282 77267 96.5665 83.625 0.0166716 0.0103508 2024-03-24T12:37:43.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:44.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:44.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:44.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:44.032 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:37:44.090 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:37:44.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:44.133 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:37:44.133 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:37:44.133 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-24T12:37:44.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd out 6 2024-03-24T12:37:44.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:43 smithi190 ceph-mon[33748]: pgmap v473: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.5 KiB/s rd, 73 MiB/s wr, 1.17k op/s; 41308/219201 objects degraded (18.845%) 2024-03-24T12:37:44.158 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:44.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:44.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:44.260 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:44.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:44.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:44.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:44.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:44.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:44.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:44.590 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:44.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:44.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:44.736 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:44.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:44.826 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:44 smithi029 ceph-mon[29919]: 6.12 scrub starts 2024-03-24T12:37:44.827 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:44 smithi029 ceph-mon[29919]: 6.12 scrub ok 2024-03-24T12:37:44.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:44 smithi029 ceph-mon[41500]: 6.12 scrub starts 2024-03-24T12:37:44.827 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:44 smithi029 ceph-mon[41500]: 6.12 scrub ok 2024-03-24T12:37:44.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:44.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:44.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:44.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:44.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 51 16 78408 78392 96.0517 70.3125 0.0103774 0.0104062 2024-03-24T12:37:44.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:45.000 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:45.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:45.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:45.128 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:45.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:45.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:44 smithi190 ceph-mon[33748]: 6.12 scrub starts 2024-03-24T12:37:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:44 smithi190 ceph-mon[33748]: 6.12 scrub ok 2024-03-24T12:37:45.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:45.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:45.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:45.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:45.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:45.389 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:45.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:45.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:45.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:45.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:45.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:45.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:45.659 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:45.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:45.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:45.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:45.796 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:45.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:45.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:45.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:45.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:37:45.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:37:45.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:37:45.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:37:45.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[29919]: pgmap v474: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 42859/229605 objects degraded (18.666%) 2024-03-24T12:37:45.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 42859/229605 objects degraded (18.666%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:45.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[29919]: osdmap e226: 8 total, 7 up, 8 in 2024-03-24T12:37:45.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:37:45.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:37:45.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:37:45.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:37:45.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[41500]: pgmap v474: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 42859/229605 objects degraded (18.666%) 2024-03-24T12:37:45.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 42859/229605 objects degraded (18.666%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:45.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:45 smithi029 ceph-mon[41500]: osdmap e226: 8 total, 7 up, 8 in 2024-03-24T12:37:45.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:45.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 52 15 79740 79725 95.8066 83.3125 0.0106019 0.0104322 2024-03-24T12:37:45.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:45.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:46.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:46.072 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:46.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:46.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:45 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:37:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:45 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:37:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:45 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:37:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:45 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:37:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:45 smithi190 ceph-mon[33748]: pgmap v474: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s; 42859/229605 objects degraded (18.666%) 2024-03-24T12:37:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:45 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 42859/229605 objects degraded (18.666%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:45 smithi190 ceph-mon[33748]: osdmap e226: 8 total, 7 up, 8 in 2024-03-24T12:37:46.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:46.224 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:46.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:46.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:46.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:46.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:46.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:46.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:46.513 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:46.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:46.514 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:46.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:46.633 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:37:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=cleanup t=2024-03-24T12:37:46.253686846Z level=info msg="Completed cleanup jobs" duration=2.363966ms 2024-03-24T12:37:46.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:46.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:46.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:46.781 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:46.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:46.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:46.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:46.924 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:46.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 53 16 81064 81048 95.5589 82.6875 0.00940197 0.0104597 2024-03-24T12:37:46.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:46.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:47.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:47.050 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:47.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:47.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:47.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:47.207 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:47.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:47.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:47.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:47.336 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:47.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:47.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:47.404 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:37:47.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:47.490 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:47.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:47.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:47.630 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:47.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:47.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:47.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:47.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:47.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:47.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:47.892 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:47.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:47.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:47.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:37:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:37:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:37:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:47 smithi029 ceph-mon[41500]: pgmap v476: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.35k op/s; 44000/233028 objects degraded (18.882%) 2024-03-24T12:37:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:47 smithi029 ceph-mon[29919]: pgmap v476: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.35k op/s; 44000/233028 objects degraded (18.882%) 2024-03-24T12:37:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:47.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 54 16 82262 82246 95.1758 74.875 0.00988148 0.0105013 2024-03-24T12:37:47.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:48.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:48.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:48.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:48.025 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:47 smithi190 ceph-mon[33748]: pgmap v476: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.35k op/s; 44000/233028 objects degraded (18.882%) 2024-03-24T12:37:48.026 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:37:48.026 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:37:48.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:48.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:48.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:48.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:48.323 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:48.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:48.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:48.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:37:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:37:48.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:48.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:48.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:48.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:48.590 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:48.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:48.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:48.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:48.726 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:48.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:48.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:48.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:48.869 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:48.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:48.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 55 16 83591 83575 94.9554 83.0625 0.0113197 0.0105253 2024-03-24T12:37:48.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:49.011 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:49.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:49.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:48 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:37:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:48 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:37:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:48 smithi190 ceph-mon[33748]: 4.1a scrub starts 2024-03-24T12:37:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:48 smithi190 ceph-mon[33748]: 4.1a scrub ok 2024-03-24T12:37:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:48 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:49.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:49.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:49.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:48 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:37:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:48 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:37:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:48 smithi029 ceph-mon[29919]: 4.1a scrub starts 2024-03-24T12:37:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:48 smithi029 ceph-mon[29919]: 4.1a scrub ok 2024-03-24T12:37:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:48 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:49.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:48 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:37:49.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:48 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:37:49.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:48 smithi029 ceph-mon[41500]: 4.1a scrub starts 2024-03-24T12:37:49.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:48 smithi029 ceph-mon[41500]: 4.1a scrub ok 2024-03-24T12:37:49.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:48 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:49.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:49.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:49.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:49.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:49.424 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:49.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:49.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:49.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:49.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:49.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:49.687 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:49.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:49.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:49.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:49.820 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:49.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:49.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:49.922 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:49.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 56 16 84724 84708 94.5242 70.8125 0.0242724 0.0105737 2024-03-24T12:37:49.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:49.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:49.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:50.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:50.091 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:50.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:49 smithi190 ceph-mon[33748]: pgmap v477: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 44752/237798 objects degraded (18.819%) 2024-03-24T12:37:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:49 smithi190 ceph-mon[33748]: 5.1 scrub starts 2024-03-24T12:37:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:49 smithi190 ceph-mon[33748]: 5.1 scrub ok 2024-03-24T12:37:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:49 smithi190 ceph-mon[33748]: 5.17 scrub starts 2024-03-24T12:37:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:49 smithi190 ceph-mon[33748]: 5.17 scrub ok 2024-03-24T12:37:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:49 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:37:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:49 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:37:50.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[29919]: pgmap v477: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 44752/237798 objects degraded (18.819%) 2024-03-24T12:37:50.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[29919]: 5.1 scrub starts 2024-03-24T12:37:50.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[29919]: 5.1 scrub ok 2024-03-24T12:37:50.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[29919]: 5.17 scrub starts 2024-03-24T12:37:50.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[29919]: 5.17 scrub ok 2024-03-24T12:37:50.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:37:50.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:37:50.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[41500]: pgmap v477: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 4.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s; 44752/237798 objects degraded (18.819%) 2024-03-24T12:37:50.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[41500]: 5.1 scrub starts 2024-03-24T12:37:50.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[41500]: 5.1 scrub ok 2024-03-24T12:37:50.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[41500]: 5.17 scrub starts 2024-03-24T12:37:50.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[41500]: 5.17 scrub ok 2024-03-24T12:37:50.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:37:50.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:49 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:37:50.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:50.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:50.235 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:50.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:50.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:50.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:50.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:50.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:50.493 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:50.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:50.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:50.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:50.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:50.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:50.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:50.758 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:50.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:50.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:50.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:50.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:50.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 57 16 85764 85748 94.0062 65 0.0318733 0.0106325 2024-03-24T12:37:50.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:51.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:51.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:51.117 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:50 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:37:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:50 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:37:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:50 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:37:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:50 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:37:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:50 smithi190 ceph-mon[33748]: 9.c scrub starts 2024-03-24T12:37:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:50 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 44752/237798 objects degraded (18.819%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:51.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:50 smithi190 ceph-mon[33748]: osdmap e227: 8 total, 7 up, 8 in 2024-03-24T12:37:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:51.141 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:51.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:51.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:37:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:37:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:37:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:37:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[29919]: 9.c scrub starts 2024-03-24T12:37:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 44752/237798 objects degraded (18.819%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[29919]: osdmap e227: 8 total, 7 up, 8 in 2024-03-24T12:37:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:37:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:37:51.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:37:51.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:37:51.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[41500]: 9.c scrub starts 2024-03-24T12:37:51.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 44752/237798 objects degraded (18.819%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:51.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:50 smithi029 ceph-mon[41500]: osdmap e227: 8 total, 7 up, 8 in 2024-03-24T12:37:51.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:51.282 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:51.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:51.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:51.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:51.417 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:51.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:51.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:51.557 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:51.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:51.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:51.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:51.692 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:51.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:51.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:51.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:51.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:51.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:51.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:51.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:51.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 58 16 86627 86611 93.3153 53.9375 0.0137568 0.0107116 2024-03-24T12:37:51.949 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:51.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:51.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:52.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:52.091 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:52.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:52.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:52.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:52.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:52.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:52.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:52.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:52.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:52.363 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:52.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:52.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:52 smithi190 ceph-mon[33748]: pgmap v479: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 46330/248529 objects degraded (18.642%) 2024-03-24T12:37:52.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:52 smithi029 ceph-mon[29919]: pgmap v479: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 46330/248529 objects degraded (18.642%) 2024-03-24T12:37:52.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:52 smithi029 ceph-mon[41500]: pgmap v479: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 5.0 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 76 MiB/s wr, 1.22k op/s; 46330/248529 objects degraded (18.642%) 2024-03-24T12:37:52.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:52.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:52.493 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:52.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:52.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:52.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:52.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:52.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:52.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:52.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:52.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:52.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:52.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:52.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:52.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 59 16 87612 87596 92.7771 61.5625 0.0137533 0.0107737 2024-03-24T12:37:52.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:53.025 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:53.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:53.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:53.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:53.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:53 smithi190 ceph-mon[33748]: 6.b scrub starts 2024-03-24T12:37:53.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:53 smithi190 ceph-mon[33748]: 6.b scrub ok 2024-03-24T12:37:53.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:53 smithi029 ceph-mon[29919]: 6.b scrub starts 2024-03-24T12:37:53.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:53 smithi029 ceph-mon[29919]: 6.b scrub ok 2024-03-24T12:37:53.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:53 smithi029 ceph-mon[41500]: 6.b scrub starts 2024-03-24T12:37:53.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:53 smithi029 ceph-mon[41500]: 6.b scrub ok 2024-03-24T12:37:53.469 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:53.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:53.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:53.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:53.609 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:53.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:53.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:53.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:53.748 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:53.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:53.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:53.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:53.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:53.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:53.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:37:53.929889+0000 min lat: 0.00103404 max lat: 0.488958 avg lat: 0.0108044 2024-03-24T12:37:53.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:37:53.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 60 16 88846 88830 92.5161 77.125 0.0104168 0.0108044 2024-03-24T12:37:53.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:54.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:54.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:54.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:54.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:54.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:54.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:54.305 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:54.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:54.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:54 smithi190 ceph-mon[33748]: 9.c scrub ok 2024-03-24T12:37:54.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:54 smithi190 ceph-mon[33748]: pgmap v480: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 48128/256119 objects degraded (18.791%) 2024-03-24T12:37:54.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:54.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:54 smithi029 ceph-mon[41500]: 9.c scrub ok 2024-03-24T12:37:54.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:54 smithi029 ceph-mon[41500]: pgmap v480: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 48128/256119 objects degraded (18.791%) 2024-03-24T12:37:54.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:54 smithi029 ceph-mon[29919]: 9.c scrub ok 2024-03-24T12:37:54.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:54 smithi029 ceph-mon[29919]: pgmap v480: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 48128/256119 objects degraded (18.791%) 2024-03-24T12:37:54.433 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:54.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:54.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:54.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:54.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:54.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:54.660 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:54.697 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:54.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:54.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:54.842 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:37:54.842 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:37:54.843 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:54.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:54.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:54.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 61 16 89722 89706 91.897 54.75 0.0121878 0.0108765 2024-03-24T12:37:54.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:54.986 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:54.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:55.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:55.130 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:55.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:55.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:55.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:55.273 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:55.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:55.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:55.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:55 smithi190 ceph-mon[33748]: 9.1 scrub starts 2024-03-24T12:37:55.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:55 smithi190 ceph-mon[33748]: pgmap v481: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 49102/261081 objects degraded (18.807%) 2024-03-24T12:37:55.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:55 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 48128/256119 objects degraded (18.791%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:55.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:55 smithi190 ceph-mon[33748]: osdmap e228: 8 total, 7 up, 8 in 2024-03-24T12:37:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:55 smithi029 ceph-mon[41500]: 9.1 scrub starts 2024-03-24T12:37:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:55 smithi029 ceph-mon[41500]: pgmap v481: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 49102/261081 objects degraded (18.807%) 2024-03-24T12:37:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:55 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 48128/256119 objects degraded (18.791%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:55.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:55 smithi029 ceph-mon[41500]: osdmap e228: 8 total, 7 up, 8 in 2024-03-24T12:37:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:55 smithi029 ceph-mon[29919]: 9.1 scrub starts 2024-03-24T12:37:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:55 smithi029 ceph-mon[29919]: pgmap v481: 148 pgs: 24 active+undersized+degraded, 33 active+undersized, 91 active+clean; 5.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 49102/261081 objects degraded (18.807%) 2024-03-24T12:37:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:55 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 48128/256119 objects degraded (18.791%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:37:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:55 smithi029 ceph-mon[29919]: osdmap e228: 8 total, 7 up, 8 in 2024-03-24T12:37:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:55.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:55.439 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:37:55.489 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:37:55.531 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:37:55.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:55.579 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:55.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:55.580 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:37:55.619 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:37:55.665 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:37:55.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:55.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:55.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:55.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:55.709 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:37:55.709 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:37:55.709 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:37:55.709 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:37:55.710 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:37:55.710 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:37:55.710 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:37:55.710 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:37:55.710 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:37:55.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:55.852 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:55.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:55.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 62 16 90718 90702 91.4188 62.25 0.0327502 0.0109333 2024-03-24T12:37:55.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:55.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:55.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:55.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:56.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:56.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:56.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:56.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:56.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:56.266 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:56.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:56.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:56 smithi029 ceph-mon[29919]: 4.0 scrub starts 2024-03-24T12:37:56.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:56 smithi029 ceph-mon[29919]: 4.0 scrub ok 2024-03-24T12:37:56.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:56 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2209054941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:37:56.359 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:56 smithi029 ceph-mon[41500]: 4.0 scrub starts 2024-03-24T12:37:56.359 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:56 smithi029 ceph-mon[41500]: 4.0 scrub ok 2024-03-24T12:37:56.359 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:56 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2209054941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:37:56.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:56.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:56 smithi190 ceph-mon[33748]: 4.0 scrub starts 2024-03-24T12:37:56.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:56 smithi190 ceph-mon[33748]: 4.0 scrub ok 2024-03-24T12:37:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:56 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2209054941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:37:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:56.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:56.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:56.553 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:56.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:56.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:56.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:56.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:56.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:56.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:56.835 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:56.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:56.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 63 16 91950 91934 91.1898 77 0.00855628 0.010962 2024-03-24T12:37:56.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:56.964 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:56.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:57.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:57.095 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:57.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:57.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:57.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:57.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:57.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:57.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:57.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T12:37:57.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:57.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:57 smithi190 ceph-mon[33748]: 9.1 scrub ok 2024-03-24T12:37:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:57 smithi190 ceph-mon[33748]: pgmap v483: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.24k op/s; 50324/267600 objects degraded (18.806%) 2024-03-24T12:37:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:57.389 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:57.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:57.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:57 smithi029 ceph-mon[41500]: 9.1 scrub ok 2024-03-24T12:37:57.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:57 smithi029 ceph-mon[41500]: pgmap v483: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.24k op/s; 50324/267600 objects degraded (18.806%) 2024-03-24T12:37:57.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:57 smithi029 ceph-mon[29919]: 9.1 scrub ok 2024-03-24T12:37:57.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:57 smithi029 ceph-mon[29919]: pgmap v483: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.4 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.24k op/s; 50324/267600 objects degraded (18.806%) 2024-03-24T12:37:57.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:57.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:57.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:57.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:37:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:57.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:37:57.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:57.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:57.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:57.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:57.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:57.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:37:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:37:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:37:57.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 64 16 93060 93044 90.8489 69.375 0.0101299 0.0110015 2024-03-24T12:37:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:57.937 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:57.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:58.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:58.088 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:58.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:58.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:37:58.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:58.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:37:58.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:58.370 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:58.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:58.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:37:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:37:58.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:58 smithi190 ceph-mon[33748]: 9.0 scrub starts 2024-03-24T12:37:58.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:58 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:37:58.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:58 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:37:58.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:58 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:58 smithi029 ceph-mon[41500]: 9.0 scrub starts 2024-03-24T12:37:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:58 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:37:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:58 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:37:58.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:58 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:58 smithi029 ceph-mon[29919]: 9.0 scrub starts 2024-03-24T12:37:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:58 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:37:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:58 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:37:58.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:58 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:37:58.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:58.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:58.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:58.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:58.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:58.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:58.743 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:58.766 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:58.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:58.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:58.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:37:58.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:58.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:58.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:37:58.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 65 16 93953 93937 90.3098 55.8125 0.0135575 0.0110686 2024-03-24T12:37:58.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:59.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:59.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:59.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:59.164 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:59.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:59.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:37:59.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:37:59.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:59.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:37:59.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:37:59 smithi190 ceph-mon[33748]: pgmap v484: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 895 op/s; 50955/271491 objects degraded (18.769%) 2024-03-24T12:37:59.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:59.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:37:59 smithi029 ceph-mon[41500]: pgmap v484: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 895 op/s; 50955/271491 objects degraded (18.769%) 2024-03-24T12:37:59.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:37:59 smithi029 ceph-mon[29919]: pgmap v484: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 56 MiB/s wr, 895 op/s; 50955/271491 objects degraded (18.769%) 2024-03-24T12:37:59.440 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:59.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:59.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:59.542 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:37:59.567 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:59.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:59.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:37:59.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:59.706 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:59.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:59.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:37:59.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:37:59.847 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:59.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:37:59.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 66 16 94900 94884 89.8383 59.1875 0.00373735 0.0111257 2024-03-24T12:37:59.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:37:59.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:37:59.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:37:59.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:00.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:00.100 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:00.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:00.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:00.239 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:00.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:00.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:00.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:00.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:00.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:00.365 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:00 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 50955/271491 objects degraded (18.769%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:00 smithi190 ceph-mon[33748]: osdmap e229: 8 total, 7 up, 8 in 2024-03-24T12:38:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:00 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 50955/271491 objects degraded (18.769%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:00 smithi029 ceph-mon[41500]: osdmap e229: 8 total, 7 up, 8 in 2024-03-24T12:38:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:00 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 50955/271491 objects degraded (18.769%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:00.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:00 smithi029 ceph-mon[29919]: osdmap e229: 8 total, 7 up, 8 in 2024-03-24T12:38:00.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:00.496 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:00.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:00.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:00.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:00.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:00.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:00.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:00.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:00.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:00.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:00.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:00.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:00.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 67 16 95648 95632 89.1951 46.75 0.0034706 0.0112052 2024-03-24T12:38:00.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:01.009 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:01.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:01.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:01.149 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:01.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:01.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:01.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:01.289 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:01.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:01.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:01.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:01.414 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:01.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:01.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:01.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:01 smithi029 ceph-mon[29919]: 9.0 scrub ok 2024-03-24T12:38:01.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:01 smithi029 ceph-mon[29919]: pgmap v486: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.00k op/s; 52234/280257 objects degraded (18.638%) 2024-03-24T12:38:01.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:01 smithi029 ceph-mon[41500]: 9.0 scrub ok 2024-03-24T12:38:01.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:01 smithi029 ceph-mon[41500]: pgmap v486: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.00k op/s; 52234/280257 objects degraded (18.638%) 2024-03-24T12:38:01.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:01.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:01.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:01.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:01 smithi190 ceph-mon[33748]: 9.0 scrub ok 2024-03-24T12:38:01.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:01 smithi190 ceph-mon[33748]: pgmap v486: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.7 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.00k op/s; 52234/280257 objects degraded (18.638%) 2024-03-24T12:38:01.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:01.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:01.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:01.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:01.821 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:01.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:01.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:01.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:01.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 68 16 96796 96780 88.9385 71.75 0.00938991 0.0112393 2024-03-24T12:38:01.949 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:01.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:01.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:02.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:02.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:02.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:02.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:02.231 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:02.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:02.333 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:02.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:02.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:02.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:02.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:02 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:38:02.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:02 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:38:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:02 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:38:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:02 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:38:02.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:02.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:02.498 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:02.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:02.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:02.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:02.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:02 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:38:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:02 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:38:02.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:02.773 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:02.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:02.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:02.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:02.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:02.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 69 15 97768 97753 88.5308 60.8125 0.00464203 0.0112904 2024-03-24T12:38:03.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:03.035 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:03.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:03.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:03.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:03.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:03.169 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:03.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:03.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:03.294 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:03.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:03.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:03.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:03 smithi029 ceph-mon[29919]: 9.4 deep-scrub starts 2024-03-24T12:38:03.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:03 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:38:03.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:03 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:38:03.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:03 smithi029 ceph-mon[29919]: pgmap v487: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.06k op/s; 53790/286653 objects degraded (18.765%) 2024-03-24T12:38:03.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:03.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:03 smithi029 ceph-mon[41500]: 9.4 deep-scrub starts 2024-03-24T12:38:03.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:03 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:38:03.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:03 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:38:03.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:03 smithi029 ceph-mon[41500]: pgmap v487: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.06k op/s; 53790/286653 objects degraded (18.765%) 2024-03-24T12:38:03.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:03.437 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:03.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:03.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:03.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:03.575 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:03.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:03.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:03 smithi190 ceph-mon[33748]: 9.4 deep-scrub starts 2024-03-24T12:38:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:03 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:38:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:03 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:38:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:03 smithi190 ceph-mon[33748]: pgmap v487: 148 pgs: 1 active+clean+scrubbing, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 67 MiB/s wr, 1.06k op/s; 53790/286653 objects degraded (18.765%) 2024-03-24T12:38:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:03.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:03.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:03.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:03.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:03.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:03.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:03.850 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:03.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:03.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 70 16 98720 98704 88.1152 59.4375 0.0146229 0.0113442 2024-03-24T12:38:03.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:03.980 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:03.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:03.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:04.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:04.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:04.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:04.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:04.207 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:04.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:04.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:04.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:04.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:04.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:04.378 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:04.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:04.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:04 smithi029 ceph-mon[29919]: osdmap e230: 8 total, 7 up, 8 in 2024-03-24T12:38:04.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:04 smithi029 ceph-mon[29919]: 9.e scrub starts 2024-03-24T12:38:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:04 smithi029 ceph-mon[41500]: osdmap e230: 8 total, 7 up, 8 in 2024-03-24T12:38:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:04 smithi029 ceph-mon[41500]: 9.e scrub starts 2024-03-24T12:38:04.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:04.502 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:04.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:04.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:04.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:04 smithi190 ceph-mon[33748]: osdmap e230: 8 total, 7 up, 8 in 2024-03-24T12:38:04.678 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:04 smithi190 ceph-mon[33748]: 9.e scrub starts 2024-03-24T12:38:04.718 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:04.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:04.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:04.861 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:04.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:04.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 71 16 99610 99594 87.6575 55.625 0.00360397 0.0113983 2024-03-24T12:38:04.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:04.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:04.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:05.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:05.125 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:05.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:05.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:05.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:05.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:05.267 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:05.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:05.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:05.422 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:05.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:05.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:05.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:05 smithi029 ceph-mon[41500]: pgmap v489: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 972 op/s; 54678/290937 objects degraded (18.794%) 2024-03-24T12:38:05.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:05 smithi029 ceph-mon[41500]: 9.e scrub ok 2024-03-24T12:38:05.426 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:05 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 53790/286653 objects degraded (18.765%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:05.426 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:05 smithi029 ceph-mon[29919]: pgmap v489: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 972 op/s; 54678/290937 objects degraded (18.794%) 2024-03-24T12:38:05.426 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:05 smithi029 ceph-mon[29919]: 9.e scrub ok 2024-03-24T12:38:05.426 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:05 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 53790/286653 objects degraded (18.765%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:05.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:05.567 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:05.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:05.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:05 smithi190 ceph-mon[33748]: pgmap v489: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 972 op/s; 54678/290937 objects degraded (18.794%) 2024-03-24T12:38:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:05 smithi190 ceph-mon[33748]: 9.e scrub ok 2024-03-24T12:38:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:05 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 53790/286653 objects degraded (18.765%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:05.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:05.692 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:05.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:05.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:05.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:05.834 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:05.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:05.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 72 16 100313 100297 87.0503 43.9375 0.0451827 0.011482 2024-03-24T12:38:05.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:05.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:05.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:06.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:06.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:06.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:06.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:06.252 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:06.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:06.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:06.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:06.400 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:06.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:06.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:06.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:06 smithi029 ceph-mon[29919]: 3.1b scrub starts 2024-03-24T12:38:06.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:06 smithi029 ceph-mon[29919]: 3.1b scrub ok 2024-03-24T12:38:06.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:06 smithi029 ceph-mon[29919]: 9.3 scrub starts 2024-03-24T12:38:06.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:06 smithi029 ceph-mon[41500]: 3.1b scrub starts 2024-03-24T12:38:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:06 smithi029 ceph-mon[41500]: 3.1b scrub ok 2024-03-24T12:38:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:06 smithi029 ceph-mon[41500]: 9.3 scrub starts 2024-03-24T12:38:06.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:06.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:06.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:06 smithi190 ceph-mon[33748]: 3.1b scrub starts 2024-03-24T12:38:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:06 smithi190 ceph-mon[33748]: 3.1b scrub ok 2024-03-24T12:38:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:06 smithi190 ceph-mon[33748]: 9.3 scrub starts 2024-03-24T12:38:06.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:06.671 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:06.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:06.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:06.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:06.812 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:06.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:06.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:06.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 73 16 101294 101278 86.6974 61.3125 0.00448277 0.0115263 2024-03-24T12:38:06.951 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:06.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:06.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:07.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:07.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:07.092 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:07.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:07.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:07.239 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:07.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:07.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:07.370 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:38:07.370 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:38:07.381 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:07.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:07.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:07.522 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:07.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:07.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:07 smithi029 ceph-mon[29919]: pgmap v490: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 638 B/s rd, 67 MiB/s wr, 1.07k op/s; 55862/297183 objects degraded (18.797%) 2024-03-24T12:38:07.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:07 smithi029 ceph-mon[41500]: pgmap v490: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 638 B/s rd, 67 MiB/s wr, 1.07k op/s; 55862/297183 objects degraded (18.797%) 2024-03-24T12:38:07.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:07 smithi190 ceph-mon[33748]: pgmap v490: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 6.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 638 B/s rd, 67 MiB/s wr, 1.07k op/s; 55862/297183 objects degraded (18.797%) 2024-03-24T12:38:07.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:07.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:07.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:07.799 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:07.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:07.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:07.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:07.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:38:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:38:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:38:07.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 74 16 102256 102240 86.3383 60.125 0.00586348 0.0115774 2024-03-24T12:38:07.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:07.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:08.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:08.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:08.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:08.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:08.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:08.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:08.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:08.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:08.334 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:08.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:08.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:08.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:38:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:38:08.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:08 smithi190 ceph-mon[33748]: 9.3 scrub ok 2024-03-24T12:38:08.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:08 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:08.412 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:08 smithi029 ceph-mon[41500]: 9.3 scrub ok 2024-03-24T12:38:08.412 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:08 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:08.412 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:08 smithi029 ceph-mon[29919]: 9.3 scrub ok 2024-03-24T12:38:08.412 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:08 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:08.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:08.481 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:08.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:08.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:08.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:08.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:08.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:08.747 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:08.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:08.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:08.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:08.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:08.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:08.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:08.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 75 16 103224 103208 85.9938 60.5 0.0432173 0.011621 2024-03-24T12:38:08.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:09.020 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:09.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:09.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:09.152 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:09.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:09.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:09.191 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:38:09.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:09.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:09.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:09.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:09.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:38:09.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:09.413 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:09 smithi029 ceph-mon[41500]: pgmap v491: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 791 op/s; 56429/300570 objects degraded (18.774%) 2024-03-24T12:38:09.413 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:09 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2799515731' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:38:09.413 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:09 smithi029 ceph-mon[29919]: pgmap v491: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 791 op/s; 56429/300570 objects degraded (18.774%) 2024-03-24T12:38:09.413 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:09 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2799515731' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:38:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:09.457 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:09.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:09.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:09.606 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:09.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:09.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:09 smithi190 ceph-mon[33748]: pgmap v491: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 6.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 49 MiB/s wr, 791 op/s; 56429/300570 objects degraded (18.774%) 2024-03-24T12:38:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:09 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2799515731' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:38:09.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:09 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a[29896]: 2024-03-24T12:38:09.470+0000 7f0eeb338700 -1 mon.a@0(leader).osd e231 definitely_dead 0 2024-03-24T12:38:09.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:09.755 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:09.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:09.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:09.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:09.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:09.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:09.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 76 16 104164 104148 85.6352 58.75 0.00615151 0.01167 2024-03-24T12:38:09.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:10.021 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:10.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:10.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:10.089 INFO:teuthology.orchestra.run.smithi029.stderr:marked out osd.6. 2024-03-24T12:38:10.123 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:10.151 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:10.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:10.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:10.277 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:10.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:10.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:10.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:10.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:10.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:10.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:38:10.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2799515731' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:38:10.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[29919]: osdmap e231: 8 total, 7 up, 8 in 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2962877730' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[29919]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 56429/300570 objects degraded (18.774%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[29919]: osdmap e232: 8 total, 7 up, 7 in 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2799515731' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[41500]: osdmap e231: 8 total, 7 up, 8 in 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2962877730' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-24T12:38:10.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-24T12:38:10.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[41500]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-24T12:38:10.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 56429/300570 objects degraded (18.774%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:10.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-24T12:38:10.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:10 smithi029 ceph-mon[41500]: osdmap e232: 8 total, 7 up, 7 in 2024-03-24T12:38:10.506 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:10.533 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:10.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:10.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:10 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:38:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:10 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2799515731' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:38:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:10 smithi190 ceph-mon[33748]: osdmap e231: 8 total, 7 up, 8 in 2024-03-24T12:38:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:10 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2962877730' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-24T12:38:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:10 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-24T12:38:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:10 smithi190 ceph-mon[33748]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-24T12:38:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:10 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 56429/300570 objects degraded (18.774%), 24 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:10 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-24T12:38:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:10 smithi190 ceph-mon[33748]: osdmap e232: 8 total, 7 up, 7 in 2024-03-24T12:38:10.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:10.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:10.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:10.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:10.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:10.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:10.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:10.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:10.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:10.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:10.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 77 16 104377 104361 84.696 13.3125 0.00469607 0.0116826 2024-03-24T12:38:11.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:11.040 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:11.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:11.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:11.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:11.160 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:38:11.166 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:11.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:11.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:11.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:11.294 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:11.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:11.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:11.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:11.397 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:38:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:11.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:11.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:11.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:11.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:11.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:11.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:11.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:11.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:11.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:11.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:11.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:11.802 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:11.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:11.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:11.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:11.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 78 16 104377 104361 83.6103 0 - 0.0116826 2024-03-24T12:38:11.930 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:11.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:11.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:12.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:12.055 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:12.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:12.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:12.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:12.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:12.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:12.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:12.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:12.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:12 smithi190 ceph-mon[33748]: pgmap v494: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 6.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 684 B/s rd, 60 MiB/s wr, 964 op/s; 57628/308286 objects degraded (18.693%) 2024-03-24T12:38:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:12 smithi190 ceph-mon[33748]: osdmap e233: 8 total, 7 up, 7 in 2024-03-24T12:38:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:12 smithi029 ceph-mon[29919]: pgmap v494: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 6.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 684 B/s rd, 60 MiB/s wr, 964 op/s; 57628/308286 objects degraded (18.693%) 2024-03-24T12:38:12.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:12 smithi029 ceph-mon[29919]: osdmap e233: 8 total, 7 up, 7 in 2024-03-24T12:38:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:12 smithi029 ceph-mon[41500]: pgmap v494: 148 pgs: 1 active+clean+scrubbing+deep, 24 active+undersized+degraded, 33 active+undersized, 90 active+clean; 6.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 684 B/s rd, 60 MiB/s wr, 964 op/s; 57628/308286 objects degraded (18.693%) 2024-03-24T12:38:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:12 smithi029 ceph-mon[41500]: osdmap e233: 8 total, 7 up, 7 in 2024-03-24T12:38:12.434 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:12.457 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:12.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:12.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:38:12.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:12.633 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:12.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:12.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:12.735 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:12.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:12.783 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:12.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:12.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:12.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:12.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:12.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 79 16 104749 104733 82.8462 11.625 0.00345943 0.0120531 2024-03-24T12:38:13.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:13.036 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:13.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:13.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:13.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:13.165 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:13.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:13.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:13.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:13.309 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:13.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:13 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:38:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:13 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:38:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:13 smithi190 ceph-mon[33748]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 231'6470 2024-03-24T12:38:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:13 smithi190 ceph-mon[33748]: osdmap e234: 8 total, 7 up, 7 in 2024-03-24T12:38:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:13 smithi190 ceph-mon[33748]: 9.b starting backfill to osd.2 from (0'0,0'0] MAX to 231'6666 2024-03-24T12:38:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:13 smithi190 ceph-mon[33748]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 231'6532 2024-03-24T12:38:13.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:13.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:38:13.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:38:13.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[41500]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 231'6470 2024-03-24T12:38:13.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[41500]: osdmap e234: 8 total, 7 up, 7 in 2024-03-24T12:38:13.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[41500]: 9.b starting backfill to osd.2 from (0'0,0'0] MAX to 231'6666 2024-03-24T12:38:13.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[41500]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 231'6532 2024-03-24T12:38:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:38:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:38:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[29919]: 9.0 starting backfill to osd.0 from (0'0,0'0] MAX to 231'6470 2024-03-24T12:38:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[29919]: osdmap e234: 8 total, 7 up, 7 in 2024-03-24T12:38:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[29919]: 9.b starting backfill to osd.2 from (0'0,0'0] MAX to 231'6666 2024-03-24T12:38:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:13 smithi029 ceph-mon[29919]: 9.1 starting backfill to osd.4 from (0'0,0'0] MAX to 231'6532 2024-03-24T12:38:13.453 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:13.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:13.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:13.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:13.606 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:13.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:13.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:13.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:13.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:13.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:13.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:13.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:38:13.931800+0000 min lat: 0.00103404 max lat: 2.16603 avg lat: 0.0121097 2024-03-24T12:38:13.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:38:13.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 80 16 105580 105564 82.4598 51.9375 0.00774628 0.0121097 2024-03-24T12:38:13.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:14.004 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:14.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:14.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:14.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:14.130 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:14.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:14.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:14 smithi190 ceph-mon[33748]: pgmap v497: 148 pgs: 8 active+clean+remapped, 88 active+clean, 25 active+undersized, 5 remapped+peering, 20 active+undersized+degraded, 1 active+clean+scrubbing+deep, 1 active+undersized+remapped; 6.3 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 62 MiB/s wr, 989 op/s; 51994/312435 objects degraded (16.642%); 19/312435 objects misplaced (0.006%); 144 KiB/s, 9 objects/s recovering 2024-03-24T12:38:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:14 smithi190 ceph-mon[33748]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-24T12:38:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:14 smithi190 ceph-mon[33748]: osdmap e235: 8 total, 7 up, 7 in 2024-03-24T12:38:14.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:14 smithi029 ceph-mon[29919]: pgmap v497: 148 pgs: 8 active+clean+remapped, 88 active+clean, 25 active+undersized, 5 remapped+peering, 20 active+undersized+degraded, 1 active+clean+scrubbing+deep, 1 active+undersized+remapped; 6.3 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 62 MiB/s wr, 989 op/s; 51994/312435 objects degraded (16.642%); 19/312435 objects misplaced (0.006%); 144 KiB/s, 9 objects/s recovering 2024-03-24T12:38:14.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:14 smithi029 ceph-mon[29919]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-24T12:38:14.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:14 smithi029 ceph-mon[29919]: osdmap e235: 8 total, 7 up, 7 in 2024-03-24T12:38:14.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:14 smithi029 ceph-mon[41500]: pgmap v497: 148 pgs: 8 active+clean+remapped, 88 active+clean, 25 active+undersized, 5 remapped+peering, 20 active+undersized+degraded, 1 active+clean+scrubbing+deep, 1 active+undersized+remapped; 6.3 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 62 MiB/s wr, 989 op/s; 51994/312435 objects degraded (16.642%); 19/312435 objects misplaced (0.006%); 144 KiB/s, 9 objects/s recovering 2024-03-24T12:38:14.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:14 smithi029 ceph-mon[41500]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-24T12:38:14.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:14 smithi029 ceph-mon[41500]: osdmap e235: 8 total, 7 up, 7 in 2024-03-24T12:38:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:14.426 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:14.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:14.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:14.557 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:14.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:14.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:14.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:14.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:14.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:14.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:14.812 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:14.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:14.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:14.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 81 16 106194 106178 81.9155 38.375 0.0401347 0.0122028 2024-03-24T12:38:14.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:14.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:15.045 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:15.068 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:15.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:15.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:15.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:15.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:15.197 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:15.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:15.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:15.341 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:15.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:15 smithi190 ceph-mon[33748]: osdmap e236: 8 total, 7 up, 7 in 2024-03-24T12:38:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:15 smithi190 ceph-mon[33748]: pgmap v500: 148 pgs: 1 active+remapped+backfilling, 12 active+clean+remapped, 86 active+clean, 25 active+undersized, 5 remapped+peering, 1 active+clean+scrubbing+deep, 17 active+undersized+degraded, 1 active+undersized+remapped; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 255 B/s rd, 33 MiB/s wr, 527 op/s; 52402/314619 objects degraded (16.656%); 6470/314619 objects misplaced (2.056%); 2.1 MiB/s, 43 objects/s recovering 2024-03-24T12:38:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:15 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 51994/312435 objects degraded (16.642%), 20 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:15.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:15 smithi029 ceph-mon[29919]: osdmap e236: 8 total, 7 up, 7 in 2024-03-24T12:38:15.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:15 smithi029 ceph-mon[29919]: pgmap v500: 148 pgs: 1 active+remapped+backfilling, 12 active+clean+remapped, 86 active+clean, 25 active+undersized, 5 remapped+peering, 1 active+clean+scrubbing+deep, 17 active+undersized+degraded, 1 active+undersized+remapped; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 255 B/s rd, 33 MiB/s wr, 527 op/s; 52402/314619 objects degraded (16.656%); 6470/314619 objects misplaced (2.056%); 2.1 MiB/s, 43 objects/s recovering 2024-03-24T12:38:15.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:15 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 51994/312435 objects degraded (16.642%), 20 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:15.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:15 smithi029 ceph-mon[41500]: osdmap e236: 8 total, 7 up, 7 in 2024-03-24T12:38:15.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:15 smithi029 ceph-mon[41500]: pgmap v500: 148 pgs: 1 active+remapped+backfilling, 12 active+clean+remapped, 86 active+clean, 25 active+undersized, 5 remapped+peering, 1 active+clean+scrubbing+deep, 17 active+undersized+degraded, 1 active+undersized+remapped; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 255 B/s rd, 33 MiB/s wr, 527 op/s; 52402/314619 objects degraded (16.656%); 6470/314619 objects misplaced (2.056%); 2.1 MiB/s, 43 objects/s recovering 2024-03-24T12:38:15.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:15 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 51994/312435 objects degraded (16.642%), 20 pgs degraded, 45 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:15.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:15.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:15.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:15.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:15.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:15.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:15.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:15.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:15.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:15.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:15.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:15.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:15.870 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:15.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:15.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 82 15 106790 106775 81.3716 37.3125 0.0102217 0.0122837 2024-03-24T12:38:15.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:16.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:16.019 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:16.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:16.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:16.144 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:16.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:16.198 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:38:16.199 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:38:16.199 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-24T12:38:16.199 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.575782 2024-03-24T12:38:16.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 1 0.5757817700508219 2024-03-24T12:38:16.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:16.297 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:16.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:16.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:16.437 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:16.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:16.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:16.563 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:16.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:16.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:16.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:16.689 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:16.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:16.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:16.836 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:16.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:16.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:16.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 83 16 107367 107351 80.8249 36 0.00434187 0.012365 2024-03-24T12:38:16.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:16.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:16.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:16.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:17.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:17.099 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:17.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:17.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:16 smithi190 ceph-mon[33748]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-24T12:38:17.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:16 smithi029 ceph-mon[29919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-24T12:38:17.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:16 smithi029 ceph-mon[41500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-24T12:38:17.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:17.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:17.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:17.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:17.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:17.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:17.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:17.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:17.508 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:17.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:17.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:17.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:17.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:17.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:17.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:17.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:17.797 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:17.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:17.876 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:17 smithi029 ceph-mon[41500]: pgmap v501: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 84 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded, 1 active+undersized+remapped; 6.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 183 B/s rd, 35 MiB/s wr, 558 op/s; 59165/317601 objects degraded (18.629%); 19603/317601 objects misplaced (6.172%); 11 MiB/s, 185 objects/s recovering 2024-03-24T12:38:17.877 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:38:17.877 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:17 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:17.877 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:17 smithi029 ceph-mon[29919]: pgmap v501: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 84 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded, 1 active+undersized+remapped; 6.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 183 B/s rd, 35 MiB/s wr, 558 op/s; 59165/317601 objects degraded (18.629%); 19603/317601 objects misplaced (6.172%); 11 MiB/s, 185 objects/s recovering 2024-03-24T12:38:17.877 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:38:17.877 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:17 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:17.877 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:38:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:38:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:38:17.898 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:17.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:17.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:17.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:17.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 84 15 107980 107965 80.3196 38.375 0.0107886 0.0124452 2024-03-24T12:38:18.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:18.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:18.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:18.062 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:17 smithi190 ceph-mon[33748]: pgmap v501: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 84 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded, 1 active+undersized+remapped; 6.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 183 B/s rd, 35 MiB/s wr, 558 op/s; 59165/317601 objects degraded (18.629%); 19603/317601 objects misplaced (6.172%); 11 MiB/s, 185 objects/s recovering 2024-03-24T12:38:18.062 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:38:18.062 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:17 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:18.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:18.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:18.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:18.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:18.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:18.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:18.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:18.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:18.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:38:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:38:18.418 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:18.446 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:18.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:18.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:18.574 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:18.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:18.675 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:18.713 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:18.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:18.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:18.859 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:18.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:18.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:18 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:18.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:18 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:18.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 85 16 108830 108814 79.9989 53.0625 0.00277682 0.0124922 2024-03-24T12:38:18.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:19.004 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:19.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:19.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:19.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:19.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:18 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:19.147 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:19.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:19.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:19.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:19.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:19.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:19.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:19.413 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:19.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:19.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:19.558 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:19.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:19.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:19.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:19.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:19.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:19.840 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:19.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:19.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:19.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 86 16 109673 109657 79.6813 52.6875 0.0129408 0.0125448 2024-03-24T12:38:19.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:19.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:19.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:20.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:20.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:20.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:20.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:20.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:19 smithi190 ceph-mon[33748]: pgmap v502: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 937 B/s rd, 30 MiB/s wr, 482 op/s; 59566/319971 objects degraded (18.616%); 19603/319971 objects misplaced (6.126%); 9.4 MiB/s, 157 objects/s recovering 2024-03-24T12:38:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:19 smithi029 ceph-mon[41500]: pgmap v502: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 937 B/s rd, 30 MiB/s wr, 482 op/s; 59566/319971 objects degraded (18.616%); 19603/319971 objects misplaced (6.126%); 9.4 MiB/s, 157 objects/s recovering 2024-03-24T12:38:20.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:19 smithi029 ceph-mon[29919]: pgmap v502: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 937 B/s rd, 30 MiB/s wr, 482 op/s; 59566/319971 objects degraded (18.616%); 19603/319971 objects misplaced (6.126%); 9.4 MiB/s, 157 objects/s recovering 2024-03-24T12:38:20.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:20.235 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:20.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:20.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:20.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:20.369 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:20.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:20.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:20.509 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:20.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:20.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:20.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:20.655 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:20.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:20.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:20.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:20.798 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:20.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:20.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:20.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:20.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 87 16 110356 110340 79.256 42.6875 0.0120703 0.0126122 2024-03-24T12:38:20.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:20.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:20.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:21.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:21.055 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:21.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:21.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:20 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 59566/319971 objects degraded (18.616%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:21.157 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:20 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 59566/319971 objects degraded (18.616%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:21.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:20 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 59566/319971 objects degraded (18.616%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:21.186 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:21.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:21.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:21.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:21.333 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:21.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:21.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:21.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:21.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:21.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:21.603 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:21.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:21.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:21.729 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:21.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:21.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:21.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:21.875 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:21.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:21.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 88 16 110852 110836 78.7077 31 0.0112204 0.0127001 2024-03-24T12:38:21.977 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:22.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:22.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:22.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:22.140 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:22.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:22.242 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:22.269 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:22.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:22.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:22.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:22.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:22 smithi190 ceph-mon[33748]: pgmap v503: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 255 B/s rd, 35 MiB/s wr, 558 op/s; 60462/325842 objects degraded (18.556%); 18754/325842 objects misplaced (5.756%); 17 MiB/s, 279 objects/s recovering 2024-03-24T12:38:22.414 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:22.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:22.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:22 smithi029 ceph-mon[41500]: pgmap v503: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 255 B/s rd, 35 MiB/s wr, 558 op/s; 60462/325842 objects degraded (18.556%); 18754/325842 objects misplaced (5.756%); 17 MiB/s, 279 objects/s recovering 2024-03-24T12:38:22.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:22 smithi029 ceph-mon[29919]: pgmap v503: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 255 B/s rd, 35 MiB/s wr, 558 op/s; 60462/325842 objects degraded (18.556%); 18754/325842 objects misplaced (5.756%); 17 MiB/s, 279 objects/s recovering 2024-03-24T12:38:22.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:22.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:22.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:22.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:22.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:22.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:22.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:22.812 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:22.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:22.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:22.927 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 89 16 111489 111473 78.2706 39.8125 0.00859361 0.0127704 2024-03-24T12:38:22.956 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:22.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:23.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:23.081 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:23.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:23.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:23.210 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:23.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:23.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:23.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:23.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:23.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:23.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:23.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:23.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:23.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:23.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:23.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:23.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:23.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:23.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:23.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:23.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:23.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Total time run: 90.009 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Total writes made: 112063 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Write size: 65536 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Object size: 65536 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Bandwidth (MB/sec): 77.8137 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev Bandwidth: 38.7773 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max bandwidth (MB/sec): 209.5 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min bandwidth (MB/sec): 0 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Average IOPS: 1245 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev IOPS: 620.437 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max IOPS: 3352 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min IOPS: 0 2024-03-24T12:38:23.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Average Latency(s): 0.0128471 2024-03-24T12:38:23.929 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev Latency(s): 0.0338677 2024-03-24T12:38:23.929 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max latency(s): 2.16603 2024-03-24T12:38:23.929 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min latency(s): 0.00103404 2024-03-24T12:38:23.935 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T12:38:23.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:24.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:24.026 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:24.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:24.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:24.150 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:24.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:24.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:24.280 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:24.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:24.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:24 smithi190 ceph-mon[33748]: pgmap v504: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 750 B/s rd, 39 MiB/s wr, 627 op/s; 60809/330378 objects degraded (18.406%); 18966/330378 objects misplaced (5.741%); 19 MiB/s, 308 objects/s recovering 2024-03-24T12:38:24.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:24 smithi029 ceph-mon[29919]: pgmap v504: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 750 B/s rd, 39 MiB/s wr, 627 op/s; 60809/330378 objects degraded (18.406%); 18966/330378 objects misplaced (5.741%); 19 MiB/s, 308 objects/s recovering 2024-03-24T12:38:24.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:24 smithi029 ceph-mon[41500]: pgmap v504: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.7 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 750 B/s rd, 39 MiB/s wr, 627 op/s; 60809/330378 objects degraded (18.406%); 18966/330378 objects misplaced (5.741%); 19 MiB/s, 308 objects/s recovering 2024-03-24T12:38:24.422 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:24.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:24.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:24.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:24.551 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:24.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:24.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:24.691 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:24.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:24.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:24.793 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:24.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:24.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:24.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:24.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:24.946 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:24.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:24.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:25.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:25.089 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:25.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:25.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:25.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:25.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:25.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:25.365 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:25.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:25.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:25 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 60809/330378 objects degraded (18.406%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:25.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:25 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 60809/330378 objects degraded (18.406%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:25.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:25 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 60809/330378 objects degraded (18.406%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:25.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:25.494 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:25.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:25.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:25.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:25.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:25.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:25.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:25.759 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:25.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:25.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:25.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:25.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:25.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:25.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:26.015 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:26.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:26.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:26.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:26.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:26.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:26.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:26.270 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:26.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:26.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:26.368 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:26 smithi029 ceph-mon[29919]: pgmap v505: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 776 B/s rd, 40 MiB/s wr, 651 op/s; 61381/333108 objects degraded (18.427%); 18733/333108 objects misplaced (5.624%); 20 MiB/s, 315 objects/s recovering 2024-03-24T12:38:26.369 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:26 smithi029 ceph-mon[41500]: pgmap v505: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 776 B/s rd, 40 MiB/s wr, 651 op/s; 61381/333108 objects degraded (18.427%); 18733/333108 objects misplaced (5.624%); 20 MiB/s, 315 objects/s recovering 2024-03-24T12:38:26.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:26 smithi190 ceph-mon[33748]: pgmap v505: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 776 B/s rd, 40 MiB/s wr, 651 op/s; 61381/333108 objects degraded (18.427%); 18733/333108 objects misplaced (5.624%); 20 MiB/s, 315 objects/s recovering 2024-03-24T12:38:26.398 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:26.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:26.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:26.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:26.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:26.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:26.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:26.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:26.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:26.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:26.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:26.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:26.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:26.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:26.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:26.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:26.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:27.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:27.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:27.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:27.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:27.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:27.194 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:27.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:27.295 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:27.338 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:27.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:27.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:27 smithi029 ceph-mon[41500]: pgmap v506: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 37 MiB/s wr, 641 op/s; 60729/334545 objects degraded (18.153%); 18204/334545 objects misplaced (5.441%); 24 MiB/s, 383 objects/s recovering 2024-03-24T12:38:27.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:27 smithi029 ceph-mon[29919]: pgmap v506: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 37 MiB/s wr, 641 op/s; 60729/334545 objects degraded (18.153%); 18204/334545 objects misplaced (5.441%); 24 MiB/s, 383 objects/s recovering 2024-03-24T12:38:27.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:27.482 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:27.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:27.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:27.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:27.610 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:27.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:27.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:27 smithi190 ceph-mon[33748]: pgmap v506: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 852 B/s rd, 37 MiB/s wr, 641 op/s; 60729/334545 objects degraded (18.153%); 18204/334545 objects misplaced (5.441%); 24 MiB/s, 383 objects/s recovering 2024-03-24T12:38:27.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:27.736 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:27.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:27.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:27.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:27.862 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:27.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:27.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:38:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:38:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:38:27.963 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:28.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:28.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:28.127 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:28.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:28.229 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:28.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:28.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:28.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:28.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:28.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:28 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:28.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:38:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:38:28.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:28.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:28.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:28 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:28.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:28 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:28.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:28.532 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:28.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:28.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:28.656 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:28.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:28.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:28.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:28.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:28.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:28.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:28.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:28.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:28.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:28.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:29.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:29.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:29.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:29.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:29.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:29.165 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:29.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:29.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:29.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:29.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:29.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:29.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:29 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:38:29.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:29 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:38:29.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:29 smithi029 ceph-mon[29919]: osdmap e237: 8 total, 7 up, 7 in 2024-03-24T12:38:29.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:29 smithi029 ceph-mon[29919]: pgmap v508: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 35 MiB/s wr, 683 op/s; 60501/333117 objects degraded (18.162%); 18204/333117 objects misplaced (5.465%); 23 MiB/s, 374 objects/s recovering 2024-03-24T12:38:29.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:29 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:38:29.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:29 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:38:29.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:29 smithi029 ceph-mon[41500]: osdmap e237: 8 total, 7 up, 7 in 2024-03-24T12:38:29.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:29 smithi029 ceph-mon[41500]: pgmap v508: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 35 MiB/s wr, 683 op/s; 60501/333117 objects degraded (18.162%); 18204/333117 objects misplaced (5.465%); 23 MiB/s, 374 objects/s recovering 2024-03-24T12:38:29.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:29.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:29.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:29.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:29.612 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:29.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:29.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:29 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:38:29.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:29 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:38:29.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:29 smithi190 ceph-mon[33748]: osdmap e237: 8 total, 7 up, 7 in 2024-03-24T12:38:29.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:29 smithi190 ceph-mon[33748]: pgmap v508: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.1 KiB/s rd, 35 MiB/s wr, 683 op/s; 60501/333117 objects degraded (18.162%); 18204/333117 objects misplaced (5.465%); 23 MiB/s, 374 objects/s recovering 2024-03-24T12:38:29.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:29.754 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:29.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:29.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:29.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:29.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:29.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:29.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:29.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:30.006 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:30.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:30.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:30.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:30.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:30.236 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:30.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:30.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:30.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:30.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:30.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:30.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:30 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 60501/333117 objects degraded (18.162%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:30 smithi029 ceph-mon[29919]: osdmap e238: 8 total, 7 up, 7 in 2024-03-24T12:38:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:30 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 60501/333117 objects degraded (18.162%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:30.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:30 smithi029 ceph-mon[41500]: osdmap e238: 8 total, 7 up, 7 in 2024-03-24T12:38:30.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:30.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:30.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:30.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:30.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:30 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 60501/333117 objects degraded (18.162%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:30 smithi190 ceph-mon[33748]: osdmap e238: 8 total, 7 up, 7 in 2024-03-24T12:38:30.646 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:30.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:30.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:30.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:30.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:30.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:30.900 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:30.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:31.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:31.043 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:31.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:31.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:31.188 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:31.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:31.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:31.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:31.313 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:31.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:31.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:31.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:31.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:31 smithi029 ceph-mon[29919]: pgmap v510: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 767 B/s rd, 17 MiB/s wr, 651 op/s; 59651/327564 objects degraded (18.210%); 16830/327564 objects misplaced (5.138%); 22 MiB/s, 356 objects/s recovering 2024-03-24T12:38:31.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:31 smithi029 ceph-mon[41500]: pgmap v510: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 767 B/s rd, 17 MiB/s wr, 651 op/s; 59651/327564 objects degraded (18.210%); 16830/327564 objects misplaced (5.138%); 22 MiB/s, 356 objects/s recovering 2024-03-24T12:38:31.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:31.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:31.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:31.566 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:31.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:31.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:31.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:31 smithi190 ceph-mon[33748]: pgmap v510: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.7 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 767 B/s rd, 17 MiB/s wr, 651 op/s; 59651/327564 objects degraded (18.210%); 16830/327564 objects misplaced (5.138%); 22 MiB/s, 356 objects/s recovering 2024-03-24T12:38:31.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:31.709 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:31.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:31.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:31.811 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:31.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:31.835 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:31.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:31.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:31.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:31.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:32.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:32.111 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:32.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:32.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:32.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:32.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:32.238 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:32.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:32.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:32.366 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:32.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:32.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:32.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:32.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:32.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:32.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:32.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:32.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:32.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:32.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:32.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:32.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:32.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:32.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:32.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:33.021 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:33.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:33.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:33.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:33.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:33.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:33.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:33.287 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:33.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:33.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:33.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:33.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:33.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:33.537 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:33.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:33.639 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:33.668 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:33.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:33.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:33.815 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:33.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:33.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:33.941 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:33.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:34.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:34.067 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:34.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:34.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:33 smithi190 ceph-mon[33748]: pgmap v511: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 9.1 MiB/s wr, 675 op/s; 58338/323925 objects degraded (18.010%); 16616/323925 objects misplaced (5.130%); 23 MiB/s, 368 objects/s recovering 2024-03-24T12:38:34.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:33 smithi029 ceph-mon[41500]: pgmap v511: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 9.1 MiB/s wr, 675 op/s; 58338/323925 objects degraded (18.010%); 16616/323925 objects misplaced (5.130%); 23 MiB/s, 368 objects/s recovering 2024-03-24T12:38:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:34.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:33 smithi029 ceph-mon[29919]: pgmap v511: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.6 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 9.1 MiB/s wr, 675 op/s; 58338/323925 objects degraded (18.010%); 16616/323925 objects misplaced (5.130%); 23 MiB/s, 368 objects/s recovering 2024-03-24T12:38:34.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:34.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:34.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:34.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:34.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:34.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:34.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:34.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:34.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:34.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:34.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:34.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:34.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:34.583 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:34.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:34.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:34.725 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:34.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:34.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:34.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:34.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:34.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:34.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:34.953 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:34.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:34.980 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:34.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:35.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:35.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:35.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:35.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:35.232 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:35.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:35.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:35.334 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:35.361 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:35.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:35.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:35.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:35.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:35.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:35.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:35.621 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:35.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:35.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:35.746 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:35.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:35.848 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:35.884 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:35.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:35.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:35.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:35 smithi029 ceph-mon[41500]: pgmap v512: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 1.7 MiB/s wr, 594 op/s; 57870/321633 objects degraded (17.993%); 16107/321633 objects misplaced (5.008%); 15 MiB/s, 241 objects/s recovering 2024-03-24T12:38:35.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:35 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 57870/321633 objects degraded (17.993%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:35.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:35 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:38:35.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:35 smithi029 ceph-mon[29919]: pgmap v512: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 1.7 MiB/s wr, 594 op/s; 57870/321633 objects degraded (17.993%); 16107/321633 objects misplaced (5.008%); 15 MiB/s, 241 objects/s recovering 2024-03-24T12:38:35.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:35 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 57870/321633 objects degraded (17.993%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:35.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:35 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:38:35.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:36.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:36.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:36.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:35 smithi190 ceph-mon[33748]: pgmap v512: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.4 KiB/s rd, 1.7 MiB/s wr, 594 op/s; 57870/321633 objects degraded (17.993%); 16107/321633 objects misplaced (5.008%); 15 MiB/s, 241 objects/s recovering 2024-03-24T12:38:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:35 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 57870/321633 objects degraded (17.993%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:35 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:38:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:36.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:36.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:36.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:36.280 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:36.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:36.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:36.382 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:36.409 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:36.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:36.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:36.510 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:36.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:36.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:36.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:36.659 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:36.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:36.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:36.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:36.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:36.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:36.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:36.920 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:36.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:37.022 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:37.045 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:37.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:37.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:37.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:37.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:37.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:37.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:37.272 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:37.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:37.300 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:37.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:37 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:38:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:37 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:38:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:37 smithi190 ceph-mon[33748]: osdmap e239: 8 total, 7 up, 7 in 2024-03-24T12:38:37.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:37 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:38:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:37 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:38:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:37 smithi029 ceph-mon[41500]: osdmap e239: 8 total, 7 up, 7 in 2024-03-24T12:38:37.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:37 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:38:37.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:37 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:38:37.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:37 smithi029 ceph-mon[29919]: osdmap e239: 8 total, 7 up, 7 in 2024-03-24T12:38:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:37.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:37.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:37.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:37.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:37.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:37.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:37.700 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:37.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:37.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:37.844 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:37.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:37.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:38:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:38:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:38:37.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:37.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:37.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:37.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:38.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:38.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:38.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:38.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:38.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:38.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:38.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:38 smithi190 ceph-mon[33748]: pgmap v514: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s; 56611/318051 objects degraded (17.799%); 15065/318051 objects misplaced (4.737%); 26 MiB/s, 414 objects/s recovering 2024-03-24T12:38:38.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:38:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:38:38.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:38.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:38.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:38.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:38 smithi029 ceph-mon[29919]: pgmap v514: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s; 56611/318051 objects degraded (17.799%); 15065/318051 objects misplaced (4.737%); 26 MiB/s, 414 objects/s recovering 2024-03-24T12:38:38.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:38 smithi029 ceph-mon[41500]: pgmap v514: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.5 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 628 op/s; 56611/318051 objects degraded (17.799%); 15065/318051 objects misplaced (4.737%); 26 MiB/s, 414 objects/s recovering 2024-03-24T12:38:38.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:38.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:38.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:38.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:38.659 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:38.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:38.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:38.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:38.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:38.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:38.922 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:38.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:38.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:39.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:39.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:39.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:39.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:39.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:39.177 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:39.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:39.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:39.321 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:39.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:39.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:39 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:38:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:39 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:38:39.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:39.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:39 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:38:39.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:39 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:38:39.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:39.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:39 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:38:39.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:39 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:38:39.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:39.423 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:39.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:39.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:39.565 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:39.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:39.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:39.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:39.695 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:39.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:39.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:39.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:39.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:39.854 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:39.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:39.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:39.979 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:39.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:39.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:40.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:40.123 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:40.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:40.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:40 smithi190 ceph-mon[33748]: pgmap v515: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 456 op/s; 56263/315855 objects degraded (17.813%); 15065/315855 objects misplaced (4.770%); 17 MiB/s, 264 objects/s recovering 2024-03-24T12:38:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:40 smithi190 ceph-mon[33748]: osdmap e240: 8 total, 7 up, 7 in 2024-03-24T12:38:40.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:40 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:38:40.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:40 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:38:40.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:40 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:38:40.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:40 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 56263/315855 objects degraded (17.813%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:40.224 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:40.249 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:40.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:40.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:40.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:40.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:40.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[29919]: pgmap v515: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 456 op/s; 56263/315855 objects degraded (17.813%); 15065/315855 objects misplaced (4.770%); 17 MiB/s, 264 objects/s recovering 2024-03-24T12:38:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[29919]: osdmap e240: 8 total, 7 up, 7 in 2024-03-24T12:38:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:38:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:38:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:38:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 56263/315855 objects degraded (17.813%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[41500]: pgmap v515: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 456 op/s; 56263/315855 objects degraded (17.813%); 15065/315855 objects misplaced (4.770%); 17 MiB/s, 264 objects/s recovering 2024-03-24T12:38:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[41500]: osdmap e240: 8 total, 7 up, 7 in 2024-03-24T12:38:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:38:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:38:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:38:40.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:40 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 56263/315855 objects degraded (17.813%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:40.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:40.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:40.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:40.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:40.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:40.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:40.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:40.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:40.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:40.799 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:40.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:40.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:40.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:40.933 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:40.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:40.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:41.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:41.072 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:41.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:41.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:41.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:41.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:41.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:41.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:41.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:41.352 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:41.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:41.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:41 smithi190 ceph-mon[33748]: pgmap v517: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 540 op/s; 55536/310974 objects degraded (17.859%); 13580/310974 objects misplaced (4.367%); 24 MiB/s, 382 objects/s recovering 2024-03-24T12:38:41.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:41 smithi029 ceph-mon[29919]: pgmap v517: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 540 op/s; 55536/310974 objects degraded (17.859%); 13580/310974 objects misplaced (4.367%); 24 MiB/s, 382 objects/s recovering 2024-03-24T12:38:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:41 smithi029 ceph-mon[41500]: pgmap v517: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 540 op/s; 55536/310974 objects degraded (17.859%); 13580/310974 objects misplaced (4.367%); 24 MiB/s, 382 objects/s recovering 2024-03-24T12:38:41.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:41.491 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:41.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:41.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:41.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:41.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:41.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:41.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:41.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:41.759 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:41.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:41.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:41.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:41.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:41.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:41.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:41.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:42.014 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:42.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:42.017 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:42.118 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:42.155 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:42.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:42.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:42.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:42.280 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:42.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:42.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:42.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:42 smithi190 ceph-mon[33748]: 9.4 deep-scrub ok 2024-03-24T12:38:42.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:42 smithi029 ceph-mon[29919]: 9.4 deep-scrub ok 2024-03-24T12:38:42.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:42 smithi029 ceph-mon[41500]: 9.4 deep-scrub ok 2024-03-24T12:38:42.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:42.427 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:42.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:42.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:42.568 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:42.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:42.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:42.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:42.704 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:42.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:42.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:42.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:42.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:42.952 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:42.980 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:42.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:43.082 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:43.111 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:43.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:43.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:43.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:43.248 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:43.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:43.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:43 smithi190 ceph-mon[33748]: pgmap v518: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 644 op/s; 53835/306186 objects degraded (17.582%); 13347/306186 objects misplaced (4.359%); 26 MiB/s, 416 objects/s recovering 2024-03-24T12:38:43.390 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:43.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:43.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:43 smithi029 ceph-mon[29919]: pgmap v518: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 644 op/s; 53835/306186 objects degraded (17.582%); 13347/306186 objects misplaced (4.359%); 26 MiB/s, 416 objects/s recovering 2024-03-24T12:38:43.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:43 smithi029 ceph-mon[41500]: pgmap v518: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 85 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 6.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 644 op/s; 53835/306186 objects degraded (17.582%); 13347/306186 objects misplaced (4.359%); 26 MiB/s, 416 objects/s recovering 2024-03-24T12:38:43.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:43.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:43.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:43.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:43.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:43.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:43.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:43.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:43.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:43.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:43.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:43.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:43.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:43.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:43.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:44.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:44.035 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:44.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:44.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:44.162 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:44.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:44.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:44.288 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:44.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:44.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:44.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:44.418 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:44.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:44.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:44.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:44.548 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:44.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:44.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:44.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:44.677 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:44.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:44.779 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:44.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:44.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:44.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:44.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:44.928 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:44.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:44.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:45.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:45.071 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:45.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:45.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:45.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:45.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:45.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:45.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:45.326 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:45.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:45.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:45.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:45.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:45.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:45.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:45.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:45.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:45.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:45.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:45.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:45.823 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:45.846 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:45.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:45.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:45.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:45.976 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:45.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:45.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:46.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:46.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:46.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:46.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:45 smithi190 ceph-mon[33748]: pgmap v519: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 6.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 682 op/s; 53027/302157 objects degraded (17.549%); 12619/302157 objects misplaced (4.176%); 23 MiB/s, 364 objects/s recovering 2024-03-24T12:38:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:45 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 53027/302157 objects degraded (17.549%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:45 smithi029 ceph-mon[29919]: pgmap v519: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 6.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 682 op/s; 53027/302157 objects degraded (17.549%); 12619/302157 objects misplaced (4.176%); 23 MiB/s, 364 objects/s recovering 2024-03-24T12:38:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:45 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 53027/302157 objects degraded (17.549%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:45 smithi029 ceph-mon[41500]: pgmap v519: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 6.1 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 682 op/s; 53027/302157 objects degraded (17.549%); 12619/302157 objects misplaced (4.176%); 23 MiB/s, 364 objects/s recovering 2024-03-24T12:38:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:45 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 53027/302157 objects degraded (17.549%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:46.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:46.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:46.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:46.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:46.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:46.398 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:46.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:46.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:46.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:46.550 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:46.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:46.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:46.692 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:46.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:46.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:46.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:46.838 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:46.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:46.939 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:46.992 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:46.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:47.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:47.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:47.180 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:38:47.180 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:38:47.180 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-03-24T12:38:47.180 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-03-24T12:38:47.180 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:38:47.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:47.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:47.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:47.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:47.309 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:38:47.355 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:38:47.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:47.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:47.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:47.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:47.396 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:38:47.435 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:38:47.475 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:38:47.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:47.514 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:38:47.516 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:38:47.540 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:47.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:47.544 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:38:47.545 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:38:47.545 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:38:47.545 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:38:47.545 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:38:47.545 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:38:47.545 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:38:47.545 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:38:47.545 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:38:47.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:47.666 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:47.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:47.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:47.706 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:38:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:38:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:38:47.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:47.807 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:47.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:47.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:47.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:47.951 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:47.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:48.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:48.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:47 smithi029 ceph-mon[29919]: pgmap v520: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 713 op/s; 51516/296664 objects degraded (17.365%); 11625/296664 objects misplaced (3.919%); 23 MiB/s, 372 objects/s recovering 2024-03-24T12:38:48.063 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:47 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1177439485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:38:48.063 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:47 smithi029 ceph-mon[41500]: pgmap v520: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 713 op/s; 51516/296664 objects degraded (17.365%); 11625/296664 objects misplaced (3.919%); 23 MiB/s, 372 objects/s recovering 2024-03-24T12:38:48.063 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:47 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1177439485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:38:48.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:48.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:48.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:48.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:47 smithi190 ceph-mon[33748]: pgmap v520: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 6.0 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 713 op/s; 51516/296664 objects degraded (17.365%); 11625/296664 objects misplaced (3.919%); 23 MiB/s, 372 objects/s recovering 2024-03-24T12:38:48.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:47 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1177439485' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:38:48.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:48.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:48.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:48.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:48.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:48.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:38:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:38:48.405 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:38:48.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:48.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T12:38:48.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:48.499 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:48.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:48.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:48.651 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:48.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:48.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:48.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:48.779 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:48.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:48.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:48.882 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:48.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:48.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:49.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:49.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:49.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:49.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:49.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:49.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:49.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:38:49.184 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:49.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:49.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:49.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:49.327 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:49.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:49.428 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:49.465 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:49.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:49.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:49.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:49.607 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:49.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:49.636 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:38:49.637 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:38:49.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:49.755 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:49.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:49.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:49.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:49.867 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:49 smithi029 ceph-mon[29919]: pgmap v521: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 6.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 753 op/s; 50914/293259 objects degraded (17.361%); 11625/293259 objects misplaced (3.964%); 23 MiB/s, 372 objects/s recovering 2024-03-24T12:38:49.867 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:49.868 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:49 smithi029 ceph-mon[41500]: pgmap v521: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 6.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 753 op/s; 50914/293259 objects degraded (17.361%); 11625/293259 objects misplaced (3.964%); 23 MiB/s, 372 objects/s recovering 2024-03-24T12:38:49.901 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:49.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:49.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:50.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:50.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:50.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:49 smithi190 ceph-mon[33748]: pgmap v521: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 6.0 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 753 op/s; 50914/293259 objects degraded (17.361%); 11625/293259 objects misplaced (3.964%); 23 MiB/s, 372 objects/s recovering 2024-03-24T12:38:50.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:50.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:50.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:50.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:50.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:50.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:50.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:50.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:50.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:50.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:50.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:50.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:50.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:50.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:50.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:50.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:50.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:50.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:50.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:50.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:50.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:50.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:50.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:50.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:50 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 50914/293259 objects degraded (17.361%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:50.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:50 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 50914/293259 objects degraded (17.361%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:50.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:51.000 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:51.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:51.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:51.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:51.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:51.126 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:51.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:51.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:50 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 50914/293259 objects degraded (17.361%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:51.227 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:51.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:51.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:51.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:51.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:51.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:51.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:51.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:51.540 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:51.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:51.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:38:51.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:51.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:51.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:51.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:51.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:51.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:51.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:51.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:51.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:51.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:52.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:52.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:52.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:52.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:52.185 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:52.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:52.287 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:52.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:52.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:52.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:52 smithi190 ceph-mon[33748]: pgmap v522: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 885 B/s rd, 0 B/s wr, 893 op/s; 49594/284346 objects degraded (17.441%); 10090/284346 objects misplaced (3.548%); 27 MiB/s, 433 objects/s recovering 2024-03-24T12:38:52.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:52.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:52 smithi029 ceph-mon[41500]: pgmap v522: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 885 B/s rd, 0 B/s wr, 893 op/s; 49594/284346 objects degraded (17.441%); 10090/284346 objects misplaced (3.548%); 27 MiB/s, 433 objects/s recovering 2024-03-24T12:38:52.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:52 smithi029 ceph-mon[29919]: pgmap v522: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.8 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 885 B/s rd, 0 B/s wr, 893 op/s; 49594/284346 objects degraded (17.441%); 10090/284346 objects misplaced (3.548%); 27 MiB/s, 433 objects/s recovering 2024-03-24T12:38:52.451 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:52.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:52.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:52.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:52.592 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:52.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:52.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:52.723 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:52.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:52.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:52.864 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:52.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:52.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:52.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:53.007 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:53.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:53.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:53.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:53.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:53.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:53.288 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:53.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:53.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:53.412 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:53.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:53.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:53.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:53.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:53.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:53.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:53.665 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:53.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:53.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:53.802 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:53.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:53.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:53.947 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:53.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:54.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:54.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:54.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:54.175 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:54.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:54.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:54.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:54.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:54.329 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:54.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:54.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:54.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:54 smithi190 ceph-mon[33748]: pgmap v523: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 47724/277956 objects degraded (17.170%); 9804/277956 objects misplaced (3.527%); 23 MiB/s, 360 objects/s recovering 2024-03-24T12:38:54.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:54 smithi029 ceph-mon[41500]: pgmap v523: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 47724/277956 objects degraded (17.170%); 9804/277956 objects misplaced (3.527%); 23 MiB/s, 360 objects/s recovering 2024-03-24T12:38:54.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:54 smithi029 ceph-mon[29919]: pgmap v523: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 917 op/s; 47724/277956 objects degraded (17.170%); 9804/277956 objects misplaced (3.527%); 23 MiB/s, 360 objects/s recovering 2024-03-24T12:38:54.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:54.471 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:54.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:54.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:54.597 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:54.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:54.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:54.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:54.723 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:54.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:54.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:54.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:54.855 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:54.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:54.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:54.987 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:54.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:54.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:55.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:55.112 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:55.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:55.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:55.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:55.238 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:55.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:55.339 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:55.364 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:55.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:55.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:38:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:38:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[41500]: osdmap e241: 8 total, 7 up, 7 in 2024-03-24T12:38:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[41500]: pgmap v525: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s; 46837/273639 objects degraded (17.116%); 9286/273639 objects misplaced (3.394%); 22 MiB/s, 347 objects/s recovering 2024-03-24T12:38:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 47724/277956 objects degraded (17.170%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[41500]: osdmap e242: 8 total, 7 up, 7 in 2024-03-24T12:38:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:38:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:38:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[29919]: osdmap e241: 8 total, 7 up, 7 in 2024-03-24T12:38:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[29919]: pgmap v525: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s; 46837/273639 objects degraded (17.116%); 9286/273639 objects misplaced (3.394%); 22 MiB/s, 347 objects/s recovering 2024-03-24T12:38:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 47724/277956 objects degraded (17.170%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:55.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:55 smithi029 ceph-mon[29919]: osdmap e242: 8 total, 7 up, 7 in 2024-03-24T12:38:55.466 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:55.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:55.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:55.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:55.591 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:55.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:55.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:55.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:55 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:38:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:55 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:38:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:55 smithi190 ceph-mon[33748]: osdmap e241: 8 total, 7 up, 7 in 2024-03-24T12:38:55.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:55 smithi190 ceph-mon[33748]: pgmap v525: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.6 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 951 op/s; 46837/273639 objects degraded (17.116%); 9286/273639 objects misplaced (3.394%); 22 MiB/s, 347 objects/s recovering 2024-03-24T12:38:55.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:55 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 47724/277956 objects degraded (17.170%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:38:55.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:55 smithi190 ceph-mon[33748]: osdmap e242: 8 total, 7 up, 7 in 2024-03-24T12:38:55.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:55.748 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:55.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:55.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:55.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:55.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:55.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:55.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:56.015 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:56.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:56.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:56.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:56.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:56.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:56.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:56.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:56.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:56.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:56.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:56.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:56.418 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:56.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:56.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:56 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:38:56.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:56 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:38:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:56 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:38:56.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:56 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:38:56.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:56.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:56.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:56.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:56 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:38:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:56 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:38:56.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:56.688 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:56.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:56.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:56.829 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:56.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:56.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:56.964 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:56.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:57.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:57.093 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:57.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:57.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:57.195 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:57.218 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:57.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:57.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:57.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:57.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:57.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:57.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:57.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:57.471 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:57.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:57.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:57.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:38:57.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:57.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:57.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:38:57.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:57 smithi029 ceph-mon[29919]: pgmap v527: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 45497/268263 objects degraded (16.960%); 8459/268263 objects misplaced (3.153%); 26 MiB/s, 417 objects/s recovering 2024-03-24T12:38:57.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:57 smithi029 ceph-mon[41500]: pgmap v527: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 45497/268263 objects degraded (16.960%); 8459/268263 objects misplaced (3.153%); 26 MiB/s, 417 objects/s recovering 2024-03-24T12:38:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:57 smithi190 ceph-mon[33748]: pgmap v527: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.04k op/s; 45497/268263 objects degraded (16.960%); 8459/268263 objects misplaced (3.153%); 26 MiB/s, 417 objects/s recovering 2024-03-24T12:38:57.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:57.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:57.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:57.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:57.862 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:57.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:57.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:38:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:38:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:38:57.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:57.987 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:57.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:58.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:58.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:58.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:58.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:58.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:58.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:58.253 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:58.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:58.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:58.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:58.383 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:58.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:58.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:58 smithi190 ceph-mon[33748]: osdmap e243: 8 total, 7 up, 7 in 2024-03-24T12:38:58.384 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:38:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:38:58.484 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:38:58.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:58.524 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:58.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:38:58.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:58.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:58.660 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:58.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:58.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:58 smithi029 ceph-mon[29919]: osdmap e243: 8 total, 7 up, 7 in 2024-03-24T12:38:58.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:58 smithi029 ceph-mon[41500]: osdmap e243: 8 total, 7 up, 7 in 2024-03-24T12:38:58.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:58.804 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:58.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:58.905 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:58.944 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:58.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:58.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:59.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:38:59.070 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:59.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:59.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:38:59.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:59.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:59.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:59.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:38:59.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:59.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:59.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:59.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:59.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:38:59.449 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:59.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:38:59.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:59.595 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:59.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:59.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:59 smithi190 ceph-mon[33748]: osdmap e244: 8 total, 7 up, 7 in 2024-03-24T12:38:59.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:38:59 smithi190 ceph-mon[33748]: pgmap v530: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 853 op/s; 45045/265491 objects degraded (16.967%); 8459/265491 objects misplaced (3.186%); 21 MiB/s, 328 objects/s recovering 2024-03-24T12:38:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:59 smithi029 ceph-mon[29919]: osdmap e244: 8 total, 7 up, 7 in 2024-03-24T12:38:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:38:59 smithi029 ceph-mon[29919]: pgmap v530: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 853 op/s; 45045/265491 objects degraded (16.967%); 8459/265491 objects misplaced (3.186%); 21 MiB/s, 328 objects/s recovering 2024-03-24T12:38:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:38:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:59 smithi029 ceph-mon[41500]: osdmap e244: 8 total, 7 up, 7 in 2024-03-24T12:38:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:38:59 smithi029 ceph-mon[41500]: pgmap v530: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 21 active+clean+remapped, 86 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.4 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 853 op/s; 45045/265491 objects degraded (16.967%); 8459/265491 objects misplaced (3.186%); 21 MiB/s, 328 objects/s recovering 2024-03-24T12:38:59.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:38:59.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:59.721 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:59.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:38:59.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:38:59.862 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:38:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:59.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:38:59.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:38:59.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:38:59.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:38:59.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:00.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:00.120 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:00.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:00.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:00.248 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:00.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:00.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:00.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:00.389 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:00.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:00.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:00.491 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:00.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:00.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:00.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:00.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:00 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 45045/265491 objects degraded (16.967%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:00.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:00.655 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:00.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:00.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:00 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 45045/265491 objects degraded (16.967%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:00.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:00 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 45045/265491 objects degraded (16.967%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:00.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:00.799 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:00.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:00.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:00.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:00.928 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:00.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:01.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:01.053 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:01.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:01.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:01.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:01.180 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:01.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:01.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:01.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:01.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:01.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:01.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:01.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:01.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:01.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:01.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:01.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:01.565 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:01.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:01.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:01 smithi190 ceph-mon[33748]: pgmap v531: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 21 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 788 op/s; 44185/259446 objects degraded (17.031%); 7231/259446 objects misplaced (2.787%); 23 MiB/s, 373 objects/s recovering 2024-03-24T12:39:01.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:01.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:01 smithi029 ceph-mon[41500]: pgmap v531: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 21 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 788 op/s; 44185/259446 objects degraded (17.031%); 7231/259446 objects misplaced (2.787%); 23 MiB/s, 373 objects/s recovering 2024-03-24T12:39:01.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:01 smithi029 ceph-mon[29919]: pgmap v531: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 21 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.3 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 788 op/s; 44185/259446 objects degraded (17.031%); 7231/259446 objects misplaced (2.787%); 23 MiB/s, 373 objects/s recovering 2024-03-24T12:39:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:01.708 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:01.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:01.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:01.845 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:01.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:01.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:01.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:01.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:02.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:02.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:02.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:02.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:02.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:02.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:02.239 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:02.340 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:02.366 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:02.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:02.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:02.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:02.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:02.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:02.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:02.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:02.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:02.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:02.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:02.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:02.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:02.767 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:02.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:02.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:02.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:02.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:02.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:02.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:03.037 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:03.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:03.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:03.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:03.178 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:03.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:03.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:03.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:03.311 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:03.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:03.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:03.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:03.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:03.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:03.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:03.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:03.581 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:03.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:03.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:03.712 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:03.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:03.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:03.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:03.843 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:03.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:03.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:03.970 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:03.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:04.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:04.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:04.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:03 smithi190 ceph-mon[33748]: pgmap v532: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 21 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 710 op/s; 42473/254217 objects degraded (16.707%); 6953/254217 objects misplaced (2.735%); 20 MiB/s, 316 objects/s recovering 2024-03-24T12:39:04.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:04.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:03 smithi029 ceph-mon[29919]: pgmap v532: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 21 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 710 op/s; 42473/254217 objects degraded (16.707%); 6953/254217 objects misplaced (2.735%); 20 MiB/s, 316 objects/s recovering 2024-03-24T12:39:04.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:04.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:03 smithi029 ceph-mon[41500]: pgmap v532: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 21 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.2 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 710 op/s; 42473/254217 objects degraded (16.707%); 6953/254217 objects misplaced (2.735%); 20 MiB/s, 316 objects/s recovering 2024-03-24T12:39:04.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:04.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:04.225 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:04.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:04.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:04.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:04.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:04.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:04.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:04.480 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:04.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:04.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:04.608 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:04.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:04.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:04.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:04.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:04.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:39:04.921 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:04.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:04.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:04.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:04 smithi029 ceph-mon[29919]: osdmap e245: 8 total, 7 up, 7 in 2024-03-24T12:39:04.995 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:04 smithi029 ceph-mon[41500]: osdmap e245: 8 total, 7 up, 7 in 2024-03-24T12:39:05.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:05.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:05.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:05.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:04 smithi190 ceph-mon[33748]: osdmap e245: 8 total, 7 up, 7 in 2024-03-24T12:39:05.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:05.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:05.205 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:05.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:05.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:05.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:05.347 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:05.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:05.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:05.495 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:05.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:05.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:05.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:05.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:05.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:05.767 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:05.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:05.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:05.908 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:05.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:05.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:05 smithi029 ceph-mon[41500]: pgmap v534: 148 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 21 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 796 op/s; 41709/250395 objects degraded (16.657%); 6071/250395 objects misplaced (2.425%); 22 MiB/s, 358 objects/s recovering 2024-03-24T12:39:05.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:05 smithi029 ceph-mon[41500]: osdmap e246: 8 total, 7 up, 7 in 2024-03-24T12:39:05.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:05 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 41709/250395 objects degraded (16.657%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:05.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:05 smithi029 ceph-mon[29919]: pgmap v534: 148 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 21 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 796 op/s; 41709/250395 objects degraded (16.657%); 6071/250395 objects misplaced (2.425%); 22 MiB/s, 358 objects/s recovering 2024-03-24T12:39:05.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:05 smithi029 ceph-mon[29919]: osdmap e246: 8 total, 7 up, 7 in 2024-03-24T12:39:05.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:05 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 41709/250395 objects degraded (16.657%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:06.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:06.052 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:06.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:06.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:06.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:05 smithi190 ceph-mon[33748]: pgmap v534: 148 pgs: 1 active+remapped+backfilling, 1 active+undersized+degraded+remapped+backfilling, 21 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 796 op/s; 41709/250395 objects degraded (16.657%); 6071/250395 objects misplaced (2.425%); 22 MiB/s, 358 objects/s recovering 2024-03-24T12:39:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:05 smithi190 ceph-mon[33748]: osdmap e246: 8 total, 7 up, 7 in 2024-03-24T12:39:06.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:05 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 41709/250395 objects degraded (16.657%), 17 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:06.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:06.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:06.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:06.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:06.191 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:39:06.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:06.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:06.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:06.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:06.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:06.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:06.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:06.596 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:06.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:06.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:06.698 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:06.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:06.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:06.764 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.1 primary-affinity to 0.575782 (8377342) 2024-03-24T12:39:06.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:06.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:06.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:06.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:06.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:07.056 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:07.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:07.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:06 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1338055353' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5757817700508219}]: dispatch 2024-03-24T12:39:07.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:06 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5757817700508219}]: dispatch 2024-03-24T12:39:07.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:07.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:06 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1338055353' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5757817700508219}]: dispatch 2024-03-24T12:39:07.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:06 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5757817700508219}]: dispatch 2024-03-24T12:39:07.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:06 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1338055353' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5757817700508219}]: dispatch 2024-03-24T12:39:07.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:06 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5757817700508219}]: dispatch 2024-03-24T12:39:07.182 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:07.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:07.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:07.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:07.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:07.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:07.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:07.434 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:07.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:07.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:07.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:07.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:07.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:07.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:07.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:39:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:39:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:39:07.684 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:07.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:07.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:07.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:07.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:07.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:07.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:07.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:07.941 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:07.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:07.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:08.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:08.067 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:08.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:08.114 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:07 smithi190 ceph-mon[33748]: pgmap v536: 148 pgs: 1 active+remapped+backfilling, 22 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 853 op/s; 40691/245019 objects degraded (16.607%); 5777/245019 objects misplaced (2.358%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:39:08.114 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:07 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5757817700508219}]': finished 2024-03-24T12:39:08.114 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:07 smithi190 ceph-mon[33748]: osdmap e247: 8 total, 7 up, 7 in 2024-03-24T12:39:08.114 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:07 smithi190 ceph-mon[33748]: osdmap e248: 8 total, 7 up, 7 in 2024-03-24T12:39:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:07 smithi029 ceph-mon[41500]: pgmap v536: 148 pgs: 1 active+remapped+backfilling, 22 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 853 op/s; 40691/245019 objects degraded (16.607%); 5777/245019 objects misplaced (2.358%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:39:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:07 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5757817700508219}]': finished 2024-03-24T12:39:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:07 smithi029 ceph-mon[41500]: osdmap e247: 8 total, 7 up, 7 in 2024-03-24T12:39:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:07 smithi029 ceph-mon[41500]: osdmap e248: 8 total, 7 up, 7 in 2024-03-24T12:39:08.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:07 smithi029 ceph-mon[29919]: pgmap v536: 148 pgs: 1 active+remapped+backfilling, 22 active+clean+remapped, 87 active+clean, 22 active+undersized, 16 active+undersized+degraded; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 853 op/s; 40691/245019 objects degraded (16.607%); 5777/245019 objects misplaced (2.358%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:39:08.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:07 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5757817700508219}]': finished 2024-03-24T12:39:08.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:07 smithi029 ceph-mon[29919]: osdmap e247: 8 total, 7 up, 7 in 2024-03-24T12:39:08.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:07 smithi029 ceph-mon[29919]: osdmap e248: 8 total, 7 up, 7 in 2024-03-24T12:39:08.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:08.192 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:08.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:08.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:08.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:08.319 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:08.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:08.385 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:39:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:39:08.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:08.443 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:08.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:08.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:08.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:08.569 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:08.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:08.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:08.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:08.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:08.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:08.834 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:08.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:08.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:08.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:08.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:09.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:09.119 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:09.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:09.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:09.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:09.264 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:09.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:09.366 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:09.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:09.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:09.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:09.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:09.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:09.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:09.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:09.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:09.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:09.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:09.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:09.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:09.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:09.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:09.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:09.945 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:09.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:10.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:10.084 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:10.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:10.098 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:39:10.098 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:10.098 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:09 smithi029 ceph-mon[41500]: pgmap v539: 148 pgs: 4 peering, 1 active+remapped+backfilling, 22 active+clean+remapped, 86 active+clean, 20 active+undersized, 15 active+undersized+degraded; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 850 op/s; 35148/241635 objects degraded (14.546%); 5777/241635 objects misplaced (2.391%); 13 MiB/s, 209 objects/s recovering 2024-03-24T12:39:10.098 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:09 smithi029 ceph-mon[41500]: osdmap e249: 8 total, 7 up, 7 in 2024-03-24T12:39:10.099 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:09 smithi029 ceph-mon[41500]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-24T12:39:10.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:10.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:09 smithi029 ceph-mon[29919]: pgmap v539: 148 pgs: 4 peering, 1 active+remapped+backfilling, 22 active+clean+remapped, 86 active+clean, 20 active+undersized, 15 active+undersized+degraded; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 850 op/s; 35148/241635 objects degraded (14.546%); 5777/241635 objects misplaced (2.391%); 13 MiB/s, 209 objects/s recovering 2024-03-24T12:39:10.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:09 smithi029 ceph-mon[29919]: osdmap e249: 8 total, 7 up, 7 in 2024-03-24T12:39:10.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:09 smithi029 ceph-mon[29919]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-24T12:39:10.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:10.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:09 smithi190 ceph-mon[33748]: pgmap v539: 148 pgs: 4 peering, 1 active+remapped+backfilling, 22 active+clean+remapped, 86 active+clean, 20 active+undersized, 15 active+undersized+degraded; 4.9 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 850 op/s; 35148/241635 objects degraded (14.546%); 5777/241635 objects misplaced (2.391%); 13 MiB/s, 209 objects/s recovering 2024-03-24T12:39:10.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:09 smithi190 ceph-mon[33748]: osdmap e249: 8 total, 7 up, 7 in 2024-03-24T12:39:10.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:09 smithi190 ceph-mon[33748]: Health check failed: Reduced data availability: 3 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2024-03-24T12:39:10.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:10.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:10.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:10.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:10.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:10.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:10.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:10.496 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:10.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:10.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:10.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:10.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:10.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:10.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:10.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:10.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:10.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:10.899 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:10.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:10.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:10.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4045051743' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:39:10.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:39:10.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 35148/241635 objects degraded (14.546%), 15 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:10.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:39:10.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:39:10.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[29919]: osdmap e250: 8 total, 7 up, 7 in 2024-03-24T12:39:10.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4045051743' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:39:10.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:39:10.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 35148/241635 objects degraded (14.546%), 15 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:10.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:39:10.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:39:10.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:10 smithi029 ceph-mon[41500]: osdmap e250: 8 total, 7 up, 7 in 2024-03-24T12:39:11.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:11.038 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:11.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:11.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:10 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4045051743' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:39:11.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:10 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:39:11.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:10 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 35148/241635 objects degraded (14.546%), 15 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:11.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:10 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:39:11.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:10 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:39:11.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:10 smithi190 ceph-mon[33748]: osdmap e250: 8 total, 7 up, 7 in 2024-03-24T12:39:11.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:11.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:11.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:11.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:11.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:11.398 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:11.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:11.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:11.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:11.526 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:11.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:11.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:11.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:11.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:11.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:11.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:11.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:11.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:11.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:11.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:11.928 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:11.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:11.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:11.961 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:39:11.961 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:39:12.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:12.067 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:12.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:12.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:12.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:12.213 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:12.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:12.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:12.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:12.352 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:12.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:12.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:12.378 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:12 smithi029 ceph-mon[41500]: pgmap v542: 148 pgs: 4 peering, 22 active+clean+remapped, 87 active+clean, 20 active+undersized, 15 active+undersized+degraded; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 749 op/s; 34516/236028 objects degraded (14.624%); 5103/236028 objects misplaced (2.162%); 10 MiB/s, 164 objects/s recovering 2024-03-24T12:39:12.378 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:12 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/294058692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:39:12.379 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:12 smithi029 ceph-mon[29919]: pgmap v542: 148 pgs: 4 peering, 22 active+clean+remapped, 87 active+clean, 20 active+undersized, 15 active+undersized+degraded; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 749 op/s; 34516/236028 objects degraded (14.624%); 5103/236028 objects misplaced (2.162%); 10 MiB/s, 164 objects/s recovering 2024-03-24T12:39:12.379 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:12 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/294058692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:39:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:12 smithi190 ceph-mon[33748]: pgmap v542: 148 pgs: 4 peering, 22 active+clean+remapped, 87 active+clean, 20 active+undersized, 15 active+undersized+degraded; 4.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 749 op/s; 34516/236028 objects degraded (14.624%); 5103/236028 objects misplaced (2.162%); 10 MiB/s, 164 objects/s recovering 2024-03-24T12:39:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:12 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/294058692' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:39:12.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:12.490 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:12.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:12.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:12.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:12.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:12.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:12.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:12.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:12.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:12.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:12.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:12.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:12.876 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:12.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:12.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:12.883 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7] out_osds: [6] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:39:12.883 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:39:12.883 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-03-24T12:39:12.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd in 6 2024-03-24T12:39:12.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:13.004 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:13.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:13.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:13.143 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:13.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:13.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:13.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:13.273 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:13.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:13.375 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:13.406 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:13.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:13.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:13.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:13 smithi029 ceph-mon[41500]: pgmap v543: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 910 op/s; 37925/228855 objects degraded (16.572%); 4824/228855 objects misplaced (2.108%); 7.0 MiB/s, 111 objects/s recovering 2024-03-24T12:39:13.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:13 smithi029 ceph-mon[41500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-24T12:39:13.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:13 smithi029 ceph-mon[29919]: pgmap v543: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 910 op/s; 37925/228855 objects degraded (16.572%); 4824/228855 objects misplaced (2.108%); 7.0 MiB/s, 111 objects/s recovering 2024-03-24T12:39:13.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:13 smithi029 ceph-mon[29919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-24T12:39:13.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:13.536 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:39:13.536 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:39:13.536 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:39:13.536 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:39:13.536 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:39:13.536 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:39:13.536 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:39:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:39:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:39:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:39:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:39:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:39:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:39:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:39:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:39:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:13.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:13.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:13 smithi190 ceph-mon[33748]: pgmap v543: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.6 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 910 op/s; 37925/228855 objects degraded (16.572%); 4824/228855 objects misplaced (2.108%); 7.0 MiB/s, 111 objects/s recovering 2024-03-24T12:39:13.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:13 smithi190 ceph-mon[33748]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 3 pgs peering) 2024-03-24T12:39:13.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:13.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:13.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:13.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:13.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:13.822 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:13.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:13.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:13.950 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:13.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:14.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:14.100 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:14.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:14.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:14.229 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:14.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:14.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:14.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:14.368 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:14.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:14.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:14.517 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:14.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:14.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:14.539 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:39:14.539 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:39:14.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:14.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:14.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:14.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:14.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:14.798 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:14.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:14.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:14.900 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:14.927 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:14.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:14.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T12:39:15.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:15.040 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:39:15.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:39:15.082 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:15.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:15.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:15.241 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:15.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:15.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:15.261 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:39:15.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:15.398 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:15.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:15.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:15.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:15.543 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:15.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:15.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:15.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:15.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:15.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:15.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:15.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:15.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:15 smithi029 ceph-mon[29919]: pgmap v544: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 591 B/s rd, 0 B/s wr, 843 op/s; 37144/224118 objects degraded (16.573%); 4824/224118 objects misplaced (2.152%); 5.9 MiB/s, 95 objects/s recovering 2024-03-24T12:39:15.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:15 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 37144/224118 objects degraded (16.573%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:15.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:15 smithi029 ceph-mon[41500]: pgmap v544: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 591 B/s rd, 0 B/s wr, 843 op/s; 37144/224118 objects degraded (16.573%); 4824/224118 objects misplaced (2.152%); 5.9 MiB/s, 95 objects/s recovering 2024-03-24T12:39:15.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:15 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 37144/224118 objects degraded (16.573%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:15.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:15.977 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:15.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:16.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:16.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:16.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:15 smithi190 ceph-mon[33748]: pgmap v544: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 591 B/s rd, 0 B/s wr, 843 op/s; 37144/224118 objects degraded (16.573%); 4824/224118 objects misplaced (2.152%); 5.9 MiB/s, 95 objects/s recovering 2024-03-24T12:39:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:15 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 37144/224118 objects degraded (16.573%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:16.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:16.246 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:16.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:16.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:16.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:16.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:16.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:16.505 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:16.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:16.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:39:16.580 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:39:16.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:16.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:16.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:16.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:16.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:16.798 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:16.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:16.899 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:16.942 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:16.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:16.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:17.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:17.086 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:17.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:17.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:17.187 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:17.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:17.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:17.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:17.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:17.348 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:17.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:17.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:17.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:17.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:17.547 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:39:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:39:17.547 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-03-24T12:39:17.548 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:39:17.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:17 smithi190 ceph-mon[33748]: pgmap v545: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.07k op/s; 35987/215970 objects degraded (16.663%); 4450/215970 objects misplaced (2.060%); 5.1 MiB/s, 82 objects/s recovering 2024-03-24T12:39:17.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:39:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:39:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:39:17.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:17 smithi029 ceph-mon[41500]: pgmap v545: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.07k op/s; 35987/215970 objects degraded (16.663%); 4450/215970 objects misplaced (2.060%); 5.1 MiB/s, 82 objects/s recovering 2024-03-24T12:39:17.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:17 smithi029 ceph-mon[29919]: pgmap v545: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.07k op/s; 35987/215970 objects degraded (16.663%); 4450/215970 objects misplaced (2.060%); 5.1 MiB/s, 82 objects/s recovering 2024-03-24T12:39:17.949 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:17.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:18.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:18.074 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:18.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:18.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:18.202 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:39:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:39:18.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:18.217 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:18.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:18.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T12:39:18.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:18.349 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:18.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:18.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:18.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:18.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:18.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:18.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:18.533 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:39:18.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:18.621 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:18.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:18.723 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:18.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:18.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:18.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:18.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:18.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:18.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:18.955 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:18.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:19.000 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:19.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:19.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:19.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:19.132 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:19.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:19.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:19.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:19.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:19.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:19.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:19.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:19.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:19.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:19.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:19.525 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:19.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:19.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:19.627 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:19.651 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:19.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:19.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:19.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:19.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:19.901 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:19.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:19.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:19.995 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:19.995 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:19 smithi029 ceph-mon[41500]: pgmap v546: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 928 B/s rd, 0 B/s wr, 1.06k op/s; 34808/210099 objects degraded (16.567%); 4450/210099 objects misplaced (2.118%); 4.1 MiB/s, 66 objects/s recovering 2024-03-24T12:39:19.997 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:19.998 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:19 smithi029 ceph-mon[29919]: pgmap v546: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 928 B/s rd, 0 B/s wr, 1.06k op/s; 34808/210099 objects degraded (16.567%); 4450/210099 objects misplaced (2.118%); 4.1 MiB/s, 66 objects/s recovering 2024-03-24T12:39:20.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:20.079 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:20.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:20.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:20.111 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:20.111 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:19 smithi190 ceph-mon[33748]: pgmap v546: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.3 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 928 B/s rd, 0 B/s wr, 1.06k op/s; 34808/210099 objects degraded (16.567%); 4450/210099 objects misplaced (2.118%); 4.1 MiB/s, 66 objects/s recovering 2024-03-24T12:39:20.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:20.216 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:20.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:20.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:20.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:20.358 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:20.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:20.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:20.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:20.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:20.539 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:39:20.540 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:39:20.540 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-03-24T12:39:20.540 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-03-24T12:39:20.540 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:39:20.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:20.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:20.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:20.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:20.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:20.752 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:20.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:20.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:20.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:20.897 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:20.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:20.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:21.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:21.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:21.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:21.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:20 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 34808/210099 objects degraded (16.567%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:21.167 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:20 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 34808/210099 objects degraded (16.567%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:20 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 34808/210099 objects degraded (16.567%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:21.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:21.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:21.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:21.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:21.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:21.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:21.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:21.422 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:21.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:21.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:21.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T12:39:21.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:21.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:21.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:21.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:21.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:21.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:21.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:21.735 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:39:21.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:21.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:21.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:21.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:21.946 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:21.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:21.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:22.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:22.084 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:22.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:22.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:22 smithi190 ceph-mon[33748]: pgmap v547: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 967 B/s rd, 0 B/s wr, 1.09k op/s; 33725/201297 objects degraded (16.754%); 4450/201297 objects misplaced (2.211%) 2024-03-24T12:39:22.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:22.229 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:22.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:22.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:22.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:22.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:22 smithi029 ceph-mon[41500]: pgmap v547: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 967 B/s rd, 0 B/s wr, 1.09k op/s; 33725/201297 objects degraded (16.754%); 4450/201297 objects misplaced (2.211%) 2024-03-24T12:39:22.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:22 smithi029 ceph-mon[29919]: pgmap v547: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 4.1 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 967 B/s rd, 0 B/s wr, 1.09k op/s; 33725/201297 objects degraded (16.754%); 4450/201297 objects misplaced (2.211%) 2024-03-24T12:39:22.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:22.500 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:22.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:22.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:22.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:22.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:22.736 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:22.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:22.776 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:22.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:22.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:22.913 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:22.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:22.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:23.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:23.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:23.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:23.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:23.165 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:23.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:23.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:23.290 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:23.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:23.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:23.416 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:23.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:23.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:23.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:23.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:23.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:23.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:23.674 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:23.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:23.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:23.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:23.816 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:23.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:23.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:23.956 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:23.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:24.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:24.081 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:24.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:24.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:24.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:24.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:24.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:24.339 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:24.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:24 smithi190 ceph-mon[33748]: pgmap v548: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 32037/192945 objects degraded (16.604%); 4084/192945 objects misplaced (2.117%) 2024-03-24T12:39:24.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:24 smithi029 ceph-mon[41500]: pgmap v548: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 32037/192945 objects degraded (16.604%); 4084/192945 objects misplaced (2.117%) 2024-03-24T12:39:24.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:24 smithi029 ceph-mon[29919]: pgmap v548: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.9 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.20k op/s; 32037/192945 objects degraded (16.604%); 4084/192945 objects misplaced (2.117%) 2024-03-24T12:39:24.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:24.470 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:24.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:24.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:24.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:24.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:24.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:24.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:24.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:24.720 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:24.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:24.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:24.847 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:24.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:24.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:24.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:24.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:24.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:25.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:25.118 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:25.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:25.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:25.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:25.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:25.260 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:25.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:25.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:25 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 32037/192945 objects degraded (16.604%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:25.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:25.397 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:25.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:25.399 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:39:25.399 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-03-24T12:39:25.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-03-24T12:39:25.399 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:39:25.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.399 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:129905 7.16 7:6a89bbaf:::benchmark_data_smithi029_84282_object29142:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.400 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.044494+0000", 2024-03-24T12:39:25.400 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 530.35481554199998, 2024-03-24T12:39:25.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91608335500000004, 2024-03-24T12:39:25.400 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.400 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.400 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.400 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.400 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.401 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 129905 2024-03-24T12:39:25.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.401 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044494+0000", 2024-03-24T12:39:25.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044494+0000", 2024-03-24T12:39:25.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044496+0000", 2024-03-24T12:39:25.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.451e-06 2024-03-24T12:39:25.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044502+0000", 2024-03-24T12:39:25.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4180000000000002e-06 2024-03-24T12:39:25.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044517+0000", 2024-03-24T12:39:25.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4985e-05 2024-03-24T12:39:25.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044525+0000", 2024-03-24T12:39:25.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6510000000000001e-06 2024-03-24T12:39:25.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.406 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044582+0000", 2024-03-24T12:39:25.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6582999999999997e-05 2024-03-24T12:39:25.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.044591+0000", 2024-03-24T12:39:25.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9679999999999995e-06 2024-03-24T12:39:25.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.954913+0000", 2024-03-24T12:39:25.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91032239999999998 2024-03-24T12:39:25.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.955039+0000", 2024-03-24T12:39:25.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000126363 2024-03-24T12:39:25.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.955424+0000", 2024-03-24T12:39:25.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038441700000000002 2024-03-24T12:39:25.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960392+0000", 2024-03-24T12:39:25.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049685270000000004 2024-03-24T12:39:25.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960405+0000", 2024-03-24T12:39:25.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2978e-05 2024-03-24T12:39:25.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960501+0000", 2024-03-24T12:39:25.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5444000000000003e-05 2024-03-24T12:39:25.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.413 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.413 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960548+0000", 2024-03-24T12:39:25.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7521999999999997e-05 2024-03-24T12:39:25.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960578+0000", 2024-03-24T12:39:25.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9648000000000001e-05 2024-03-24T12:39:25.414 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.414 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.414 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.415 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:129941 7.16 7:6aa68b6f:::benchmark_data_smithi029_84282_object29178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.415 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.089151+0000", 2024-03-24T12:39:25.415 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 530.31015876599997, 2024-03-24T12:39:25.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.87456397399999997, 2024-03-24T12:39:25.415 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.416 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.416 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.416 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.416 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.416 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 129941 2024-03-24T12:39:25.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.417 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089151+0000", 2024-03-24T12:39:25.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089151+0000", 2024-03-24T12:39:25.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089152+0000", 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.176e-06 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089157+0000", 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0440000000000003e-06 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089168+0000", 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0969e-05 2024-03-24T12:39:25.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089176+0000", 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7910000000000005e-06 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089229+0000", 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2738000000000002e-05 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.089238+0000", 2024-03-24T12:39:25.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8259999999999999e-06 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.957558+0000", 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.86831984799999995 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.957655+0000", 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7683000000000006e-05 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.957971+0000", 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031544199999999998 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960434+0000", 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024637130000000002 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960672+0000", 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000237376 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963635+0000", 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029635180000000001 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963682+0000", 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6480999999999998e-05 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963715+0000", 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3368999999999998e-05 2024-03-24T12:39:25.423 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:129994 7.16 7:6e06b474:::benchmark_data_smithi029_84282_object29231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.149614+0000", 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 530.249695567, 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82120363100000004, 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.424 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 129994 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149614+0000", 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149614+0000", 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149616+0000", 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7740000000000001e-06 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149624+0000", 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7859999999999997e-06 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149640+0000", 2024-03-24T12:39:25.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5509e-05 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149648+0000", 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8289999999999997e-06 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149723+0000", 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5117999999999995e-05 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.149736+0000", 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2721000000000001e-05 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.959238+0000", 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.80950209900000003 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.959331+0000", 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2386999999999995e-05 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.959505+0000", 2024-03-24T12:39:25.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017448400000000001 2024-03-24T12:39:25.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963533+0000", 2024-03-24T12:39:25.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040280330000000003 2024-03-24T12:39:25.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963588+0000", 2024-03-24T12:39:25.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5074000000000002e-05 2024-03-24T12:39:25.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970720+0000", 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0071316569999999996 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970769+0000", 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9456999999999998e-05 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970818+0000", 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8702999999999998e-05 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.431 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130009 7.16 7:6b59aaed:::benchmark_data_smithi029_84282_object29246:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.161268+0000", 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 530.23804180699995, 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.82047960200000003, 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130009 2024-03-24T12:39:25.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161268+0000", 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161268+0000", 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161269+0000", 2024-03-24T12:39:25.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.232e-06 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161275+0000", 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1590000000000001e-06 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161285+0000", 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0713999999999999e-05 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161292+0000", 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6989999999999997e-06 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161354+0000", 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2483999999999993e-05 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.161368+0000", 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3234000000000001e-05 2024-03-24T12:39:25.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960723+0000", 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.79935483699999998 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.960812+0000", 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9931000000000003e-05 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.961009+0000", 2024-03-24T12:39:25.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019644500000000001 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963724+0000", 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027152299999999999 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967454+0000", 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003729675 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981661+0000", 2024-03-24T12:39:25.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014207054 2024-03-24T12:39:25.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981698+0000", 2024-03-24T12:39:25.439 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:25 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 32037/192945 objects degraded (16.604%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:25.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:25 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 32037/192945 objects degraded (16.604%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:25.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6838999999999997e-05 2024-03-24T12:39:25.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981748+0000", 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0068999999999998e-05 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130018 7.16 7:6d78b553:::benchmark_data_smithi029_84282_object29255:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.175266+0000", 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 530.22404411800005, 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.80664670699999996, 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.441 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130018 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175266+0000", 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175266+0000", 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175268+0000", 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7400000000000001e-06 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175274+0000", 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0329999999999997e-06 2024-03-24T12:39:25.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175291+0000", 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7430000000000001e-05 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175301+0000", 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0370000000000001e-05 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175382+0000", 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0277000000000004e-05 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.175392+0000", 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0190999999999999e-05 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.962152+0000", 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.78675967000000002 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.962267+0000", 2024-03-24T12:39:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011578000000000001 2024-03-24T12:39:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.962502+0000", 2024-03-24T12:39:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023468 2024-03-24T12:39:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967413+0000", 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0049112950000000004 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967498+0000", 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4874000000000001e-05 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981793+0000", 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.014295112 2024-03-24T12:39:25.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981863+0000", 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9636999999999998e-05 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.981913+0000", 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9617999999999998e-05 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.448 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130039 7.16 7:6daa3191:::benchmark_data_smithi029_84282_object29276:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.202829+0000", 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 530.19648075999999, 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.77952253199999999, 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130039 2024-03-24T12:39:25.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202829+0000", 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202829+0000", 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202831+0000", 2024-03-24T12:39:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3060000000000001e-06 2024-03-24T12:39:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202835+0000", 2024-03-24T12:39:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7299999999999996e-06 2024-03-24T12:39:25.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202854+0000", 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8416000000000001e-05 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202862+0000", 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4500000000000004e-06 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202915+0000", 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3100999999999999e-05 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.202923+0000", 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1610000000000003e-06 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963748+0000", 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.76082477900000001 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.963868+0000", 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000119997 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.964095+0000", 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000226758 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967465+0000", 2024-03-24T12:39:25.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033697689999999999 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970870+0000", 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003405277 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982254+0000", 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011383926000000001 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.455 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982307+0000", 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3313000000000003e-05 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982352+0000", 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4549000000000003e-05 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.456 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130093 7.16 7:6e160392:::benchmark_data_smithi029_84282_object29330:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.269634+0000", 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 530.12967593999997, 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.71289628299999996, 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130093 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269634+0000", 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269634+0000", 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269635+0000", 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.378e-06 2024-03-24T12:39:25.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269641+0000", 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8540000000000002e-06 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269662+0000", 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0469e-05 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269670+0000", 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6309999999999992e-06 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269771+0000", 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100249 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.269781+0000", 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8670000000000006e-06 2024-03-24T12:39:25.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.965429+0000", 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.69564800500000001 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.965547+0000", 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000118021 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.965797+0000", 2024-03-24T12:39:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025022800000000002 2024-03-24T12:39:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970833+0000", 2024-03-24T12:39:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0050361690000000001 2024-03-24T12:39:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970904+0000", 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1487000000000002e-05 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982425+0000", 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011520829 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982486+0000", 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1149999999999996e-05 2024-03-24T12:39:25.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982530+0000", 2024-03-24T12:39:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3946000000000002e-05 2024-03-24T12:39:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.464 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130098 7.16 7:6aef2ee6:::benchmark_data_smithi029_84282_object29335:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.277429+0000", 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 530.12188134300004, 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.70522084100000004, 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130098 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277429+0000", 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277429+0000", 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277430+0000", 2024-03-24T12:39:25.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5600000000000001e-06 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277436+0000", 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5389999999999998e-06 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277447+0000", 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1333e-05 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277455+0000", 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4079999999999998e-06 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277513+0000", 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7728e-05 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.277522+0000", 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6309999999999992e-06 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967527+0000", 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.69000489499999995 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967630+0000", 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010367200000000001 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.967864+0000", 2024-03-24T12:39:25.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000233458 2024-03-24T12:39:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970914+0000", 2024-03-24T12:39:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003049997 2024-03-24T12:39:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970949+0000", 2024-03-24T12:39:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5073000000000001e-05 2024-03-24T12:39:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982572+0000", 2024-03-24T12:39:25.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011622937999999999 2024-03-24T12:39:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982620+0000", 2024-03-24T12:39:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7654999999999997e-05 2024-03-24T12:39:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.982649+0000", 2024-03-24T12:39:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9954000000000002e-05 2024-03-24T12:39:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130156 7.16 7:6f38da85:::benchmark_data_smithi029_84282_object29393:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.463174+0000", 2024-03-24T12:39:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 529.93613593400005, 2024-03-24T12:39:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.52157916000000004, 2024-03-24T12:39:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.474 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130156 2024-03-24T12:39:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463174+0000", 2024-03-24T12:39:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463174+0000", 2024-03-24T12:39:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463176+0000", 2024-03-24T12:39:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.53e-06 2024-03-24T12:39:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463183+0000", 2024-03-24T12:39:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3350000000000001e-06 2024-03-24T12:39:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463202+0000", 2024-03-24T12:39:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9423999999999999e-05 2024-03-24T12:39:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463214+0000", 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1616e-05 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463300+0000", 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6518000000000001e-05 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.463309+0000", 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3529999999999995e-06 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.969326+0000", 2024-03-24T12:39:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50601668300000002 2024-03-24T12:39:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.969419+0000", 2024-03-24T12:39:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3696000000000006e-05 2024-03-24T12:39:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.969625+0000", 2024-03-24T12:39:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000205451 2024-03-24T12:39:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970921+0000", 2024-03-24T12:39:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012960809999999999 2024-03-24T12:39:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.974161+0000", 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032402080000000001 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984642+0000", 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010480835000000001 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984699+0000", 2024-03-24T12:39:25.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7404999999999997e-05 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984753+0000", 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4024999999999999e-05 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.483 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130168 7.16 7:6d007d8d:::benchmark_data_smithi029_84282_object29405:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.474576+0000", 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 529.92473374799999, 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51029922000000005, 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130168 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474576+0000", 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474576+0000", 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474578+0000", 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8080000000000001e-06 2024-03-24T12:39:25.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474586+0000", 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5920000000000003e-06 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474605+0000", 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9602999999999998e-05 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474616+0000", 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.062e-05 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474684+0000", 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8435999999999993e-05 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.474694+0000", 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0186999999999999e-05 2024-03-24T12:39:25.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.970974+0000", 2024-03-24T12:39:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.496279313 2024-03-24T12:39:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.971054+0000", 2024-03-24T12:39:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0469000000000005e-05 2024-03-24T12:39:25.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.971241+0000", 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000186369 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.974173+0000", 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029318899999999999 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.977894+0000", 2024-03-24T12:39:25.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037216699999999998 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984795+0000", 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0069006240000000002 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984842+0000", 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6751e-05 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984875+0000", 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3887999999999998e-05 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130176 7.16 7:6ba24fa6:::benchmark_data_smithi029_84282_object29413:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.483707+0000", 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 529.91560331400001, 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.50131378900000001, 2024-03-24T12:39:25.491 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130176 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483707+0000", 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483707+0000", 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483708+0000", 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.251e-06 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483712+0000", 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5340000000000001e-06 2024-03-24T12:39:25.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483723+0000", 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0831e-05 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483730+0000", 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9789999999999996e-06 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483783+0000", 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2848e-05 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.483792+0000", 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4069999999999993e-06 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.972312+0000", 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.48852052499999998 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.972433+0000", 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000121291 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.972611+0000", 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017786000000000001 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.977913+0000", 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0053022870000000001 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.977983+0000", 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9985000000000002e-05 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984939+0000", 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006955363 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.984984+0000", 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4808000000000002e-05 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.985020+0000", 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6820000000000003e-05 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.498 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130190 7.16 7:693003cf:::benchmark_data_smithi029_84282_object29427:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.507558+0000", 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 529.89175206799996, 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.47759035900000002, 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130190 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.499 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507558+0000", 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507558+0000", 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507559+0000", 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.265e-06 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507564+0000", 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9370000000000003e-06 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507579+0000", 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4741e-05 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507587+0000", 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1650000000000006e-06 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507650+0000", 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3230000000000003e-05 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.507660+0000", 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5380000000000008e-06 2024-03-24T12:39:25.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.974217+0000", 2024-03-24T12:39:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.46655671999999998 2024-03-24T12:39:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.974340+0000", 2024-03-24T12:39:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012344600000000001 2024-03-24T12:39:25.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.974619+0000", 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027936399999999999 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.979433+0000", 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0048138870000000002 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.979476+0000", 2024-03-24T12:39:25.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2755e-05 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.985061+0000", 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055851820000000002 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.985103+0000", 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2043999999999999e-05 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.985148+0000", 2024-03-24T12:39:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5084999999999999e-05 2024-03-24T12:39:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130217 7.16 7:6a8e9ff5:::benchmark_data_smithi029_84282_object29454:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.554201+0000", 2024-03-24T12:39:25.506 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 529.84510871999998, 2024-03-24T12:39:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.435903606, 2024-03-24T12:39:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.507 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130217 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554201+0000", 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554201+0000", 2024-03-24T12:39:25.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554202+0000", 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5600000000000005e-07 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554206+0000", 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9650000000000002e-06 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554215+0000", 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4690000000000003e-06 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554223+0000", 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3440000000000003e-06 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554291+0000", 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7779000000000003e-05 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.554298+0000", 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9380000000000003e-06 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.976066+0000", 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.42176839199999999 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.976209+0000", 2024-03-24T12:39:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014275600000000001 2024-03-24T12:39:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.976477+0000", 2024-03-24T12:39:25.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000268189 2024-03-24T12:39:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.979488+0000", 2024-03-24T12:39:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030113319999999998 2024-03-24T12:39:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.979527+0000", 2024-03-24T12:39:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8349000000000002e-05 2024-03-24T12:39:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.989983+0000", 2024-03-24T12:39:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010456081000000001 2024-03-24T12:39:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.990059+0000", 2024-03-24T12:39:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6579e-05 2024-03-24T12:39:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.990105+0000", 2024-03-24T12:39:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5677000000000003e-05 2024-03-24T12:39:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:130228 7.16 7:68cad5ca:::benchmark_data_smithi029_84282_object29465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-03-24T12:39:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:30:35.577413+0000", 2024-03-24T12:39:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 529.821897272, 2024-03-24T12:39:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41282179299999999, 2024-03-24T12:39:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.518 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 130228 2024-03-24T12:39:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.519 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577413+0000", 2024-03-24T12:39:25.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577413+0000", 2024-03-24T12:39:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577414+0000", 2024-03-24T12:39:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.638e-06 2024-03-24T12:39:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577423+0000", 2024-03-24T12:39:25.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3750000000000003e-06 2024-03-24T12:39:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577443+0000", 2024-03-24T12:39:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9769e-05 2024-03-24T12:39:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577454+0000", 2024-03-24T12:39:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1328000000000001e-05 2024-03-24T12:39:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577554+0000", 2024-03-24T12:39:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100449 2024-03-24T12:39:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.577572+0000", 2024-03-24T12:39:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7306e-05 2024-03-24T12:39:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.978019+0000", 2024-03-24T12:39:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.40044787399999998 2024-03-24T12:39:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.978095+0000", 2024-03-24T12:39:25.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5640000000000001e-05 2024-03-24T12:39:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:39:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.978278+0000", 2024-03-24T12:39:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000182704 2024-03-24T12:39:25.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.979607+0000", 2024-03-24T12:39:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001329521 2024-03-24T12:39:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.980997+0000", 2024-03-24T12:39:25.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013895859999999999 2024-03-24T12:39:25.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.990145+0000", 2024-03-24T12:39:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0091478749999999998 2024-03-24T12:39:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.990197+0000", 2024-03-24T12:39:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2012000000000003e-05 2024-03-24T12:39:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:30:35.990235+0000", 2024-03-24T12:39:25.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7716000000000003e-05 2024-03-24T12:39:25.533 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.533 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.533 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:132160 8.2 8:461ed8a7:::benchmark_data_smithi029_108893_object18300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-03-24T12:39:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:34:46.032122+0000", 2024-03-24T12:39:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 279.36718793900002, 2024-03-24T12:39:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41847558200000001, 2024-03-24T12:39:25.534 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:39:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:39:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 132160 2024-03-24T12:39:25.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.032122+0000", 2024-03-24T12:39:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.032122+0000", 2024-03-24T12:39:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.032123+0000", 2024-03-24T12:39:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7899999999999998e-07 2024-03-24T12:39:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.032126+0000", 2024-03-24T12:39:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.399e-06 2024-03-24T12:39:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.032134+0000", 2024-03-24T12:39:25.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0209999999999999e-06 2024-03-24T12:39:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.032141+0000", 2024-03-24T12:39:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5760000000000002e-06 2024-03-24T12:39:25.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.034958+0000", 2024-03-24T12:39:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0028171889999999999 2024-03-24T12:39:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.035062+0000", 2024-03-24T12:39:25.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010458699999999999 2024-03-24T12:39:25.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:39:25.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.035172+0000", 2024-03-24T12:39:25.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000109854 2024-03-24T12:39:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.037072+0000", 2024-03-24T12:39:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001899523 2024-03-24T12:39:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.037122+0000", 2024-03-24T12:39:25.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9932999999999998e-05 2024-03-24T12:39:25.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.450493+0000", 2024-03-24T12:39:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.41337083400000002 2024-03-24T12:39:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.450549+0000", 2024-03-24T12:39:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6307999999999997e-05 2024-03-24T12:39:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:34:46.450598+0000", 2024-03-24T12:39:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8779000000000002e-05 2024-03-24T12:39:25.545 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:206532 8.4 8:29bf5401:::benchmark_data_smithi029_108893_object92672:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:39:25.546 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:35:57.699575+0000", 2024-03-24T12:39:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 207.699734764, 2024-03-24T12:39:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0692478439999999, 2024-03-24T12:39:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:39:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:39:25.547 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 206532 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699575+0000", 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699575+0000", 2024-03-24T12:39:25.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699577+0000", 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8309999999999999e-06 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699581+0000", 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3710000000000002e-06 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699593+0000", 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1155000000000001e-05 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699628+0000", 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5784000000000002e-05 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699697+0000", 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8514000000000002e-05 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699717+0000", 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0409000000000001e-05 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.750884+0000", 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0511667039999999 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.753108+0000", 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022241439999999999 2024-03-24T12:39:25.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2", 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.753333+0000", 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022524399999999999 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.762027+0000", 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0086935009999999993 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.768680+0000", 2024-03-24T12:39:25.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066534660000000002 2024-03-24T12:39:25.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.768724+0000", 2024-03-24T12:39:25.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3405999999999997e-05 2024-03-24T12:39:25.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:25.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.768823+0000", 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9314999999999994e-05 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:206552 8.2 8:4f176569:::benchmark_data_smithi029_108893_object92692:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:35:57.724703+0000", 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 207.67460664800001, 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.029992944, 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.555 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 206552 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.724703+0000", 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.724703+0000", 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.724705+0000", 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4500000000000001e-06 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.724711+0000", 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.088e-06 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.724725+0000", 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3658999999999999e-05 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.724733+0000", 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8349999999999993e-06 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.724804+0000", 2024-03-24T12:39:25.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0877999999999995e-05 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.724812+0000", 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8099999999999998e-06 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.747940+0000", 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0231275150000001 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.748071+0000", 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013127599999999999 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.748195+0000", 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012430500000000001 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.754520+0000", 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063244310000000001 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.754536+0000", 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6169999999999999e-05 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.754590+0000", 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3816000000000001e-05 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.754696+0000", 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106711 2024-03-24T12:39:25.561 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:206553 8.2 8:47618320:::benchmark_data_smithi029_108893_object92693:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:35:57.728057+0000", 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 207.67125252599999, 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.031451884, 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:39:25.562 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 206553 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.728057+0000", 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.728057+0000", 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.728058+0000", 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5600000000000005e-07 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.728062+0000", 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9929999999999997e-06 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.728072+0000", 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5929999999999995e-06 2024-03-24T12:39:25.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.728078+0000", 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2670000000000003e-06 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.728127+0000", 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.935e-05 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.728133+0000", 2024-03-24T12:39:25.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8819999999999998e-06 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.751237+0000", 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.023104104 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.751500+0000", 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026240299999999997 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0", 2024-03-24T12:39:25.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.751627+0000", 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012746400000000001 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.756974+0000", 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0053471869999999998 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.759420+0000", 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024457340000000002 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.759454+0000", 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3343000000000002e-05 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.759509+0000", 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5807999999999998e-05 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24344.0:2088 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 93828934256768 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:39:25.568 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:35:58.114932+0000", 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 207.28437822399999, 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.60251821500000002, 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24344", 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2088 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.114932+0000", 2024-03-24T12:39:25.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.114932+0000", 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.114934+0000", 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7400000000000001e-06 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.114941+0000", 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8509999999999999e-06 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.114956+0000", 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4378e-05 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.114965+0000", 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0580000000000004e-06 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.115036+0000", 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1273999999999996e-05 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.115049+0000", 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2527000000000001e-05 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.714335+0000", 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.599286651 2024-03-24T12:39:25.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.714972+0000", 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00063721700000000004 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.717450+0000", 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0024775190000000001 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24344.0:2096 5.e 5:7759931f:::notify.3:head [watch ping cookie 93828934256768 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:35:58.327437+0000", 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 207.071873294, 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38950282600000002, 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24344", 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 2096 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.327437+0000", 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.327437+0000", 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.327437+0000", 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0699999999999997e-07 2024-03-24T12:39:25.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.327441+0000", 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9140000000000001e-06 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.327448+0000", 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8020000000000003e-06 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.327457+0000", 2024-03-24T12:39:25.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9369999999999993e-06 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.327541+0000", 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4209999999999995e-05 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.327554+0000", 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3074000000000001e-05 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:25.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.716611+0000", 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.38905645300000002 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.716713+0000", 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010250400000000001 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.716940+0000", 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022632500000000001 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.578 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.579 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.579 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:25.579 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:25.579 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:39:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:25.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:25.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:25.706 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:25.733 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:25.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:25.835 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:25.863 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:25.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:25.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:25.965 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:25.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:25.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:25.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:26.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:26.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:26.119 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:26.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:26.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:26.249 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:26.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:26.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:26.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:26 smithi190 ceph-mon[33748]: pgmap v549: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s; 31420/189288 objects degraded (16.599%); 4084/189288 objects misplaced (2.158%) 2024-03-24T12:39:26.389 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:26.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:26 smithi029 ceph-mon[29919]: pgmap v549: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s; 31420/189288 objects degraded (16.599%); 4084/189288 objects misplaced (2.158%) 2024-03-24T12:39:26.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:26 smithi029 ceph-mon[41500]: pgmap v549: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.8 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.10k op/s; 31420/189288 objects degraded (16.599%); 4084/189288 objects misplaced (2.158%) 2024-03-24T12:39:26.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T12:39:26.490 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:26.537 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:26.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:26.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:26.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:26.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:26.726 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:39:26.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:26.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:26.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:26.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:26.922 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:26.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:27.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:27.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:27.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:27.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:27.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:27.179 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:27.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:27.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:27.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:27.304 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:27.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:27.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:27.377 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:27 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:39:27.378 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:27 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:39:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:27 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:39:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:27 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:39:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:27 smithi190 ceph-mon[33748]: osdmap e251: 8 total, 7 up, 7 in 2024-03-24T12:39:27.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:27 smithi190 ceph-mon[33748]: pgmap v551: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 30574/183336 objects degraded (16.676%); 3818/183336 objects misplaced (2.083%) 2024-03-24T12:39:27.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:27.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:27.434 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:27.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:27.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:27.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:27.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:27.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:27.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:27.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:27 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:39:27.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:27 smithi029 ceph-mon[41500]: osdmap e251: 8 total, 7 up, 7 in 2024-03-24T12:39:27.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:27 smithi029 ceph-mon[41500]: pgmap v551: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 30574/183336 objects degraded (16.676%); 3818/183336 objects misplaced (2.083%) 2024-03-24T12:39:27.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:27 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:39:27.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:27 smithi029 ceph-mon[29919]: osdmap e251: 8 total, 7 up, 7 in 2024-03-24T12:39:27.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:27 smithi029 ceph-mon[29919]: pgmap v551: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.09k op/s; 30574/183336 objects degraded (16.676%); 3818/183336 objects misplaced (2.083%) 2024-03-24T12:39:27.668 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:39:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:39:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:39:27.687 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:27.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:27.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:27.814 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:27.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:27.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:27.915 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:27.944 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:27.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:27.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:28.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:28.079 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:28.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:28.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:28.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:28.204 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:28.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:28.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:28.303 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:28 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:39:28.304 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:28 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:39:28.304 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:39:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:39:28.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:28.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:28.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:28.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:28.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:28 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:39:28.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:28 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:39:28.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:28 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:39:28.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:28 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:39:28.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:28.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:28.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:28.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:28.572 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:39:28.572 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:39:28.572 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-03-24T12:39:28.572 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:39:28.591 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:28.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:28.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:28.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:28.731 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:28.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:28.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:28.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:28.861 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:28.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:28.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:28.991 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:28.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:28.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:29.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:29.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:29.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:29.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:29.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:29.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:29.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T12:39:29.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:29.381 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:29.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:29.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:29.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:29.526 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:29.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:29.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:29.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:29 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:29.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:29 smithi190 ceph-mon[33748]: osdmap e252: 8 total, 7 up, 7 in 2024-03-24T12:39:29.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:29 smithi190 ceph-mon[33748]: pgmap v553: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 29785/179460 objects degraded (16.597%); 3818/179460 objects misplaced (2.127%) 2024-03-24T12:39:29.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:29.665 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:29.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:29 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:29 smithi029 ceph-mon[41500]: osdmap e252: 8 total, 7 up, 7 in 2024-03-24T12:39:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:29 smithi029 ceph-mon[41500]: pgmap v553: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 29785/179460 objects degraded (16.597%); 3818/179460 objects misplaced (2.127%) 2024-03-24T12:39:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:29 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:29 smithi029 ceph-mon[29919]: osdmap e252: 8 total, 7 up, 7 in 2024-03-24T12:39:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:29 smithi029 ceph-mon[29919]: pgmap v553: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.6 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s; 29785/179460 objects degraded (16.597%); 3818/179460 objects misplaced (2.127%) 2024-03-24T12:39:29.693 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:39:29.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:29.790 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:29.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:29.892 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:29.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:29.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:29.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:30.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:30.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:30.058 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:30.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:30.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:30.195 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:30.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:30.299 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:30.335 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:30.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:30.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:30.437 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:30.462 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:30.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:30.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:30.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:30.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:30.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:30.605 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:30 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:39:30.605 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:30 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:39:30.605 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:30 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 29785/179460 objects degraded (16.597%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:30.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:30 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:39:30.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:30 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:39:30.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:30 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 29785/179460 objects degraded (16.597%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:30.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:30 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:39:30.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:30 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:39:30.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:30 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 29785/179460 objects degraded (16.597%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:30.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:30.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:30.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:30.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:30.868 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:30.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:30.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:30.996 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:30.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:31.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:31.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:31.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:31.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:31.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:31.263 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:31.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:31.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:31.365 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:31.398 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:31.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:31.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:31.540 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:31.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:31.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:31.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:31 smithi190 ceph-mon[33748]: osdmap e253: 8 total, 7 up, 7 in 2024-03-24T12:39:31.620 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:31 smithi190 ceph-mon[33748]: pgmap v555: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 889 op/s; 29023/173295 objects degraded (16.748%); 3818/173295 objects misplaced (2.203%) 2024-03-24T12:39:31.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:31.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:31 smithi029 ceph-mon[41500]: osdmap e253: 8 total, 7 up, 7 in 2024-03-24T12:39:31.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:31 smithi029 ceph-mon[41500]: pgmap v555: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 889 op/s; 29023/173295 objects degraded (16.748%); 3818/173295 objects misplaced (2.203%) 2024-03-24T12:39:31.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:31 smithi029 ceph-mon[29919]: osdmap e253: 8 total, 7 up, 7 in 2024-03-24T12:39:31.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:31 smithi029 ceph-mon[29919]: pgmap v555: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.5 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 889 op/s; 29023/173295 objects degraded (16.748%); 3818/173295 objects misplaced (2.203%) 2024-03-24T12:39:31.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:31.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:31.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:31.676 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:39:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:39:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-03-24T12:39:31.676 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-03-24T12:39:31.676 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:39:31.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:31.798 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:31.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:31.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:31.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:31.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:32.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:32.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:32.078 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:32.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:32.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:32.202 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:32.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:32.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:32.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:32.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T12:39:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:32.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:32.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:32.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:32.460 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:32.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:32.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:32.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:32.604 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:32.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:32.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:32.614 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:39:32.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:32.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:32.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:32.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:32.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:32.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:32.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:33.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:33.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:33.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:33.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:33.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:33.142 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:33.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:33.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:33.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:33.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:33.384 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:33.410 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:33.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:39:33.513 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:33.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:33.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:33.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:33.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a[29896]: 2024-03-24T12:39:33.340+0000 7f0eeb338700 -1 mon.a@0(leader).osd e253 definitely_dead 0 2024-03-24T12:39:33.692 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:33.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:33.737 INFO:teuthology.orchestra.run.smithi029.stderr:marked in osd.6. 2024-03-24T12:39:33.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:33.821 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:33.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:33.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:33 smithi190 ceph-mon[33748]: pgmap v556: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 993 op/s; 27502/166215 objects degraded (16.546%); 3547/166215 objects misplaced (2.134%) 2024-03-24T12:39:33.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:33.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:33 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3900963870' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-24T12:39:33.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:33 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-24T12:39:33.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:33.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:33.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:34.047 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:34.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:33 smithi029 ceph-mon[41500]: pgmap v556: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 993 op/s; 27502/166215 objects degraded (16.546%); 3547/166215 objects misplaced (2.134%) 2024-03-24T12:39:34.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:34.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:33 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3900963870' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-24T12:39:34.048 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:33 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-24T12:39:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:33 smithi029 ceph-mon[29919]: pgmap v556: 148 pgs: 22 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.4 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 993 op/s; 27502/166215 objects degraded (16.546%); 3547/166215 objects misplaced (2.134%) 2024-03-24T12:39:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:33 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3900963870' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-24T12:39:34.048 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:33 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-24T12:39:34.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:34.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:34.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:34.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:34.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:34.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:34.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:34.299 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:34.323 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:34.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:34.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:34.448 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:34.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:34.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:34.458 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:39:34.458 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:105270 7.12 7:4e48be4a:::benchmark_data_smithi029_84282_object4507:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e123)", 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:29:52.951150+0000", 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 581.50986145499996, 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2059817719999999, 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.459 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 105270 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.951150+0000", 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.951150+0000", 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.951152+0000", 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4759999999999999e-06 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.951155+0000", 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7309999999999999e-06 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.951167+0000", 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1516e-05 2024-03-24T12:39:34.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.951175+0000", 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8809999999999996e-06 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.951224+0000", 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9079000000000002e-05 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.951229+0000", 2024-03-24T12:39:34.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4500000000000003e-06 2024-03-24T12:39:34.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.157047+0000", 2024-03-24T12:39:34.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.205817846 2024-03-24T12:39:34.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.157132+0000", 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4792999999999999e-05 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:105296 7.12 7:4c1616b2:::benchmark_data_smithi029_84282_object4533:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:29:52.959568+0000", 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 581.501443311, 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2061874720000001, 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.464 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 105296 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.959568+0000", 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.959568+0000", 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.959569+0000", 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1999999999999998e-07 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.959573+0000", 2024-03-24T12:39:34.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8340000000000001e-06 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.959585+0000", 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1651000000000001e-05 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.959592+0000", 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5689999999999999e-06 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.467 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.959644+0000", 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2306999999999999e-05 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.959653+0000", 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7020000000000007e-06 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.157522+0000", 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.197868669 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.157623+0000", 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100913 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.157706+0000", 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3696000000000006e-05 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.161670+0000", 2024-03-24T12:39:34.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0039632870000000002 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.162457+0000", 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00078756400000000004 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.165677+0000", 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032194960000000001 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.165711+0000", 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3791000000000002e-05 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.165756+0000", 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5173000000000003e-05 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.471 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:105300 7.12 7:4e351ab6:::benchmark_data_smithi029_84282_object4537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:29:52.966251+0000", 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 581.49476082900003, 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.201743301, 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 105300 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.966251+0000", 2024-03-24T12:39:34.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.966251+0000", 2024-03-24T12:39:34.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.966252+0000", 2024-03-24T12:39:34.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.006e-06 2024-03-24T12:39:34.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.966256+0000", 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7850000000000002e-06 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.966265+0000", 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5359999999999998e-06 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.966271+0000", 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.251e-06 2024-03-24T12:39:34.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.966319+0000", 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7459000000000001e-05 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.966326+0000", 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0190000000000001e-06 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.158702+0000", 2024-03-24T12:39:34.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.192376511 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.158778+0000", 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5637999999999996e-05 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.158857+0000", 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8992000000000005e-05 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.162467+0000", 2024-03-24T12:39:34.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036104689999999998 2024-03-24T12:39:34.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.165826+0000", 2024-03-24T12:39:34.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0033587270000000002 2024-03-24T12:39:34.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.167919+0000", 2024-03-24T12:39:34.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020929809999999998 2024-03-24T12:39:34.479 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.479 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.479 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.479 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.167963+0000", 2024-03-24T12:39:34.479 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4279000000000001e-05 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.167994+0000", 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0648000000000002e-05 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:105315 7.12 7:49f60590:::benchmark_data_smithi029_84282_object4552:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-24T12:39:34.480 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:29:52.987878+0000", 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 581.47313371899997, 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.187832582, 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 105315 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.481 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.987878+0000", 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.987878+0000", 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.987879+0000", 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5600000000000004e-07 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.482 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.987882+0000", 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5300000000000001e-06 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.987893+0000", 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0356e-05 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.987899+0000", 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9499999999999998e-06 2024-03-24T12:39:34.483 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.987944+0000", 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5274000000000001e-05 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.987951+0000", 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9600000000000003e-06 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.159715+0000", 2024-03-24T12:39:34.484 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.171764319 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.159833+0000", 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000117687 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.159936+0000", 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000103198 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.485 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.162474+0000", 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002538415 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.168042+0000", 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055677779999999998 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.175647+0000", 2024-03-24T12:39:34.486 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0076047220000000004 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.175680+0000", 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3429999999999997e-05 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.175710+0000", 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0107000000000002e-05 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.487 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:105320 7.12 7:4feee3cc:::benchmark_data_smithi029_84282_object4557:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:29:52.993072+0000", 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 581.46793984299995, 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1860627990000001, 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 105320 2024-03-24T12:39:34.488 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.993072+0000", 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.993072+0000", 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.489 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.993074+0000", 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.849e-06 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.993081+0000", 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3429999999999998e-06 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.993101+0000", 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9771000000000001e-05 2024-03-24T12:39:34.490 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.993111+0000", 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.045e-05 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.993191+0000", 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9992000000000003e-05 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.993201+0000", 2024-03-24T12:39:34.491 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8500000000000006e-06 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.160906+0000", 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1677051540000001 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.160975+0000", 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9315999999999998e-05 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.492 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.161035+0000", 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9481000000000002e-05 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.163889+0000", 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0028538090000000001 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.168086+0000", 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041971309999999998 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.493 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.179070+0000", 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010983689 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.179102+0000", 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2517000000000001e-05 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.179135+0000", 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2447000000000003e-05 2024-03-24T12:39:34.494 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:105326 7.12 7:4bebb817:::benchmark_data_smithi029_84282_object4563:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:29:52.999417+0000", 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 581.46159435499999, 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1798010910000001, 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.495 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 105326 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.999417+0000", 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.999417+0000", 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.496 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.999419+0000", 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3599999999999999e-06 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.999422+0000", 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8089999999999999e-06 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.999432+0000", 2024-03-24T12:39:34.497 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8539999999999992e-06 2024-03-24T12:39:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.999438+0000", 2024-03-24T12:39:34.498 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9150000000000001e-06 2024-03-24T12:39:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.999482+0000", 2024-03-24T12:39:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3803000000000001e-05 2024-03-24T12:39:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:52.999488+0000", 2024-03-24T12:39:34.499 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6860000000000003e-06 2024-03-24T12:39:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.500 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.161704+0000", 2024-03-24T12:39:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1622165309999999 2024-03-24T12:39:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.161772+0000", 2024-03-24T12:39:34.501 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7577999999999996e-05 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.161846+0000", 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4402999999999999e-05 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.165639+0000", 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0037930300000000002 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.502 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.170790+0000", 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051504020000000001 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.179161+0000", 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083713840000000008 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.179195+0000", 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4239000000000002e-05 2024-03-24T12:39:34.503 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.179218+0000", 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3096999999999999e-05 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:105389 7.12 7:4e03460a:::benchmark_data_smithi029_84282_object4626:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:29:53.082112+0000", 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 581.37889915599999, 2024-03-24T12:39:34.504 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1013325970000001, 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 105389 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.082112+0000", 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.505 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.082112+0000", 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.082114+0000", 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1969999999999999e-06 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.082118+0000", 2024-03-24T12:39:34.506 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5430000000000004e-06 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.082129+0000", 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1063999999999999e-05 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.082137+0000", 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7049999999999999e-06 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.507 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.082197+0000", 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9743e-05 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.082206+0000", 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0219999999999993e-06 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.162496+0000", 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.080290194 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.508 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.162566+0000", 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9962000000000005e-05 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.162626+0000", 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9840999999999998e-05 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.165644+0000", 2024-03-24T12:39:34.509 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030186900000000001 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.179259+0000", 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013614219 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.183345+0000", 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0040862870000000001 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.510 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.183400+0000", 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4644000000000001e-05 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.183445+0000", 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5485999999999998e-05 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.511 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:105410 7.12 7:49ac9525:::benchmark_data_smithi029_84282_object4647:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:29:53.106615+0000", 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 581.35439699000005, 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.076992666, 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 105410 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.512 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.106615+0000", 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.106615+0000", 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.106616+0000", 2024-03-24T12:39:34.513 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9399999999999993e-07 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.106620+0000", 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6469999999999997e-06 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.106631+0000", 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1168e-05 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.514 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.106638+0000", 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5629999999999997e-06 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.106686+0000", 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7556999999999999e-05 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.106692+0000", 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2729999999999999e-06 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.515 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.163203+0000", 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.056510885 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.163270+0000", 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7411999999999998e-05 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.163330+0000", 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9827000000000002e-05 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.165848+0000", 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025179170000000002 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.179295+0000", 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0134468 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.183501+0000", 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042063159999999999 2024-03-24T12:39:34.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.183559+0000", 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8155999999999997e-05 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.183607+0000", 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8151e-05 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:105426 7.12 7:4dc91452:::benchmark_data_smithi029_84282_object4663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:29:53.125998+0000", 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 581.33501409899998, 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0577143710000001, 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 105426 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.125998+0000", 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.125998+0000", 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.125999+0000", 2024-03-24T12:39:34.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.085e-06 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.126004+0000", 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3759999999999999e-06 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.126016+0000", 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2459e-05 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.521 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.126025+0000", 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8640000000000002e-06 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.126074+0000", 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8554000000000003e-05 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:53.126080+0000", 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.925e-06 2024-03-24T12:39:34.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.163922+0000", 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.037842038 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.163987+0000", 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5140000000000003e-05 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:39:34.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.164048+0000", 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0569999999999998e-05 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.165855+0000", 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018069749999999999 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.179330+0000", 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013475067 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.183650+0000", 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0043207829999999999 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.183680+0000", 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9669999999999999e-05 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:29:54.183712+0000", 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1865000000000001e-05 2024-03-24T12:39:34.525 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:171803 7.19 7:9f2017e5:::benchmark_data_smithi029_84282_object71040:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e148)", 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:36.898505+0000", 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.562506692, 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0120888159999999, 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.526 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 171803 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.898505+0000", 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.898505+0000", 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.527 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.898506+0000", 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0219999999999999e-06 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.898510+0000", 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7060000000000002e-06 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.898521+0000", 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1802e-05 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.898528+0000", 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2310000000000001e-06 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.898572+0000", 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4697999999999998e-05 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.898579+0000", 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7920000000000004e-06 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.910536+0000", 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0119570689999999 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.910594+0000", 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7496e-05 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.530 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:171894 7.19 7:9f3865cb:::benchmark_data_smithi029_84282_object71131:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:36.962222+0000", 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.49878921499999, 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.95486352200000002, 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 171894 2024-03-24T12:39:34.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.962222+0000", 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.962222+0000", 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.962223+0000", 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7899999999999997e-07 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.962228+0000", 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2590000000000002e-06 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.962238+0000", 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0735e-05 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.962245+0000", 2024-03-24T12:39:34.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.114e-06 2024-03-24T12:39:34.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.962313+0000", 2024-03-24T12:39:34.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7342999999999995e-05 2024-03-24T12:39:34.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.535 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:36.962323+0000", 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0489000000000001e-05 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.910969+0000", 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.94864558700000001 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.911049+0000", 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9839999999999995e-05 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.911395+0000", 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034685100000000002 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.916922+0000", 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0055260229999999997 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.916930+0000", 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.789e-06 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.916981+0000", 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1128000000000002e-05 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.917038+0000", 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6836999999999998e-05 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.917086+0000", 2024-03-24T12:39:34.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7648000000000002e-05 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:171948 7.19 7:9f7404ba:::benchmark_data_smithi029_84282_object71185:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:37.003137+0000", 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.45787447399999, 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91695912199999996, 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.539 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 171948 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.003137+0000", 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.003137+0000", 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.003138+0000", 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2139999999999999e-06 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.003143+0000", 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7840000000000003e-06 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.003157+0000", 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3599e-05 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.003165+0000", 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7989999999999994e-06 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.003231+0000", 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6836999999999997e-05 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.003242+0000", 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0589e-05 2024-03-24T12:39:34.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.913324+0000", 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.91008220100000004 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.913452+0000", 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012742600000000001 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:39:34.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.913706+0000", 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00025399100000000001 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.918504+0000", 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0047983710000000001 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.919971+0000", 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001466618 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.920033+0000", 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2928000000000005e-05 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.920068+0000", 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4594000000000001e-05 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.920096+0000", 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.8170999999999998e-05 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:171966 7.19 7:9e197bf8:::benchmark_data_smithi029_84282_object71203:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:37.017703+0000", 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.44330878400001, 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.90426718699999997, 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 171966 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.017703+0000", 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.017703+0000", 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.017703+0000", 2024-03-24T12:39:34.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.68e-07 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.017707+0000", 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9210000000000002e-06 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.017715+0000", 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.011e-06 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.017721+0000", 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3730000000000001e-06 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.017776+0000", 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5211999999999999e-05 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.017787+0000", 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0573e-05 2024-03-24T12:39:34.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.915231+0000", 2024-03-24T12:39:34.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.89744394000000005 2024-03-24T12:39:34.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.915338+0000", 2024-03-24T12:39:34.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107113 2024-03-24T12:39:34.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.915608+0000", 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027070799999999998 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.919983+0000", 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004374953 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.920136+0000", 2024-03-24T12:39:34.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000153008 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.921879+0000", 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0017428109999999999 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.921929+0000", 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9731000000000002e-05 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.921970+0000", 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1165000000000003e-05 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:171986 7.19 7:9e6f2ce6:::benchmark_data_smithi029_84282_object71223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:37.031166+0000", 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.42984598300001, 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.89751377399999999, 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.553 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 171986 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.031166+0000", 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.031166+0000", 2024-03-24T12:39:34.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.031167+0000", 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0529999999999999e-06 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.031171+0000", 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4630000000000003e-06 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.031182+0000", 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0502999999999999e-05 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.031188+0000", 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3210000000000001e-06 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.031277+0000", 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9409999999999999e-05 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.031289+0000", 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1698e-05 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.918529+0000", 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.88723980499999999 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.918595+0000", 2024-03-24T12:39:34.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6605000000000004e-05 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.918776+0000", 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00018018299999999999 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.922042+0000", 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032662469999999999 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.922075+0000", 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3535000000000003e-05 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.928574+0000", 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064983840000000003 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.928621+0000", 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7422000000000001e-05 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.928679+0000", 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8145e-05 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.14799.0:171988 7.19 7:99593341:::benchmark_data_smithi029_84282_object71225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e148)", 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:31:37.032551+0000", 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 477.42846057100002, 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.89628418600000004, 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:39:34.560 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.14799", 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 171988 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.032551+0000", 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.032551+0000", 2024-03-24T12:39:34.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.032552+0000", 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5300000000000002e-07 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.032556+0000", 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2220000000000004e-06 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.032637+0000", 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0282000000000001e-05 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.032645+0000", 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.123e-06 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.032721+0000", 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6069999999999995e-05 2024-03-24T12:39:34.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.032757+0000", 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6078000000000003e-05 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.920166+0000", 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.88740933300000002 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.920230+0000", 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4179999999999999e-05 2024-03-24T12:39:34.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.920422+0000", 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000191387 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.922230+0000", 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001808037 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.922263+0000", 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3254999999999999e-05 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.928733+0000", 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064699290000000001 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.928797+0000", 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4583000000000002e-05 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:31:37.928835+0000", 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7753999999999998e-05 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:206512 8.9 8:994915ff:::benchmark_data_smithi029_108893_object92652:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:35:57.665379+0000", 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 216.79563310699999, 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0863911829999999, 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.567 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 206512 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665379+0000", 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665379+0000", 2024-03-24T12:39:34.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665379+0000", 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8400000000000004e-07 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665382+0000", 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2490000000000001e-06 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665416+0000", 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3278999999999998e-05 2024-03-24T12:39:34.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665425+0000", 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6390000000000004e-06 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.670571+0000", 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051460949999999998 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.670884+0000", 2024-03-24T12:39:34.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031259099999999998 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.671224+0000", 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034033099999999998 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.681544+0000", 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010319274 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:39:34.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.693519+0000", 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011975725 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.698865+0000", 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0053460399999999998 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.698882+0000", 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6656000000000001e-05 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.751698+0000", 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.052816142 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.751770+0000", 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1577999999999998e-05 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:206515 8.9 8:93ca0348:::benchmark_data_smithi029_108893_object92655:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-24T12:39:34.573 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:35:57.665886+0000", 2024-03-24T12:39:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 216.79512522900001, 2024-03-24T12:39:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.086225156, 2024-03-24T12:39:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:39:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:39:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:39:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 206515 2024-03-24T12:39:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.574 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665886+0000", 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665886+0000", 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665887+0000", 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5600000000000004e-07 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665892+0000", 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4839999999999998e-06 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665900+0000", 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1070000000000005e-06 2024-03-24T12:39:34.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.665909+0000", 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0729999999999994e-06 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.674437+0000", 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0085282020000000004 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.577 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.674526+0000", 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8801999999999994e-05 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.674601+0000", 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4832000000000005e-05 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.685233+0000", 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010632424999999999 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699095+0000", 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013861998 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699103+0000", 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9619999999999993e-06 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.752058+0000", 2024-03-24T12:39:34.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0529552090000001 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.752112+0000", 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3205999999999998e-05 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:206517 8.9 8:9b79229e:::benchmark_data_smithi029_108893_object92657:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:35:57.666350+0000", 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 216.794661365, 2024-03-24T12:39:34.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.085838708, 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 206517 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.666350+0000", 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.666350+0000", 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.666351+0000", 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8400000000000003e-07 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.666355+0000", 2024-03-24T12:39:34.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.732e-06 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.666368+0000", 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2845e-05 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.666375+0000", 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9290000000000001e-06 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.677566+0000", 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011191303999999999 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.677692+0000", 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012579900000000001 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.677764+0000", 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2811000000000005e-05 2024-03-24T12:39:34.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.685254+0000", 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074891569999999998 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699141+0000", 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013887183000000001 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699145+0000", 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0820000000000001e-06 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.752142+0000", 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.052997261 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.752189+0000", 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6820999999999998e-05 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.586 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:206519 8.9 8:98cbf005:::benchmark_data_smithi029_108893_object92659:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:35:57.669951+0000", 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 216.791060875, 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.082314674, 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 206519 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.587 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.669951+0000", 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.669951+0000", 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.669952+0000", 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.184e-06 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.669958+0000", 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6459999999999998e-06 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.669968+0000", 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0219e-05 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.669976+0000", 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.225e-06 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.681574+0000", 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011597546 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.681689+0000", 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011548900000000001 2024-03-24T12:39:34.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.681801+0000", 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000111815 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.693244+0000", 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011443296 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699182+0000", 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059374609999999998 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699186+0000", 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4909999999999999e-06 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.752218+0000", 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.053032059 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.752265+0000", 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7243000000000002e-05 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24815.0:206523 8.9 8:9b836455:::benchmark_data_smithi029_108893_object92663:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:35:57.673899+0000", 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 216.787112412, 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0792909079999999, 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:39:34.593 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24815", 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 206523 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.673899+0000", 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:39:34.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.673899+0000", 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.673901+0000", 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5910000000000001e-06 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.673906+0000", 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2390000000000002e-06 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.673920+0000", 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3509999999999999e-05 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.673928+0000", 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1000000000000004e-06 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.685380+0000", 2024-03-24T12:39:34.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011452583000000001 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.685728+0000", 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034806399999999998 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,5", 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.686068+0000", 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000339668 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699207+0000", 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013138837 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:57.699211+0000", 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7969999999999998e-06 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.753127+0000", 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0539159659999999 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:39:34.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:39:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:39:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:35:58.753190+0000", 2024-03-24T12:39:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3553000000000006e-05 2024-03-24T12:39:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:39:34.599 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:39:34.599 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:39:34.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:34.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:34.625 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:34.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:34.726 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:34.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:34.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:34.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:34.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:34.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:34.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:34.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:34 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-24T12:39:34.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:34 smithi029 ceph-mon[41500]: osdmap e254: 8 total, 7 up, 8 in 2024-03-24T12:39:34.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:34 smithi029 ceph-mon[41500]: osdmap e255: 8 total, 7 up, 8 in 2024-03-24T12:39:34.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:34 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-24T12:39:34.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:34 smithi029 ceph-mon[29919]: osdmap e254: 8 total, 7 up, 8 in 2024-03-24T12:39:34.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:34 smithi029 ceph-mon[29919]: osdmap e255: 8 total, 7 up, 8 in 2024-03-24T12:39:34.991 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-03-24T12:39:34.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:35.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:35.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:35.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:35.117 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:35.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:34 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-24T12:39:35.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:34 smithi190 ceph-mon[33748]: osdmap e254: 8 total, 7 up, 8 in 2024-03-24T12:39:35.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:34 smithi190 ceph-mon[33748]: osdmap e255: 8 total, 7 up, 8 in 2024-03-24T12:39:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:35.155 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:35.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:35.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:35.297 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:35.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:35.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:35.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:35.409 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:39:35.437 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:35.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:35.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:35.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:35.580 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:35.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:35.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:35.714 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:39:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:35.721 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:35.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:35.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:35.850 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:35.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:35.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:35.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:35.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:35.991 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:35.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:36.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:35 smithi190 ceph-mon[33748]: pgmap v558: 148 pgs: 3 peering, 4 remapped+peering, 18 active+clean+remapped, 85 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 809 B/s rd, 0 B/s wr, 895 op/s; 26835/162486 objects degraded (16.515%); 3528/162486 objects misplaced (2.171%) 2024-03-24T12:39:36.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:35 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 26835/162486 objects degraded (16.515%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:36.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:35 smithi190 ceph-mon[33748]: osdmap e256: 8 total, 7 up, 8 in 2024-03-24T12:39:36.007 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:35 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3656483963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:39:36.092 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:36.110 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:39:36.111 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:35 smithi029 ceph-mon[41500]: pgmap v558: 148 pgs: 3 peering, 4 remapped+peering, 18 active+clean+remapped, 85 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 809 B/s rd, 0 B/s wr, 895 op/s; 26835/162486 objects degraded (16.515%); 3528/162486 objects misplaced (2.171%) 2024-03-24T12:39:36.111 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:35 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 26835/162486 objects degraded (16.515%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:36.111 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:35 smithi029 ceph-mon[41500]: osdmap e256: 8 total, 7 up, 8 in 2024-03-24T12:39:36.111 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:35 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3656483963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:39:36.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:35 smithi029 ceph-mon[29919]: pgmap v558: 148 pgs: 3 peering, 4 remapped+peering, 18 active+clean+remapped, 85 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 809 B/s rd, 0 B/s wr, 895 op/s; 26835/162486 objects degraded (16.515%); 3528/162486 objects misplaced (2.171%) 2024-03-24T12:39:36.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:35 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 26835/162486 objects degraded (16.515%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:36.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:35 smithi029 ceph-mon[29919]: osdmap e256: 8 total, 7 up, 8 in 2024-03-24T12:39:36.118 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:35 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3656483963' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:39:36.141 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:36.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:36.243 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:36.274 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:36.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:36.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:36.426 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:36.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:36.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:36.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:36.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:36.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:36.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:36.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:36.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:36.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:36.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:36.849 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:36.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:36.943 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:36 smithi190 ceph-mon[33748]: 9.0 starting backfill to osd.6 from (0'0,0'0] MAX to 253'10646 2024-03-24T12:39:36.943 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:36 smithi190 ceph-mon[33748]: 9.1 starting backfill to osd.6 from (0'0,0'0] MAX to 253'10812 2024-03-24T12:39:36.944 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:36 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:39:36.944 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:36 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3656483963' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:39:36.944 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:36 smithi190 ceph-mon[33748]: osdmap e257: 8 total, 7 up, 8 in 2024-03-24T12:39:36.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:36.988 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:36.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:36.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:37.009 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:36 smithi029 ceph-mon[41500]: 9.0 starting backfill to osd.6 from (0'0,0'0] MAX to 253'10646 2024-03-24T12:39:37.009 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:36 smithi029 ceph-mon[41500]: 9.1 starting backfill to osd.6 from (0'0,0'0] MAX to 253'10812 2024-03-24T12:39:37.009 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:36 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:39:37.009 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:36 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3656483963' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:39:37.009 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:36 smithi029 ceph-mon[41500]: osdmap e257: 8 total, 7 up, 8 in 2024-03-24T12:39:37.009 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:36 smithi029 ceph-mon[29919]: 9.0 starting backfill to osd.6 from (0'0,0'0] MAX to 253'10646 2024-03-24T12:39:37.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:36 smithi029 ceph-mon[29919]: 9.1 starting backfill to osd.6 from (0'0,0'0] MAX to 253'10812 2024-03-24T12:39:37.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:36 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:39:37.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:36 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3656483963' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:39:37.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:36 smithi029 ceph-mon[29919]: osdmap e257: 8 total, 7 up, 8 in 2024-03-24T12:39:37.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:37.133 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:37.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:37.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:37.247 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:39:37.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:37.278 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:37.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:37.301 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:39:37.343 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:39:37.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:37.388 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:39:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:37.424 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:37.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:37.430 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:39:37.473 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:39:37.514 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:39:37.514 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:39:37.514 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:39:37.514 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:39:37.514 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:39:37.515 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:39:37.515 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:39:37.515 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:39:37.515 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:39:37.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:37.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:37.562 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:37.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:37.593 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:39:37.593 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:39:37.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:37.703 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:37.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:37.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:37.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:37.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:37.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:37.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:37 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:39:37.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:37 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:39:37.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:37 smithi190 ceph-mon[33748]: pgmap v562: 148 pgs: 3 peering, 9 remapped+peering, 13 active+clean+remapped, 85 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 26063/156987 objects degraded (16.602%); 3274/156987 objects misplaced (2.086%); 0 B/s, 0 objects/s recovering 2024-03-24T12:39:37.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:37 smithi190 ceph-mon[33748]: 5.3 scrub starts 2024-03-24T12:39:37.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:37 smithi190 ceph-mon[33748]: 5.3 scrub ok 2024-03-24T12:39:37.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:37 smithi190 ceph-mon[33748]: osdmap e258: 8 total, 7 up, 8 in 2024-03-24T12:39:37.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:37 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2477646335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:39:37.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:39:37.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:39:37.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[29919]: pgmap v562: 148 pgs: 3 peering, 9 remapped+peering, 13 active+clean+remapped, 85 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 26063/156987 objects degraded (16.602%); 3274/156987 objects misplaced (2.086%); 0 B/s, 0 objects/s recovering 2024-03-24T12:39:37.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[29919]: 5.3 scrub starts 2024-03-24T12:39:37.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[29919]: 5.3 scrub ok 2024-03-24T12:39:37.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[29919]: osdmap e258: 8 total, 7 up, 8 in 2024-03-24T12:39:37.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2477646335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:39:37.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:39:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:39:37.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:39:37.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:39:37.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[41500]: pgmap v562: 148 pgs: 3 peering, 9 remapped+peering, 13 active+clean+remapped, 85 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.2 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 0 B/s wr, 768 op/s; 26063/156987 objects degraded (16.602%); 3274/156987 objects misplaced (2.086%); 0 B/s, 0 objects/s recovering 2024-03-24T12:39:37.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[41500]: 5.3 scrub starts 2024-03-24T12:39:37.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[41500]: 5.3 scrub ok 2024-03-24T12:39:37.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[41500]: osdmap e258: 8 total, 7 up, 8 in 2024-03-24T12:39:37.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:37 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2477646335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:39:37.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:37.966 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:37.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:38.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:38.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:38.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:38.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:38.225 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:38.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:38.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:39:38.265 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:39:38.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:38.354 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:38.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:38.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:38.378 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:39:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:39:38.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:39:38.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:38.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:38.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:38.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:38.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:38.629 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:38.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:38.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:38.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:38.772 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:38.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:38.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:38.783 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:39:38.875 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:38.911 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:38.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:39.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:39.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:39.045 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:39.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:39.175 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:39.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:39.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:39.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:39.320 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:39.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:39.421 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:39.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:39.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:39.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:39.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:39.602 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:39.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:39.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:39:39.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:39.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:39.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:39.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:39.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:39.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:39.992 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:39:39.992 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:39:39.993 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-24T12:39:39.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:39:40.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:40.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:40.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:40.069 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:40.069 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:39 smithi029 ceph-mon[29919]: pgmap v564: 148 pgs: 3 peering, 6 remapped+peering, 13 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 793 op/s; 25235/152982 objects degraded (16.495%); 3278/152982 objects misplaced (2.143%); 144 KiB/s, 2 objects/s recovering 2024-03-24T12:39:40.069 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:39 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:39:40.069 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:40.070 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:39 smithi029 ceph-mon[41500]: pgmap v564: 148 pgs: 3 peering, 6 remapped+peering, 13 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 793 op/s; 25235/152982 objects degraded (16.495%); 3278/152982 objects misplaced (2.143%); 144 KiB/s, 2 objects/s recovering 2024-03-24T12:39:40.070 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:39 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:39:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:39 smithi190 ceph-mon[33748]: pgmap v564: 148 pgs: 3 peering, 6 remapped+peering, 13 active+clean+remapped, 88 active+clean, 22 active+undersized, 16 active+undersized+degraded; 3.1 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 793 op/s; 25235/152982 objects degraded (16.495%); 3278/152982 objects misplaced (2.143%); 144 KiB/s, 2 objects/s recovering 2024-03-24T12:39:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:39 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:39:40.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:40.212 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:39:40.245 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:40.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:40.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:39:40.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:40.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:40.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:40.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:40.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:40.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:40.531 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:40.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:40.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:40.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:40.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:40.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:40.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:40.812 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:40.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:40.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:40.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:40.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:40.965 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:40.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:41.024 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:40 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 25235/152982 objects degraded (16.495%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:41.024 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:40 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:39:41.024 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:40 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:39:41.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:41.075 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:39:41.075 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:39:41.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:41.115 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:41.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:41.119 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:40 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 25235/152982 objects degraded (16.495%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:41.119 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:40 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:39:41.119 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:40 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:39:41.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:40 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 25235/152982 objects degraded (16.495%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:41.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:40 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:39:41.120 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:40 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:39:41.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:41.246 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:41.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:41.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:41.374 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:41.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:41.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:41.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:41.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:41.521 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:41.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:41.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:41.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:41.667 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:41.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:41.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:41.810 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:41.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:41.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:41.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:39:41.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:41.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:41.955 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:41.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:42.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:42.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:42.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:42.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:41 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:39:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:41 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:39:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:41 smithi190 ceph-mon[33748]: pgmap v565: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 976 op/s; 24308/145086 objects degraded (16.754%); 7786/145086 objects misplaced (5.366%); 15 MiB/s, 242 objects/s recovering 2024-03-24T12:39:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:41 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:39:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:41 smithi190 ceph-mon[33748]: 3.1d scrub starts 2024-03-24T12:39:42.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:41 smithi190 ceph-mon[33748]: 3.1d scrub ok 2024-03-24T12:39:42.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:39:42.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:39:42.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[29919]: pgmap v565: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 976 op/s; 24308/145086 objects degraded (16.754%); 7786/145086 objects misplaced (5.366%); 15 MiB/s, 242 objects/s recovering 2024-03-24T12:39:42.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:39:42.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[29919]: 3.1d scrub starts 2024-03-24T12:39:42.161 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[29919]: 3.1d scrub ok 2024-03-24T12:39:42.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:39:42.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:39:42.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[41500]: pgmap v565: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 976 op/s; 24308/145086 objects degraded (16.754%); 7786/145086 objects misplaced (5.366%); 15 MiB/s, 242 objects/s recovering 2024-03-24T12:39:42.161 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:39:42.162 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[41500]: 3.1d scrub starts 2024-03-24T12:39:42.162 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:41 smithi029 ceph-mon[41500]: 3.1d scrub ok 2024-03-24T12:39:42.185 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:39:42.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:42.209 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:39:42.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:42.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:42.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:42.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:42.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:42.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:42.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:42.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:42.485 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:42.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:42.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:42.613 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:42.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:42.622 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:39:42.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:42.745 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:42.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:42.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:42.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:42 smithi190 ceph-mon[33748]: 6.1e scrub starts 2024-03-24T12:39:42.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:42 smithi190 ceph-mon[33748]: 6.1e scrub ok 2024-03-24T12:39:42.888 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:42.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:42.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:42.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:43.029 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:43.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:43.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:43.045 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:42 smithi029 ceph-mon[29919]: 6.1e scrub starts 2024-03-24T12:39:43.045 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:42 smithi029 ceph-mon[29919]: 6.1e scrub ok 2024-03-24T12:39:43.046 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:42 smithi029 ceph-mon[41500]: 6.1e scrub starts 2024-03-24T12:39:43.046 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:42 smithi029 ceph-mon[41500]: 6.1e scrub ok 2024-03-24T12:39:43.131 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:43.161 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:43.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:43.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:43.289 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:43.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:43.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:43.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:43.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:43.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:43.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:43.579 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:43.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:43.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:43.708 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:43.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:43.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:43.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:43.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:43.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:43.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:43.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:43.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:43.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:44.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:44.093 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:43 smithi190 ceph-mon[33748]: 4.8 scrub starts 2024-03-24T12:39:44.093 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:43 smithi190 ceph-mon[33748]: 4.8 scrub ok 2024-03-24T12:39:44.093 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:43 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:39:44.093 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:43 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:39:44.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:43 smithi190 ceph-mon[33748]: pgmap v566: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 985 op/s; 22594/136782 objects degraded (16.518%); 7435/136782 objects misplaced (5.436%); 12 MiB/s, 189 objects/s recovering 2024-03-24T12:39:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:44.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:43 smithi029 ceph-mon[29919]: 4.8 scrub starts 2024-03-24T12:39:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:43 smithi029 ceph-mon[29919]: 4.8 scrub ok 2024-03-24T12:39:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:43 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:39:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:43 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:39:44.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:43 smithi029 ceph-mon[29919]: pgmap v566: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 985 op/s; 22594/136782 objects degraded (16.518%); 7435/136782 objects misplaced (5.436%); 12 MiB/s, 189 objects/s recovering 2024-03-24T12:39:44.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:43 smithi029 ceph-mon[41500]: 4.8 scrub starts 2024-03-24T12:39:44.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:43 smithi029 ceph-mon[41500]: 4.8 scrub ok 2024-03-24T12:39:44.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:43 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:39:44.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:43 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:39:44.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:43 smithi029 ceph-mon[41500]: pgmap v566: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 985 op/s; 22594/136782 objects degraded (16.518%); 7435/136782 objects misplaced (5.436%); 12 MiB/s, 189 objects/s recovering 2024-03-24T12:39:44.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:44.254 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:39:44.254 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:39:44.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:44.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:44.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:44.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:44.406 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:44.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:44.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:44.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:44.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:44.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:44.662 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:44.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:44.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:44.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:44.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:44.806 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:44.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:44.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:44 smithi190 ceph-mon[33748]: 9.b scrub starts 2024-03-24T12:39:44.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:44.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:44.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:44.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:44.953 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:39:45.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:45.096 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:45.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:45.164 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:44 smithi029 ceph-mon[29919]: 9.b scrub starts 2024-03-24T12:39:45.165 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:44 smithi029 ceph-mon[41500]: 9.b scrub starts 2024-03-24T12:39:45.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:45.223 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:45.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:45.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:45.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:45.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:45.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:45.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:45.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:45.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:45.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:45.612 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:45.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:45.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:45.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:45.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:45.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:45.835 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:39:45.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:45.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:45.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:45.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:46.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:46.028 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:46.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:46.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:46.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:45 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:39:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:45 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:39:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:45 smithi190 ceph-mon[33748]: 9.b scrub ok 2024-03-24T12:39:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:45 smithi190 ceph-mon[33748]: pgmap v567: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 911 op/s; 22263/135789 objects degraded (16.395%); 7435/135789 objects misplaced (5.475%); 11 MiB/s, 168 objects/s recovering 2024-03-24T12:39:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:45 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/766473879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:39:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:45 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 22263/135789 objects degraded (16.395%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:39:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:39:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[29919]: 9.b scrub ok 2024-03-24T12:39:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[29919]: pgmap v567: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 911 op/s; 22263/135789 objects degraded (16.395%); 7435/135789 objects misplaced (5.475%); 11 MiB/s, 168 objects/s recovering 2024-03-24T12:39:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/766473879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:39:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 22263/135789 objects degraded (16.395%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:46.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:39:46.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:39:46.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[41500]: 9.b scrub ok 2024-03-24T12:39:46.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[41500]: pgmap v567: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 911 op/s; 22263/135789 objects degraded (16.395%); 7435/135789 objects misplaced (5.475%); 11 MiB/s, 168 objects/s recovering 2024-03-24T12:39:46.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/766473879' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:39:46.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:45 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 22263/135789 objects degraded (16.395%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:46.174 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:46.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:46.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:46.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:46.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:46.318 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:46.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:46.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:46.459 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:46.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:46.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:46.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:46.607 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:46.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:46.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:46.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:46.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:46.748 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:46.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:46.849 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:46.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:46.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:46.979 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:47.103 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:47.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:47.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:47.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:46 smithi190 ceph-mon[33748]: 3.d scrub starts 2024-03-24T12:39:47.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:46 smithi190 ceph-mon[33748]: 3.d scrub ok 2024-03-24T12:39:47.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:46 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:39:47.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:46 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/766473879' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:39:47.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:46 smithi190 ceph-mon[33748]: osdmap e259: 8 total, 7 up, 8 in 2024-03-24T12:39:47.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:46 smithi029 ceph-mon[29919]: 3.d scrub starts 2024-03-24T12:39:47.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:46 smithi029 ceph-mon[29919]: 3.d scrub ok 2024-03-24T12:39:47.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:46 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:39:47.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:46 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/766473879' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:39:47.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:46 smithi029 ceph-mon[29919]: osdmap e259: 8 total, 7 up, 8 in 2024-03-24T12:39:47.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:46 smithi029 ceph-mon[41500]: 3.d scrub starts 2024-03-24T12:39:47.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:46 smithi029 ceph-mon[41500]: 3.d scrub ok 2024-03-24T12:39:47.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:46 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:39:47.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:46 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/766473879' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:39:47.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:46 smithi029 ceph-mon[41500]: osdmap e259: 8 total, 7 up, 8 in 2024-03-24T12:39:47.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:47.240 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:47.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:47.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:47.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:47.377 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:47.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:47.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:47.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:47.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:47.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:47.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:47.637 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:47.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:47.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:47.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:47.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:47.767 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:47.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:47.868 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:47.904 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:47.905 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:47.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[29919]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:39:47.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[29919]: 3.11 scrub starts 2024-03-24T12:39:47.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[29919]: 3.11 scrub ok 2024-03-24T12:39:47.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[29919]: 5.a scrub starts 2024-03-24T12:39:47.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[29919]: 5.a scrub ok 2024-03-24T12:39:47.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[29919]: pgmap v569: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.07k op/s; 20974/126288 objects degraded (16.608%); 5207/126288 objects misplaced (4.123%); 19 MiB/s, 310 objects/s recovering 2024-03-24T12:39:47.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:39:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[41500]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:39:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[41500]: 3.11 scrub starts 2024-03-24T12:39:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[41500]: 3.11 scrub ok 2024-03-24T12:39:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[41500]: 5.a scrub starts 2024-03-24T12:39:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[41500]: 5.a scrub ok 2024-03-24T12:39:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[41500]: pgmap v569: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.07k op/s; 20974/126288 objects degraded (16.608%); 5207/126288 objects misplaced (4.123%); 19 MiB/s, 310 objects/s recovering 2024-03-24T12:39:47.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:39:47.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:39:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:39:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:39:48.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:48.030 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:48.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:48.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:48.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:47 smithi190 ceph-mon[33748]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:39:48.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:47 smithi190 ceph-mon[33748]: 3.11 scrub starts 2024-03-24T12:39:48.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:47 smithi190 ceph-mon[33748]: 3.11 scrub ok 2024-03-24T12:39:48.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:47 smithi190 ceph-mon[33748]: 5.a scrub starts 2024-03-24T12:39:48.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:47 smithi190 ceph-mon[33748]: 5.a scrub ok 2024-03-24T12:39:48.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:47 smithi190 ceph-mon[33748]: pgmap v569: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.6 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.07k op/s; 20974/126288 objects degraded (16.608%); 5207/126288 objects misplaced (4.123%); 19 MiB/s, 310 objects/s recovering 2024-03-24T12:39:48.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:39:48.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:48.159 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:48.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:48.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:48.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:48.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:48.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:48.427 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:48.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:48.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:48.551 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:48.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:48.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:39:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:39:48.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:48.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:48.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:48.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:48.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:48.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:48.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:48.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:49.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:49.099 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:49.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:49.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:48 smithi190 ceph-mon[33748]: 4.4 deep-scrub starts 2024-03-24T12:39:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:48 smithi190 ceph-mon[33748]: 4.4 deep-scrub ok 2024-03-24T12:39:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:48 smithi190 ceph-mon[33748]: 4.1 scrub starts 2024-03-24T12:39:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:48 smithi190 ceph-mon[33748]: 4.1 scrub ok 2024-03-24T12:39:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:49.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:48 smithi029 ceph-mon[41500]: 4.4 deep-scrub starts 2024-03-24T12:39:49.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:48 smithi029 ceph-mon[41500]: 4.4 deep-scrub ok 2024-03-24T12:39:49.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:48 smithi029 ceph-mon[41500]: 4.1 scrub starts 2024-03-24T12:39:49.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:48 smithi029 ceph-mon[41500]: 4.1 scrub ok 2024-03-24T12:39:49.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:49.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:48 smithi029 ceph-mon[29919]: 4.4 deep-scrub starts 2024-03-24T12:39:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:48 smithi029 ceph-mon[29919]: 4.4 deep-scrub ok 2024-03-24T12:39:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:48 smithi029 ceph-mon[29919]: 4.1 scrub starts 2024-03-24T12:39:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:48 smithi029 ceph-mon[29919]: 4.1 scrub ok 2024-03-24T12:39:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:39:49.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:49.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:49.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:49.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:49.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:49.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:49.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:49.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:49.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:49.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:49.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:49.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:49.625 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:49.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:49.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:49.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:49.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:49.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:49.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:49.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:49.892 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:49.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:49.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:50.034 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:50.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:50.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:49 smithi190 ceph-mon[33748]: 3.e scrub starts 2024-03-24T12:39:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:49 smithi190 ceph-mon[33748]: 3.e scrub ok 2024-03-24T12:39:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:49 smithi190 ceph-mon[33748]: pgmap v570: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 20056/121695 objects degraded (16.481%); 5207/121695 objects misplaced (4.279%); 18 MiB/s, 295 objects/s recovering 2024-03-24T12:39:50.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:50.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:50.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:50.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:50.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:50.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:49 smithi029 ceph-mon[41500]: 3.e scrub starts 2024-03-24T12:39:50.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:49 smithi029 ceph-mon[41500]: 3.e scrub ok 2024-03-24T12:39:50.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:49 smithi029 ceph-mon[41500]: pgmap v570: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 20056/121695 objects degraded (16.481%); 5207/121695 objects misplaced (4.279%); 18 MiB/s, 295 objects/s recovering 2024-03-24T12:39:50.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:39:50.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:49 smithi029 ceph-mon[29919]: 3.e scrub starts 2024-03-24T12:39:50.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:49 smithi029 ceph-mon[29919]: 3.e scrub ok 2024-03-24T12:39:50.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:49 smithi029 ceph-mon[29919]: pgmap v570: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.5 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.04k op/s; 20056/121695 objects degraded (16.481%); 5207/121695 objects misplaced (4.279%); 18 MiB/s, 295 objects/s recovering 2024-03-24T12:39:50.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:50.291 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:50.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:50.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:50.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:50.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:50.441 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:50.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:50.542 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:50.571 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:50.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:50.671 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:50.695 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:50.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:50.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:50.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:50.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:50.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:50.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:50.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:50.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:50.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:51.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:51.087 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:51.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:51.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:51.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:50 smithi190 ceph-mon[33748]: 6.18 scrub starts 2024-03-24T12:39:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:50 smithi190 ceph-mon[33748]: 6.18 scrub ok 2024-03-24T12:39:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:50 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 20056/121695 objects degraded (16.481%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:50 smithi190 ceph-mon[33748]: osdmap e260: 8 total, 7 up, 8 in 2024-03-24T12:39:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:50 smithi029 ceph-mon[41500]: 6.18 scrub starts 2024-03-24T12:39:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:50 smithi029 ceph-mon[41500]: 6.18 scrub ok 2024-03-24T12:39:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:50 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 20056/121695 objects degraded (16.481%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:50 smithi029 ceph-mon[41500]: osdmap e260: 8 total, 7 up, 8 in 2024-03-24T12:39:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:50 smithi029 ceph-mon[29919]: 6.18 scrub starts 2024-03-24T12:39:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:50 smithi029 ceph-mon[29919]: 6.18 scrub ok 2024-03-24T12:39:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:50 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 20056/121695 objects degraded (16.481%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:50 smithi029 ceph-mon[29919]: osdmap e260: 8 total, 7 up, 8 in 2024-03-24T12:39:51.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:51.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:51.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:51.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:51.321 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:51.362 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:51.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:51.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:51.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:51.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:51.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:51.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:51.594 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:51.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:51.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:51.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:51.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:51.768 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:51.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:51.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:51.894 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:51.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:51.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:51.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:52.025 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:52.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:52.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:52.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:52.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:51 smithi190 ceph-mon[33748]: 6.9 scrub starts 2024-03-24T12:39:52.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:51 smithi190 ceph-mon[33748]: 6.9 scrub ok 2024-03-24T12:39:52.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:51 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:39:52.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:51 smithi190 ceph-mon[33748]: pgmap v572: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 931 op/s; 19119/114438 objects degraded (16.707%); 3619/114438 objects misplaced (3.162%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:39:52.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:51 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:39:52.162 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:52.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:52.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:52.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:51 smithi029 ceph-mon[41500]: 6.9 scrub starts 2024-03-24T12:39:52.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:51 smithi029 ceph-mon[41500]: 6.9 scrub ok 2024-03-24T12:39:52.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:51 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:39:52.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:51 smithi029 ceph-mon[41500]: pgmap v572: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 931 op/s; 19119/114438 objects degraded (16.707%); 3619/114438 objects misplaced (3.162%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:39:52.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:51 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:39:52.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:51 smithi029 ceph-mon[29919]: 6.9 scrub starts 2024-03-24T12:39:52.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:51 smithi029 ceph-mon[29919]: 6.9 scrub ok 2024-03-24T12:39:52.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:51 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:39:52.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:51 smithi029 ceph-mon[29919]: pgmap v572: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 90 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 931 op/s; 19119/114438 objects degraded (16.707%); 3619/114438 objects misplaced (3.162%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:39:52.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:51 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:39:52.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:52.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:52.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:52.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:52.418 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:52.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:52.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:52.545 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:52.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:52.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:52.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:52.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:52.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:52.811 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:52.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:52.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:52.950 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:52.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:52.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:53.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:53.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:53.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:52 smithi190 ceph-mon[33748]: 4.6 deep-scrub starts 2024-03-24T12:39:53.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:52 smithi190 ceph-mon[33748]: 4.6 deep-scrub ok 2024-03-24T12:39:53.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:52 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:39:53.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:52 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:39:53.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:52 smithi029 ceph-mon[29919]: 4.6 deep-scrub starts 2024-03-24T12:39:53.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:52 smithi029 ceph-mon[29919]: 4.6 deep-scrub ok 2024-03-24T12:39:53.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:52 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:39:53.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:52 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:39:53.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:52 smithi029 ceph-mon[41500]: 4.6 deep-scrub starts 2024-03-24T12:39:53.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:52 smithi029 ceph-mon[41500]: 4.6 deep-scrub ok 2024-03-24T12:39:53.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:52 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:39:53.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:52 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:39:53.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:53.203 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:53.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:53.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:53.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:53.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:53.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:53.454 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:53.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:53.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:53.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:53.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:53.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:53.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:53.705 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:53.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:53.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:53.830 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:53.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:53.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:53.959 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:53.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:54.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:54.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:54.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:54.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:53 smithi190 ceph-mon[33748]: pgmap v573: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 89 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 2.2 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 17726/107601 objects degraded (16.474%); 3320/107601 objects misplaced (3.085%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:39:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:53 smithi190 ceph-mon[33748]: osdmap e261: 8 total, 7 up, 8 in 2024-03-24T12:39:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:53 smithi190 ceph-mon[33748]: 6.a scrub starts 2024-03-24T12:39:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:53 smithi190 ceph-mon[33748]: 6.a scrub ok 2024-03-24T12:39:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:53 smithi029 ceph-mon[29919]: pgmap v573: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 89 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 2.2 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 17726/107601 objects degraded (16.474%); 3320/107601 objects misplaced (3.085%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:39:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:53 smithi029 ceph-mon[29919]: osdmap e261: 8 total, 7 up, 8 in 2024-03-24T12:39:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:53 smithi029 ceph-mon[29919]: 6.a scrub starts 2024-03-24T12:39:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:53 smithi029 ceph-mon[29919]: 6.a scrub ok 2024-03-24T12:39:54.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:53 smithi029 ceph-mon[41500]: pgmap v573: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 89 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 2.2 GiB data, 9.7 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.18k op/s; 17726/107601 objects degraded (16.474%); 3320/107601 objects misplaced (3.085%); 23 MiB/s, 361 objects/s recovering 2024-03-24T12:39:54.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:53 smithi029 ceph-mon[41500]: osdmap e261: 8 total, 7 up, 8 in 2024-03-24T12:39:54.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:53 smithi029 ceph-mon[41500]: 6.a scrub starts 2024-03-24T12:39:54.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:53 smithi029 ceph-mon[41500]: 6.a scrub ok 2024-03-24T12:39:54.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:54.209 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:54.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:54.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:54.334 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:54.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:54.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:54.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:54.460 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:54.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:54.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:54.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:54.586 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:54.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:54.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:54.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:54.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:54.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:54.841 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:54.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:54.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:54.973 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:54.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:55.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:55.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:55.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:55.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:54 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:39:55.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:54 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:39:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:54 smithi190 ceph-mon[33748]: 5.15 scrub starts 2024-03-24T12:39:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:54 smithi190 ceph-mon[33748]: 5.15 scrub ok 2024-03-24T12:39:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:54 smithi190 ceph-mon[33748]: 6.f deep-scrub starts 2024-03-24T12:39:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:54 smithi190 ceph-mon[33748]: 6.f deep-scrub ok 2024-03-24T12:39:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:54 smithi190 ceph-mon[33748]: osdmap e262: 8 total, 7 up, 8 in 2024-03-24T12:39:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:54 smithi190 ceph-mon[33748]: 3.2 scrub starts 2024-03-24T12:39:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:54 smithi190 ceph-mon[33748]: 3.2 scrub ok 2024-03-24T12:39:55.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:39:55.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:39:55.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[29919]: 5.15 scrub starts 2024-03-24T12:39:55.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[29919]: 5.15 scrub ok 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[29919]: 6.f deep-scrub starts 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[29919]: 6.f deep-scrub ok 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[29919]: osdmap e262: 8 total, 7 up, 8 in 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[29919]: 3.2 scrub starts 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[29919]: 3.2 scrub ok 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[41500]: 5.15 scrub starts 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[41500]: 5.15 scrub ok 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[41500]: 6.f deep-scrub starts 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[41500]: 6.f deep-scrub ok 2024-03-24T12:39:55.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[41500]: osdmap e262: 8 total, 7 up, 8 in 2024-03-24T12:39:55.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[41500]: 3.2 scrub starts 2024-03-24T12:39:55.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:54 smithi029 ceph-mon[41500]: 3.2 scrub ok 2024-03-24T12:39:55.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:55.241 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:55.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:55.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:55.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:55.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:55.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:55.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:55.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:55.495 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:55.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:55.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:55.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:55.620 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:55.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:55.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:55.722 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:55.751 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:55.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:55.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:55.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:55.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:55.891 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:55.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:55.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:56.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:56.033 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:56.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:55 smithi190 ceph-mon[33748]: pgmap v576: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 89 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 2.2 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 824 op/s; 17480/106863 objects degraded (16.357%); 3320/106863 objects misplaced (3.107%); 14 MiB/s, 227 objects/s recovering 2024-03-24T12:39:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:55 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:39:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:55 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:39:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:55 smithi190 ceph-mon[33748]: osdmap e263: 8 total, 7 up, 8 in 2024-03-24T12:39:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:55 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 17480/106863 objects degraded (16.357%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:55 smithi190 ceph-mon[33748]: osdmap e264: 8 total, 7 up, 8 in 2024-03-24T12:39:56.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:56.158 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:56.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[41500]: pgmap v576: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 89 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 2.2 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 824 op/s; 17480/106863 objects degraded (16.357%); 3320/106863 objects misplaced (3.107%); 14 MiB/s, 227 objects/s recovering 2024-03-24T12:39:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:39:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:39:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[41500]: osdmap e263: 8 total, 7 up, 8 in 2024-03-24T12:39:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 17480/106863 objects degraded (16.357%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[41500]: osdmap e264: 8 total, 7 up, 8 in 2024-03-24T12:39:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[29919]: pgmap v576: 148 pgs: 2 active+remapped+backfilling, 18 active+clean+remapped, 89 active+clean, 22 active+undersized, 1 active+clean+scrubbing+deep, 16 active+undersized+degraded; 2.2 GiB data, 9.7 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 824 op/s; 17480/106863 objects degraded (16.357%); 3320/106863 objects misplaced (3.107%); 14 MiB/s, 227 objects/s recovering 2024-03-24T12:39:56.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:39:56.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:39:56.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[29919]: osdmap e263: 8 total, 7 up, 8 in 2024-03-24T12:39:56.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 17480/106863 objects degraded (16.357%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:39:56.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:55 smithi029 ceph-mon[29919]: osdmap e264: 8 total, 7 up, 8 in 2024-03-24T12:39:56.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:56.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:56.284 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:56.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:56.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:56.424 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:56.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:56.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:56.562 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:56.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:56.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:56.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:56.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:56.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:56.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:56.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:56.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:56.942 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:56.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:56.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:57.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:57.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:57.087 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:57.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:57.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:56 smithi029 ceph-mon[41500]: 6.c scrub starts 2024-03-24T12:39:57.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 6.c scrub ok 2024-03-24T12:39:57.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:39:57.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 6.15 deep-scrub starts 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 6.15 deep-scrub ok 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 5.1c scrub starts 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 5.1c scrub ok 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:56 smithi029 ceph-mon[29919]: 6.c scrub starts 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 6.c scrub ok 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 6.15 deep-scrub starts 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 6.15 deep-scrub ok 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:39:57.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 5.1c scrub starts 2024-03-24T12:39:57.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 5.1c scrub ok 2024-03-24T12:39:57.188 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:57.215 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:57.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:57.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:57.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:57.342 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:57.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:57.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:56 smithi190 ceph-mon[33748]: 6.c scrub starts 2024-03-24T12:39:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:56 smithi190 ceph-mon[33748]: 6.c scrub ok 2024-03-24T12:39:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:56 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:39:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:56 smithi190 ceph-mon[33748]: 6.15 deep-scrub starts 2024-03-24T12:39:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:56 smithi190 ceph-mon[33748]: 6.15 deep-scrub ok 2024-03-24T12:39:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:56 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:39:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:56 smithi190 ceph-mon[33748]: 5.1c scrub starts 2024-03-24T12:39:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:56 smithi190 ceph-mon[33748]: 5.1c scrub ok 2024-03-24T12:39:57.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:57.466 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:57.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:57.568 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:57.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:57.592 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:57.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:57.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:57.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:57.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:57.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:57.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:57.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:57.888 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:39:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:39:57.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:39:58.000 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:58.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:58.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:39:58.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:58.127 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:58.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:58.158 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:57 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:39:58.159 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:57 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:39:58.159 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:57 smithi190 ceph-mon[33748]: pgmap v579: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 545 op/s; 16704/101052 objects degraded (16.530%); 2121/101052 objects misplaced (2.099%); 15 MiB/s, 244 objects/s recovering 2024-03-24T12:39:58.159 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:57 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:39:58.159 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:57 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:39:58.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:39:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:39:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: pgmap v579: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 545 op/s; 16704/101052 objects degraded (16.530%); 2121/101052 objects misplaced (2.099%); 15 MiB/s, 244 objects/s recovering 2024-03-24T12:39:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:39:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:39:58.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:39:58.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:39:58.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: pgmap v579: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.0 GiB data, 9.3 GiB used, 706 GiB / 715 GiB avail; 0 B/s wr, 545 op/s; 16704/101052 objects degraded (16.530%); 2121/101052 objects misplaced (2.099%); 15 MiB/s, 244 objects/s recovering 2024-03-24T12:39:58.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:39:58.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:57 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:39:58.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:58.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:58.252 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:58.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:58.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:58.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:58.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:58.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:58.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:58.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:58.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:58.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:58.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:39:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:39:58.647 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:58.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:58.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:58.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:58.784 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:58.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:58.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:58.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:58.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:58.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:58.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:59.022 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:59.063 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:59.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:59.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:59.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:58 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:39:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:58 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:39:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:58 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:39:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:58 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:39:59.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:39:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:58 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:39:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:58 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:39:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:58 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:39:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:58 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:39:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:58 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:39:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:58 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:39:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:58 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:39:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:58 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:39:59.190 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:59.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:59.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:39:59.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:59.316 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:59.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:59.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:39:59.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:39:59.444 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:59.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:59.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:39:59.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:39:59.571 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:59.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:59.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:39:59.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:39:59.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:59.697 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:59.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:39:59.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:39:59.822 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:39:59.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:59.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:39:59.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:39:59.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:39:59.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:39:59.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:00.049 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:00.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:00.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:00.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:00.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:59 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:40:00.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:59 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:40:00.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:59 smithi190 ceph-mon[33748]: pgmap v580: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 880 B/s rd, 0 B/s wr, 526 op/s; 16200/98436 objects degraded (16.457%); 2121/98436 objects misplaced (2.155%); 11 MiB/s, 168 objects/s recovering 2024-03-24T12:40:00.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:59 smithi190 ceph-mon[33748]: 4.12 scrub starts 2024-03-24T12:40:00.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:59 smithi190 ceph-mon[33748]: 4.12 scrub ok 2024-03-24T12:40:00.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:59 smithi190 ceph-mon[33748]: osdmap e265: 8 total, 7 up, 8 in 2024-03-24T12:40:00.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:39:59 smithi190 ceph-mon[33748]: 6.1f scrub starts 2024-03-24T12:40:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:40:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:40:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[29919]: pgmap v580: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 880 B/s rd, 0 B/s wr, 526 op/s; 16200/98436 objects degraded (16.457%); 2121/98436 objects misplaced (2.155%); 11 MiB/s, 168 objects/s recovering 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[29919]: 4.12 scrub starts 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[29919]: 4.12 scrub ok 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[29919]: osdmap e265: 8 total, 7 up, 8 in 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[29919]: 6.1f scrub starts 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[41500]: pgmap v580: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 2.0 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 880 B/s rd, 0 B/s wr, 526 op/s; 16200/98436 objects degraded (16.457%); 2121/98436 objects misplaced (2.155%); 11 MiB/s, 168 objects/s recovering 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[41500]: 4.12 scrub starts 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[41500]: 4.12 scrub ok 2024-03-24T12:40:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[41500]: osdmap e265: 8 total, 7 up, 8 in 2024-03-24T12:40:00.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:39:59 smithi029 ceph-mon[41500]: 6.1f scrub starts 2024-03-24T12:40:00.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:00.212 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:00.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:00.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:00.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:00.342 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:00.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:00.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:00.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:00.484 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:00.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:00.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:00.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:00.626 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:00.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:00.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:00.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:00.751 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:00.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:00.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:00.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:00.878 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:00.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:00.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:00.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:01.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:01.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:01.148 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:01.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:01.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: 6.1f scrub ok 2024-03-24T12:40:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:40:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:40:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 16200/98436 objects degraded (16.457%), 16 pgs degraded, 34 pgs undersized 2024-03-24T12:40:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: [WRN] OSD_DOWN: 1 osds down 2024-03-24T12:40:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: osd.5 (root=default,host=smithi190) is down 2024-03-24T12:40:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: [WRN] PG_DEGRADED: Degraded data redundancy: 16200/98436 objects degraded (16.457%), 16 pgs degraded, 34 pgs undersized 2024-03-24T12:40:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 3.5 is stuck undersized for 3m, current state active+undersized, last acting [3,2] 2024-03-24T12:40:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 3.16 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,1] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 3.17 is stuck undersized for 3m, current state active+undersized, last acting [0,3] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 3.1c is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 3.1f is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 4.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,4] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 4.7 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,0] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 4.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,3] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 5.4 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 5.6 is stuck undersized for 3m, current state active+undersized, last acting [2,7] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 5.7 is stuck undersized for 3m, current state active+undersized, last acting [1,0] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 5.b is stuck undersized for 3m, current state active+undersized, last acting [2,0] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 5.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,7] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 5.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,0] 2024-03-24T12:40:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 5.12 is stuck undersized for 3m, current state active+undersized, last acting [1,3] 2024-03-24T12:40:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 5.16 is stuck undersized for 3m, current state active+undersized, last acting [3,1] 2024-03-24T12:40:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [0,7] 2024-03-24T12:40:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 6.4 is stuck undersized for 3m, current state active+undersized, last acting [1,3] 2024-03-24T12:40:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 6.7 is stuck undersized for 3m, current state active+undersized, last acting [3,4] 2024-03-24T12:40:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [1,0] 2024-03-24T12:40:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 6.10 is stuck undersized for 3m, current state active+undersized, last acting [0,1] 2024-03-24T12:40:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 6.11 is stuck undersized for 3m, current state active+undersized, last acting [3,0] 2024-03-24T12:40:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 6.17 is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-03-24T12:40:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 6.19 is stuck undersized for 3m, current state active+undersized, last acting [1,3] 2024-03-24T12:40:01.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: 6.1f scrub ok 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 16200/98436 objects degraded (16.457%), 16 pgs degraded, 34 pgs undersized 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: [WRN] OSD_DOWN: 1 osds down 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: osd.5 (root=default,host=smithi190) is down 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: [WRN] PG_DEGRADED: Degraded data redundancy: 16200/98436 objects degraded (16.457%), 16 pgs degraded, 34 pgs undersized 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 3.5 is stuck undersized for 3m, current state active+undersized, last acting [3,2] 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 3.16 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,1] 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 3.17 is stuck undersized for 3m, current state active+undersized, last acting [0,3] 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 3.1c is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 3.1f is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 4.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,4] 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-24T12:40:01.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 4.7 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,0] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 4.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,3] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 5.4 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 5.6 is stuck undersized for 3m, current state active+undersized, last acting [2,7] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 5.7 is stuck undersized for 3m, current state active+undersized, last acting [1,0] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 5.b is stuck undersized for 3m, current state active+undersized, last acting [2,0] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 5.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,7] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 5.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,0] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 5.12 is stuck undersized for 3m, current state active+undersized, last acting [1,3] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 5.16 is stuck undersized for 3m, current state active+undersized, last acting [3,1] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [0,7] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 6.4 is stuck undersized for 3m, current state active+undersized, last acting [1,3] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 6.7 is stuck undersized for 3m, current state active+undersized, last acting [3,4] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [1,0] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 6.10 is stuck undersized for 3m, current state active+undersized, last acting [0,1] 2024-03-24T12:40:01.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 6.11 is stuck undersized for 3m, current state active+undersized, last acting [3,0] 2024-03-24T12:40:01.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 6.17 is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-03-24T12:40:01.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 6.19 is stuck undersized for 3m, current state active+undersized, last acting [1,3] 2024-03-24T12:40:01.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 6.1a is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-03-24T12:40:01.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 6.1c is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,2] 2024-03-24T12:40:01.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 9.5 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-24T12:40:01.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 9.6 is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,1] 2024-03-24T12:40:01.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 9.7 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,3] 2024-03-24T12:40:01.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 9.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [3,0] 2024-03-24T12:40:01.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 9.9 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-24T12:40:01.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 9.a is active+undersized+degraded, acting [7,1] 2024-03-24T12:40:01.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 9.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,1] 2024-03-24T12:40:01.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: pg 9.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,3] 2024-03-24T12:40:01.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 16200/98436 objects degraded (16.457%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:01.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[29919]: osdmap e266: 8 total, 7 up, 8 in 2024-03-24T12:40:01.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 6.1a is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-03-24T12:40:01.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 6.1c is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,2] 2024-03-24T12:40:01.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 9.5 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-24T12:40:01.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 9.6 is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,1] 2024-03-24T12:40:01.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 9.7 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,3] 2024-03-24T12:40:01.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 9.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [3,0] 2024-03-24T12:40:01.175 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 9.9 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-24T12:40:01.175 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 9.a is active+undersized+degraded, acting [7,1] 2024-03-24T12:40:01.175 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 9.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,1] 2024-03-24T12:40:01.175 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: pg 9.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,3] 2024-03-24T12:40:01.175 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 16200/98436 objects degraded (16.457%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:01.175 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:00 smithi029 ceph-mon[41500]: osdmap e266: 8 total, 7 up, 8 in 2024-03-24T12:40:01.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:01.286 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:01.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: 6.1f scrub ok 2024-03-24T12:40:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 16200/98436 objects degraded (16.457%), 16 pgs degraded, 34 pgs undersized 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: [WRN] OSD_DOWN: 1 osds down 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: osd.5 (root=default,host=smithi190) is down 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: [WRN] PG_DEGRADED: Degraded data redundancy: 16200/98436 objects degraded (16.457%), 16 pgs degraded, 34 pgs undersized 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 3.5 is stuck undersized for 3m, current state active+undersized, last acting [3,2] 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 3.16 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,1] 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 3.17 is stuck undersized for 3m, current state active+undersized, last acting [0,3] 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 3.1c is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 3.1f is stuck undersized for 3m, current state active+undersized, last acting [0,2] 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 4.2 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,4] 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 4.3 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-24T12:40:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 4.7 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,0] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 4.15 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,3] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 5.4 is stuck undersized for 3m, current state active+undersized, last acting [7,2] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 5.6 is stuck undersized for 3m, current state active+undersized, last acting [2,7] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 5.7 is stuck undersized for 3m, current state active+undersized, last acting [1,0] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 5.b is stuck undersized for 3m, current state active+undersized, last acting [2,0] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 5.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,7] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 5.e is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,0] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 5.12 is stuck undersized for 3m, current state active+undersized, last acting [1,3] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 5.16 is stuck undersized for 3m, current state active+undersized, last acting [3,1] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 5.1b is stuck undersized for 3m, current state active+undersized, last acting [0,7] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 6.4 is stuck undersized for 3m, current state active+undersized, last acting [1,3] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 6.7 is stuck undersized for 3m, current state active+undersized, last acting [3,4] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 6.d is stuck undersized for 3m, current state active+undersized, last acting [1,0] 2024-03-24T12:40:01.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 6.10 is stuck undersized for 3m, current state active+undersized, last acting [0,1] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 6.11 is stuck undersized for 3m, current state active+undersized, last acting [3,0] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 6.17 is stuck undersized for 3m, current state active+undersized, last acting [4,2] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 6.19 is stuck undersized for 3m, current state active+undersized, last acting [1,3] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 6.1a is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 6.1c is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,2] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 9.5 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 9.6 is stuck undersized for 3m, current state active+undersized+degraded, last acting [2,1] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 9.7 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,3] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 9.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [3,0] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 9.9 is stuck undersized for 3m, current state active+undersized+degraded, last acting [0,7] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 9.a is active+undersized+degraded, acting [7,1] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 9.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,1] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: pg 9.f is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,3] 2024-03-24T12:40:01.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 16200/98436 objects degraded (16.457%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:01.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:00 smithi190 ceph-mon[33748]: osdmap e266: 8 total, 7 up, 8 in 2024-03-24T12:40:01.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:01.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:01.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:01.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:01.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:01.561 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:01.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:01.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:01.699 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:01.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:01.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:01.847 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:01.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:01.876 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:40:01.883 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:40:01.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:01.989 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:01.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:01.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:02.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:02.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:02.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:02.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:01 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:40:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:01 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:40:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:01 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:40:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:01 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:40:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:01 smithi029 ceph-mon[41500]: pgmap v583: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 881 B/s rd, 0 B/s wr, 709 op/s; 15483/92685 objects degraded (16.705%); 2121/92685 objects misplaced (2.288%); 4.5 MiB/s, 72 objects/s recovering 2024-03-24T12:40:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:01 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:40:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:01 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:40:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:01 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:40:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:01 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:40:02.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:01 smithi029 ceph-mon[29919]: pgmap v583: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 881 B/s rd, 0 B/s wr, 709 op/s; 15483/92685 objects degraded (16.705%); 2121/92685 objects misplaced (2.288%); 4.5 MiB/s, 72 objects/s recovering 2024-03-24T12:40:02.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:02.243 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:02.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:02.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:02.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:02.373 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:02.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:02.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:01 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:40:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:01 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:40:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:01 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:40:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:01 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:40:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:01 smithi190 ceph-mon[33748]: pgmap v583: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.9 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 881 B/s rd, 0 B/s wr, 709 op/s; 15483/92685 objects degraded (16.705%); 2121/92685 objects misplaced (2.288%); 4.5 MiB/s, 72 objects/s recovering 2024-03-24T12:40:02.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:02.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:02.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:02.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:02.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:02.644 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:02.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:02.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:02.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:02.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:02.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:02.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:02.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:03.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:03.062 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:03.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:03.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:03.192 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:03.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:03.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:03.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:03.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:03.335 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:03.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:03.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:02 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:40:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:02 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:40:03.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:02 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:03.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:02 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:40:03.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:02 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:40:03.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:02 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:03.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:02 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:40:03.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:02 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:40:03.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:02 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:03.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:03.480 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:03.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:03.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:03.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:03.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:03.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:03.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:03.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:03.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:03.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:03.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:03.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:03.867 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:03.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:03.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:03.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:03.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:03.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:04.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:04.129 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:04.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:04.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:04.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:04.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:04.259 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:04.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:04.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:03 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:40:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:03 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:40:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:03 smithi190 ceph-mon[33748]: pgmap v584: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 730 op/s; 14191/86274 objects degraded (16.449%); 1847/86274 objects misplaced (2.141%) 2024-03-24T12:40:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:03 smithi190 ceph-mon[33748]: 3.1a scrub starts 2024-03-24T12:40:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:03 smithi190 ceph-mon[33748]: 3.1a scrub ok 2024-03-24T12:40:04.397 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:04.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:04.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:03 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:40:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:03 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:40:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:03 smithi029 ceph-mon[41500]: pgmap v584: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 730 op/s; 14191/86274 objects degraded (16.449%); 1847/86274 objects misplaced (2.141%) 2024-03-24T12:40:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:03 smithi029 ceph-mon[41500]: 3.1a scrub starts 2024-03-24T12:40:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:03 smithi029 ceph-mon[41500]: 3.1a scrub ok 2024-03-24T12:40:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:03 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:40:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:03 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:40:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:03 smithi029 ceph-mon[29919]: pgmap v584: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 730 op/s; 14191/86274 objects degraded (16.449%); 1847/86274 objects misplaced (2.141%) 2024-03-24T12:40:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:03 smithi029 ceph-mon[29919]: 3.1a scrub starts 2024-03-24T12:40:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:03 smithi029 ceph-mon[29919]: 3.1a scrub ok 2024-03-24T12:40:04.500 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:04.542 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:04.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:04.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:04.666 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:04.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:04.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:04.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:04.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:04.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:04.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:05.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:05.060 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:05.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:05.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:05.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:05.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:05.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:05.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:05.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:05.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:05.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:05.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:04 smithi190 ceph-mon[33748]: 3.c scrub starts 2024-03-24T12:40:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:04 smithi190 ceph-mon[33748]: 3.c scrub ok 2024-03-24T12:40:05.417 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:05.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:04 smithi029 ceph-mon[41500]: 3.c scrub starts 2024-03-24T12:40:05.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:04 smithi029 ceph-mon[41500]: 3.c scrub ok 2024-03-24T12:40:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:04 smithi029 ceph-mon[29919]: 3.c scrub starts 2024-03-24T12:40:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:04 smithi029 ceph-mon[29919]: 3.c scrub ok 2024-03-24T12:40:05.460 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:05.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:05.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:05.562 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:05.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:05.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:05.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:05.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:05.719 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:05.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:05.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:05.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:05.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:05.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:05.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:05.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:05.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:05.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:06.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:06.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:06.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:06.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:06.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:05 smithi029 ceph-mon[41500]: pgmap v585: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 651 op/s; 13912/85437 objects degraded (16.283%); 1847/85437 objects misplaced (2.162%) 2024-03-24T12:40:06.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:05 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 13912/85437 objects degraded (16.283%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:06.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:05 smithi029 ceph-mon[29919]: pgmap v585: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 651 op/s; 13912/85437 objects degraded (16.283%); 1847/85437 objects misplaced (2.162%) 2024-03-24T12:40:06.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:05 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 13912/85437 objects degraded (16.283%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:06.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:06.250 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:06.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:06.265 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":568,"stamp":"2024-03-24T12:39:45.824402+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2809862518,"num_objects":43065,"num_object_clones":0,"num_object_copies":129195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21607,"num_objects_misplaced":5543,"num_objects_unfound":0,"num_objects_dirty":43065,"num_whiteouts":0,"num_read":2165,"num_read_kb":2011,"num_write":182162,"num_write_kb":7174494,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21718,"num_bytes_recovered":1420890554,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79285,"ondisk_log_size":79285,"up":388,"acting":406,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":465,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11140892,"kb_used_data":7721820,"kb_used_omap":12,"kb_used_meta":3418803,"kb_avail":738820324,"statfs":{"total":767960285184,"available":756552011776,"internally_reserved":0,"allocated":7907143680,"data_stored":7902582129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3500854833},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-607125504,"num_objects":-9264,"num_object_clones":0,"num_object_copies":-27792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4456,"num_objects_misplaced":2269,"num_objects_unfound":0,"num_objects_dirty":-9264,"num_whiteouts":0,"num_read":13,"num_read_kb":13,"num_write":9264,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2968,"num_bytes_recovered":194576928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.146299"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":421,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.112567+0000","last_change":"2024-03-24T12:35:58.701806+0000","last_active":"2024-03-24T12:39:37.112567+0000","last_peered":"2024-03-24T12:39:37.112567+0000","last_clean":"2024-03-24T12:35:54.676897+0000","last_became_active":"2024-03-24T12:35:58.701806+0000","last_became_peered":"2024-03-24T12:35:58.701806+0000","last_unstale":"2024-03-24T12:39:37.112567+0000","last_undegraded":"2024-03-24T12:39:37.112567+0000","last_fullsized":"2024-03-24T12:35:58.695795+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:08.859869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.859869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":341,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.121415+0000","last_change":"2024-03-24T12:39:07.750122+0000","last_active":"2024-03-24T12:39:37.121415+0000","last_peered":"2024-03-24T12:39:37.121415+0000","last_clean":"2024-03-24T12:35:54.687134+0000","last_became_active":"2024-03-24T12:39:07.750122+0000","last_became_peered":"2024-03-24T12:39:07.750122+0000","last_unstale":"2024-03-24T12:39:37.121415+0000","last_undegraded":"2024-03-24T12:39:37.121415+0000","last_fullsized":"2024-03-24T12:39:07.746192+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:48.744937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.744937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":426,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.119051+0000","last_change":"2024-03-24T12:35:58.712452+0000","last_active":"2024-03-24T12:39:37.119051+0000","last_peered":"2024-03-24T12:39:37.119051+0000","last_clean":"2024-03-24T12:35:54.683285+0000","last_became_active":"2024-03-24T12:35:58.712452+0000","last_became_peered":"2024-03-24T12:35:58.712452+0000","last_unstale":"2024-03-24T12:39:37.119051+0000","last_undegraded":"2024-03-24T12:39:37.119051+0000","last_fullsized":"2024-03-24T12:35:58.707789+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:14.966328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:14.966328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.960939+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"207'15","reported_seq":491,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.112299+0000","last_change":"2024-03-24T12:39:34.765696+0000","last_active":"2024-03-24T12:39:37.112299+0000","last_peered":"2024-03-24T12:39:37.112299+0000","last_clean":"2024-03-24T12:39:37.112299+0000","last_became_active":"2024-03-24T12:39:34.752258+0000","last_became_peered":"2024-03-24T12:39:34.752258+0000","last_unstale":"2024-03-24T12:39:37.112299+0000","last_undegraded":"2024-03-24T12:39:37.112299+0000","last_fullsized":"2024-03-24T12:39:37.112299+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:37:31.757831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:31.757831+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.857712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":430,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.124054+0000","last_change":"2024-03-24T12:39:34.758724+0000","last_active":"2024-03-24T12:39:37.124054+0000","last_peered":"2024-03-24T12:39:37.124054+0000","last_clean":"2024-03-24T12:39:37.124054+0000","last_became_active":"2024-03-24T12:39:34.758425+0000","last_became_peered":"2024-03-24T12:39:34.758425+0000","last_unstale":"2024-03-24T12:39:37.124054+0000","last_undegraded":"2024-03-24T12:39:37.124054+0000","last_fullsized":"2024-03-24T12:39:37.124054+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:24.826030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:24.826030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":442,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.112360+0000","last_change":"2024-03-24T12:39:34.744518+0000","last_active":"2024-03-24T12:39:37.112360+0000","last_peered":"2024-03-24T12:39:37.112360+0000","last_clean":"2024-03-24T12:39:37.112360+0000","last_became_active":"2024-03-24T12:39:34.744030+0000","last_became_peered":"2024-03-24T12:39:34.744030+0000","last_unstale":"2024-03-24T12:39:37.112360+0000","last_undegraded":"2024-03-24T12:39:37.112360+0000","last_fullsized":"2024-03-24T12:39:37.112360+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:22.803996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:22.803996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.857657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":443,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.126174+0000","last_change":"2024-03-24T12:39:34.761525+0000","last_active":"2024-03-24T12:39:37.126174+0000","last_peered":"2024-03-24T12:39:37.126174+0000","last_clean":"2024-03-24T12:39:37.126174+0000","last_became_active":"2024-03-24T12:39:34.761198+0000","last_became_peered":"2024-03-24T12:39:34.761198+0000","last_unstale":"2024-03-24T12:39:37.126174+0000","last_undegraded":"2024-03-24T12:39:37.126174+0000","last_fullsized":"2024-03-24T12:39:37.126174+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:12.693343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:12.693343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"207'34","reported_seq":476,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.126324+0000","last_change":"2024-03-24T12:37:09.518641+0000","last_active":"2024-03-24T12:39:37.126324+0000","last_peered":"2024-03-24T12:39:37.126324+0000","last_clean":"2024-03-24T12:39:37.126324+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:39:37.126324+0000","last_undegraded":"2024-03-24T12:39:37.126324+0000","last_fullsized":"2024-03-24T12:39:37.126324+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-24T12:37:09.517668+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:37:09.517668+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":426,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.118258+0000","last_change":"2024-03-24T12:36:59.693310+0000","last_active":"2024-03-24T12:39:37.118258+0000","last_peered":"2024-03-24T12:39:37.118258+0000","last_clean":"2024-03-24T12:39:37.118258+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:39:37.118258+0000","last_undegraded":"2024-03-24T12:39:37.118258+0000","last_fullsized":"2024-03-24T12:39:37.118258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:59.693187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:36:59.693187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968269+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":117,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.118978+0000","last_change":"2024-03-24T12:35:58.711151+0000","last_active":"2024-03-24T12:39:37.118978+0000","last_peered":"2024-03-24T12:39:37.118978+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.711151+0000","last_became_peered":"2024-03-24T12:35:58.711151+0000","last_unstale":"2024-03-24T12:39:37.118978+0000","last_undegraded":"2024-03-24T12:39:37.118978+0000","last_fullsized":"2024-03-24T12:35:58.703392+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:44.161505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.161505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.960882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":134,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:40.790035+0000","last_change":"2024-03-24T12:39:40.790112+0000","last_active":"2024-03-24T12:39:40.790035+0000","last_peered":"2024-03-24T12:39:40.790035+0000","last_clean":"2024-03-24T12:39:40.790035+0000","last_became_active":"2024-03-24T12:39:35.101206+0000","last_became_peered":"2024-03-24T12:39:35.101206+0000","last_unstale":"2024-03-24T12:39:40.790035+0000","last_undegraded":"2024-03-24T12:39:40.790035+0000","last_fullsized":"2024-03-24T12:39:40.790035+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:40.789901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:39:40.789901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:58.046154+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"207'15","reported_seq":459,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.112749+0000","last_change":"2024-03-24T12:37:47.713933+0000","last_active":"2024-03-24T12:39:37.112749+0000","last_peered":"2024-03-24T12:39:37.112749+0000","last_clean":"2024-03-24T12:39:37.112749+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:39:37.112749+0000","last_undegraded":"2024-03-24T12:39:37.112749+0000","last_fullsized":"2024-03-24T12:39:37.112749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:37:47.713795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:47.713795+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:38.818464+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":116,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.130299+0000","last_change":"2024-03-24T12:35:58.718922+0000","last_active":"2024-03-24T12:39:37.130299+0000","last_peered":"2024-03-24T12:39:37.130299+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.718922+0000","last_became_peered":"2024-03-24T12:35:58.718922+0000","last_unstale":"2024-03-24T12:39:37.130299+0000","last_undegraded":"2024-03-24T12:39:37.130299+0000","last_fullsized":"2024-03-24T12:35:58.702792+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.155345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.155345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.183780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":430,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.118924+0000","last_change":"2024-03-24T12:37:30.157827+0000","last_active":"2024-03-24T12:39:37.118924+0000","last_peered":"2024-03-24T12:39:37.118924+0000","last_clean":"2024-03-24T12:39:37.118924+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:39:37.118924+0000","last_undegraded":"2024-03-24T12:39:37.118924+0000","last_fullsized":"2024-03-24T12:39:37.118924+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:30.157717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:37:30.157717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":116,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.112749+0000","last_change":"2024-03-24T12:35:58.708742+0000","last_active":"2024-03-24T12:39:37.112749+0000","last_peered":"2024-03-24T12:39:37.112749+0000","last_clean":"2024-03-24T12:26:02.143860+0000","last_became_active":"2024-03-24T12:35:58.708742+0000","last_became_peered":"2024-03-24T12:35:58.708742+0000","last_unstale":"2024-03-24T12:39:37.112749+0000","last_undegraded":"2024-03-24T12:39:37.112749+0000","last_fullsized":"2024-03-24T12:35:58.699613+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:23.095706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:33:23.095706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"207'7","reported_seq":447,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.112859+0000","last_change":"2024-03-24T12:36:55.749365+0000","last_active":"2024-03-24T12:39:37.112859+0000","last_peered":"2024-03-24T12:39:37.112859+0000","last_clean":"2024-03-24T12:39:37.112859+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:39:37.112859+0000","last_undegraded":"2024-03-24T12:39:37.112859+0000","last_fullsized":"2024-03-24T12:39:37.112859+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-24T12:36:55.749205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:36:55.749205+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":440,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:41.822477+0000","last_change":"2024-03-24T12:39:41.822559+0000","last_active":"2024-03-24T12:39:41.822477+0000","last_peered":"2024-03-24T12:39:41.822477+0000","last_clean":"2024-03-24T12:39:41.822477+0000","last_became_active":"2024-03-24T12:39:35.101660+0000","last_became_peered":"2024-03-24T12:39:35.101660+0000","last_unstale":"2024-03-24T12:39:41.822477+0000","last_undegraded":"2024-03-24T12:39:41.822477+0000","last_fullsized":"2024-03-24T12:39:41.822477+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:41.822379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:39:41.822379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:01.137843+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,3],"avail_no_missing":["4","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":446,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.132092+0000","last_change":"2024-03-24T12:37:34.824331+0000","last_active":"2024-03-24T12:39:37.132092+0000","last_peered":"2024-03-24T12:39:37.132092+0000","last_clean":"2024-03-24T12:39:37.132092+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:39:37.132092+0000","last_undegraded":"2024-03-24T12:39:37.132092+0000","last_fullsized":"2024-03-24T12:39:37.132092+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:34.824088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:34.824088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.183851+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":438,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119821+0000","last_change":"2024-03-24T12:38:04.474810+0000","last_active":"2024-03-24T12:39:37.119821+0000","last_peered":"2024-03-24T12:39:37.119821+0000","last_clean":"2024-03-24T12:39:37.119821+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:39:37.119821+0000","last_undegraded":"2024-03-24T12:39:37.119821+0000","last_fullsized":"2024-03-24T12:39:37.119821+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:38:04.474685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:38:04.474685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"207'27","reported_seq":485,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:42.561774+0000","last_change":"2024-03-24T12:39:42.561825+0000","last_active":"2024-03-24T12:39:42.561774+0000","last_peered":"2024-03-24T12:39:42.561774+0000","last_clean":"2024-03-24T12:39:42.561774+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:39:42.561774+0000","last_undegraded":"2024-03-24T12:39:42.561774+0000","last_fullsized":"2024-03-24T12:39:42.561774+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-24T12:39:42.561693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:39:42.561693+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.995842+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":424,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.131278+0000","last_change":"2024-03-24T12:39:35.130875+0000","last_active":"2024-03-24T12:39:37.131278+0000","last_peered":"2024-03-24T12:39:37.131278+0000","last_clean":"2024-03-24T12:39:37.131278+0000","last_became_active":"2024-03-24T12:39:35.129797+0000","last_became_peered":"2024-03-24T12:39:35.129797+0000","last_unstale":"2024-03-24T12:39:37.131278+0000","last_undegraded":"2024-03-24T12:39:37.131278+0000","last_fullsized":"2024-03-24T12:39:37.131278+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:51.927644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:51.927644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":91,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":430,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.113259+0000","last_change":"2024-03-24T12:37:11.692320+0000","last_active":"2024-03-24T12:39:37.113259+0000","last_peered":"2024-03-24T12:39:37.113259+0000","last_clean":"2024-03-24T12:39:37.113259+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:39:37.113259+0000","last_undegraded":"2024-03-24T12:39:37.113259+0000","last_fullsized":"2024-03-24T12:39:37.113259+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:11.692219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:37:11.692219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":438,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.112814+0000","last_change":"2024-03-24T12:37:26.794282+0000","last_active":"2024-03-24T12:39:37.112814+0000","last_peered":"2024-03-24T12:39:37.112814+0000","last_clean":"2024-03-24T12:39:37.112814+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:39:37.112814+0000","last_undegraded":"2024-03-24T12:39:37.112814+0000","last_fullsized":"2024-03-24T12:39:37.112814+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:26.794181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:26.794181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"207'18","reported_seq":458,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.124500+0000","last_change":"2024-03-24T12:37:34.887625+0000","last_active":"2024-03-24T12:39:37.124500+0000","last_peered":"2024-03-24T12:39:37.124500+0000","last_clean":"2024-03-24T12:39:37.124500+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:39:37.124500+0000","last_undegraded":"2024-03-24T12:39:37.124500+0000","last_fullsized":"2024-03-24T12:39:37.124500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-24T12:37:34.887491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:34.887491+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":414,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:37.119508+0000","last_change":"2024-03-24T12:35:58.711430+0000","last_active":"2024-03-24T12:39:37.119508+0000","last_peered":"2024-03-24T12:39:37.119508+0000","last_clean":"2024-03-24T12:35:54.688013+0000","last_became_active":"2024-03-24T12:35:58.711430+0000","last_became_peered":"2024-03-24T12:35:58.711430+0000","last_unstale":"2024-03-24T12:39:37.119508+0000","last_undegraded":"2024-03-24T12:35:58.697902+0000","last_fullsized":"2024-03-24T12:35:58.697784+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:34:42.820487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:42.820487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"258'10","reported_seq":22,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.595165+0000","last_change":"2024-03-24T12:39:34.758535+0000","last_active":"2024-03-24T12:39:37.595165+0000","last_peered":"2024-03-24T12:39:37.595165+0000","last_clean":"2024-03-24T12:39:37.595165+0000","last_became_active":"2024-03-24T12:39:34.750827+0000","last_became_peered":"2024-03-24T12:39:34.750827+0000","last_unstale":"2024-03-24T12:39:37.595165+0000","last_undegraded":"2024-03-24T12:39:37.595165+0000","last_fullsized":"2024-03-24T12:39:37.595165+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:37:46.975910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:46.975910+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":447,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.130048+0000","last_change":"2024-03-24T12:37:22.038277+0000","last_active":"2024-03-24T12:39:37.130048+0000","last_peered":"2024-03-24T12:39:37.130048+0000","last_clean":"2024-03-24T12:39:37.130048+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:39:37.130048+0000","last_undegraded":"2024-03-24T12:39:37.130048+0000","last_fullsized":"2024-03-24T12:39:37.130048+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:37:22.038139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:22.038139+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.183835+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"207'20","reported_seq":465,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119253+0000","last_change":"2024-03-24T12:37:07.473900+0000","last_active":"2024-03-24T12:39:37.119253+0000","last_peered":"2024-03-24T12:39:37.119253+0000","last_clean":"2024-03-24T12:39:37.119253+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:39:37.119253+0000","last_undegraded":"2024-03-24T12:39:37.119253+0000","last_fullsized":"2024-03-24T12:39:37.119253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:37:07.473758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:07.473758+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968423+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":336,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.112818+0000","last_change":"2024-03-24T12:39:07.748878+0000","last_active":"2024-03-24T12:39:37.112818+0000","last_peered":"2024-03-24T12:39:37.112818+0000","last_clean":"2024-03-24T12:35:54.688035+0000","last_became_active":"2024-03-24T12:39:07.748878+0000","last_became_peered":"2024-03-24T12:39:07.748878+0000","last_unstale":"2024-03-24T12:39:37.112818+0000","last_undegraded":"2024-03-24T12:39:37.112818+0000","last_fullsized":"2024-03-24T12:39:07.746317+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:03.555975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.555975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":423,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119516+0000","last_change":"2024-03-24T12:37:21.521384+0000","last_active":"2024-03-24T12:39:37.119516+0000","last_peered":"2024-03-24T12:39:37.119516+0000","last_clean":"2024-03-24T12:39:37.119516+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:39:37.119516+0000","last_undegraded":"2024-03-24T12:39:37.119516+0000","last_fullsized":"2024-03-24T12:39:37.119516+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:21.521268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:21.521268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968542+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":430,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.123751+0000","last_change":"2024-03-24T12:37:08.569047+0000","last_active":"2024-03-24T12:39:37.123751+0000","last_peered":"2024-03-24T12:39:37.123751+0000","last_clean":"2024-03-24T12:39:37.123751+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:39:37.123751+0000","last_undegraded":"2024-03-24T12:39:37.123751+0000","last_fullsized":"2024-03-24T12:39:37.123751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:08.568803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:37:08.568803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"207'19","reported_seq":421,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.120687+0000","last_change":"2024-03-24T12:39:35.107698+0000","last_active":"2024-03-24T12:39:37.120687+0000","last_peered":"2024-03-24T12:39:37.120687+0000","last_clean":"2024-03-24T12:39:37.120687+0000","last_became_active":"2024-03-24T12:39:35.107173+0000","last_became_peered":"2024-03-24T12:39:35.107173+0000","last_unstale":"2024-03-24T12:39:37.120687+0000","last_undegraded":"2024-03-24T12:39:37.120687+0000","last_fullsized":"2024-03-24T12:39:37.120687+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-24T12:34:43.138814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:34:43.138814+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1,0],"avail_no_missing":["6","0","1"],"object_location_counts":[{"shards":"0,1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":426,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.118788+0000","last_change":"2024-03-24T12:37:43.192667+0000","last_active":"2024-03-24T12:39:37.118788+0000","last_peered":"2024-03-24T12:39:37.118788+0000","last_clean":"2024-03-24T12:39:37.118788+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:39:37.118788+0000","last_undegraded":"2024-03-24T12:39:37.118788+0000","last_fullsized":"2024-03-24T12:39:37.118788+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:43.192575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:43.192575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":17,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.120374+0000","last_change":"2024-03-24T12:39:34.751413+0000","last_active":"2024-03-24T12:39:37.120374+0000","last_peered":"2024-03-24T12:39:37.120374+0000","last_clean":"2024-03-24T12:39:37.120374+0000","last_became_active":"2024-03-24T12:39:34.751127+0000","last_became_peered":"2024-03-24T12:39:34.751127+0000","last_unstale":"2024-03-24T12:39:37.120374+0000","last_undegraded":"2024-03-24T12:39:37.120374+0000","last_fullsized":"2024-03-24T12:39:37.120374+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:05.081968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:05.081968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":425,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.119239+0000","last_change":"2024-03-24T12:35:58.718236+0000","last_active":"2024-03-24T12:39:37.119239+0000","last_peered":"2024-03-24T12:39:37.119239+0000","last_clean":"2024-03-24T12:35:54.683418+0000","last_became_active":"2024-03-24T12:35:58.718236+0000","last_became_peered":"2024-03-24T12:35:58.718236+0000","last_unstale":"2024-03-24T12:39:37.119239+0000","last_undegraded":"2024-03-24T12:39:37.119239+0000","last_fullsized":"2024-03-24T12:35:58.708687+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.720633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.720633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.960952+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"207'8","reported_seq":462,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.123843+0000","last_change":"2024-03-24T12:39:34.768974+0000","last_active":"2024-03-24T12:39:37.123843+0000","last_peered":"2024-03-24T12:39:37.123843+0000","last_clean":"2024-03-24T12:39:37.123843+0000","last_became_active":"2024-03-24T12:39:34.759619+0000","last_became_peered":"2024-03-24T12:39:34.759619+0000","last_unstale":"2024-03-24T12:39:37.123843+0000","last_undegraded":"2024-03-24T12:39:37.123843+0000","last_fullsized":"2024-03-24T12:39:37.123843+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-24T12:37:01.416468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:01.416468+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":430,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.124409+0000","last_change":"2024-03-24T12:39:34.763955+0000","last_active":"2024-03-24T12:39:37.124409+0000","last_peered":"2024-03-24T12:39:37.124409+0000","last_clean":"2024-03-24T12:39:37.124409+0000","last_became_active":"2024-03-24T12:39:34.763422+0000","last_became_peered":"2024-03-24T12:39:34.763422+0000","last_unstale":"2024-03-24T12:39:37.124409+0000","last_undegraded":"2024-03-24T12:39:37.124409+0000","last_fullsized":"2024-03-24T12:39:37.124409+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:21.837500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:37:21.837500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":442,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119551+0000","last_change":"2024-03-24T12:39:34.752852+0000","last_active":"2024-03-24T12:39:37.119551+0000","last_peered":"2024-03-24T12:39:37.119551+0000","last_clean":"2024-03-24T12:39:37.119551+0000","last_became_active":"2024-03-24T12:39:34.752397+0000","last_became_peered":"2024-03-24T12:39:34.752397+0000","last_unstale":"2024-03-24T12:39:37.119551+0000","last_undegraded":"2024-03-24T12:39:37.119551+0000","last_fullsized":"2024-03-24T12:39:37.119551+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:24.170677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:24.170677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:40.219268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":124,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:37.119506+0000","last_change":"2024-03-24T12:35:58.708757+0000","last_active":"2024-03-24T12:39:37.119506+0000","last_peered":"2024-03-24T12:39:37.119506+0000","last_clean":"2024-03-24T12:26:03.166952+0000","last_became_active":"2024-03-24T12:35:58.708757+0000","last_became_peered":"2024-03-24T12:35:58.708757+0000","last_unstale":"2024-03-24T12:39:37.119506+0000","last_undegraded":"2024-03-24T12:35:58.698998+0000","last_fullsized":"2024-03-24T12:35:58.698185+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:28.059288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.059288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"207'19","reported_seq":468,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.126231+0000","last_change":"2024-03-24T12:39:34.751709+0000","last_active":"2024-03-24T12:39:37.126231+0000","last_peered":"2024-03-24T12:39:37.126231+0000","last_clean":"2024-03-24T12:39:37.126231+0000","last_became_active":"2024-03-24T12:39:34.750031+0000","last_became_peered":"2024-03-24T12:39:34.750031+0000","last_unstale":"2024-03-24T12:39:37.126231+0000","last_undegraded":"2024-03-24T12:39:37.126231+0000","last_fullsized":"2024-03-24T12:39:37.126231+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:37:19.728825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:19.728825+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":413,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.117867+0000","last_change":"2024-03-24T12:35:58.706939+0000","last_active":"2024-03-24T12:39:37.117867+0000","last_peered":"2024-03-24T12:39:37.117867+0000","last_clean":"2024-03-24T12:35:54.681672+0000","last_became_active":"2024-03-24T12:35:58.706939+0000","last_became_peered":"2024-03-24T12:35:58.706939+0000","last_unstale":"2024-03-24T12:39:37.117867+0000","last_undegraded":"2024-03-24T12:39:37.117867+0000","last_fullsized":"2024-03-24T12:35:58.695715+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:53.895902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.895902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.960859+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":422,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.123968+0000","last_change":"2024-03-24T12:37:04.438187+0000","last_active":"2024-03-24T12:39:37.123968+0000","last_peered":"2024-03-24T12:39:37.123968+0000","last_clean":"2024-03-24T12:39:37.123968+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:39:37.123968+0000","last_undegraded":"2024-03-24T12:39:37.123968+0000","last_fullsized":"2024-03-24T12:39:37.123968+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:04.438083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_clean_scrub_stamp":"2024-03-24T12:37:04.438083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":438,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.120812+0000","last_change":"2024-03-24T12:37:19.143611+0000","last_active":"2024-03-24T12:39:37.120812+0000","last_peered":"2024-03-24T12:39:37.120812+0000","last_clean":"2024-03-24T12:39:37.120812+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:39:37.120812+0000","last_undegraded":"2024-03-24T12:39:37.120812+0000","last_fullsized":"2024-03-24T12:39:37.120812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:19.143536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:19.143536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"207'15","reported_seq":120,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.120381+0000","last_change":"2024-03-24T12:39:35.104205+0000","last_active":"2024-03-24T12:39:37.120381+0000","last_peered":"2024-03-24T12:39:37.120381+0000","last_clean":"2024-03-24T12:39:37.120381+0000","last_became_active":"2024-03-24T12:39:35.103739+0000","last_became_peered":"2024-03-24T12:39:35.103739+0000","last_unstale":"2024-03-24T12:39:37.120381+0000","last_undegraded":"2024-03-24T12:39:37.120381+0000","last_fullsized":"2024-03-24T12:39:37.120381+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-24T12:35:02.976915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:35:02.976915+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2,0],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":413,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.125840+0000","last_change":"2024-03-24T12:35:58.714875+0000","last_active":"2024-03-24T12:39:37.125840+0000","last_peered":"2024-03-24T12:39:37.125840+0000","last_clean":"2024-03-24T12:35:54.689143+0000","last_became_active":"2024-03-24T12:35:58.714875+0000","last_became_peered":"2024-03-24T12:35:58.714875+0000","last_unstale":"2024-03-24T12:39:37.125840+0000","last_undegraded":"2024-03-24T12:39:37.125840+0000","last_fullsized":"2024-03-24T12:35:58.702457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.898988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.898988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":425,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.138938+0000","last_change":"2024-03-24T12:35:58.718562+0000","last_active":"2024-03-24T12:39:37.138938+0000","last_peered":"2024-03-24T12:39:37.138938+0000","last_clean":"2024-03-24T12:35:54.687557+0000","last_became_active":"2024-03-24T12:35:58.718562+0000","last_became_peered":"2024-03-24T12:35:58.718562+0000","last_unstale":"2024-03-24T12:39:37.138938+0000","last_undegraded":"2024-03-24T12:39:37.138938+0000","last_fullsized":"2024-03-24T12:35:58.697301+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.144883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.144883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.183771+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":458,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.112942+0000","last_change":"2024-03-24T12:39:34.748732+0000","last_active":"2024-03-24T12:39:37.112942+0000","last_peered":"2024-03-24T12:39:37.112942+0000","last_clean":"2024-03-24T12:39:37.112942+0000","last_became_active":"2024-03-24T12:39:34.748131+0000","last_became_peered":"2024-03-24T12:39:34.748131+0000","last_unstale":"2024-03-24T12:39:37.112942+0000","last_undegraded":"2024-03-24T12:39:37.112942+0000","last_fullsized":"2024-03-24T12:39:37.112942+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:36.815404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:36.815404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.857746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"207'24","reported_seq":488,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.113190+0000","last_change":"2024-03-24T12:39:35.102197+0000","last_active":"2024-03-24T12:39:37.113190+0000","last_peered":"2024-03-24T12:39:37.113190+0000","last_clean":"2024-03-24T12:39:37.113190+0000","last_became_active":"2024-03-24T12:39:35.101521+0000","last_became_peered":"2024-03-24T12:39:35.101521+0000","last_unstale":"2024-03-24T12:39:37.113190+0000","last_undegraded":"2024-03-24T12:39:37.113190+0000","last_fullsized":"2024-03-24T12:39:37.113190+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-24T12:37:05.778387+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:37:05.778387+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":419,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.118880+0000","last_change":"2024-03-24T12:37:10.768815+0000","last_active":"2024-03-24T12:39:37.118880+0000","last_peered":"2024-03-24T12:39:37.118880+0000","last_clean":"2024-03-24T12:39:37.118880+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:39:37.118880+0000","last_undegraded":"2024-03-24T12:39:37.118880+0000","last_fullsized":"2024-03-24T12:39:37.118880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:37:10.768567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:10.768567+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968363+0000","scrub_duration":289,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":143,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119863+0000","last_change":"2024-03-24T12:37:01.978571+0000","last_active":"2024-03-24T12:39:37.119863+0000","last_peered":"2024-03-24T12:39:37.119863+0000","last_clean":"2024-03-24T12:39:37.119863+0000","last_became_active":"2024-03-24T12:36:52.065482+0000","last_became_peered":"2024-03-24T12:36:52.065482+0000","last_unstale":"2024-03-24T12:39:37.119863+0000","last_undegraded":"2024-03-24T12:39:37.119863+0000","last_fullsized":"2024-03-24T12:39:37.119863+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-24T12:37:01.978436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:01.978436+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":438,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119280+0000","last_change":"2024-03-24T12:37:31.133173+0000","last_active":"2024-03-24T12:39:37.119280+0000","last_peered":"2024-03-24T12:39:37.119280+0000","last_clean":"2024-03-24T12:39:37.119280+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:39:37.119280+0000","last_undegraded":"2024-03-24T12:39:37.119280+0000","last_fullsized":"2024-03-24T12:39:37.119280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:31.133103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:31.133103+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"207'20","reported_seq":464,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.124938+0000","last_change":"2024-03-24T12:37:27.793372+0000","last_active":"2024-03-24T12:39:37.124938+0000","last_peered":"2024-03-24T12:39:37.124938+0000","last_clean":"2024-03-24T12:39:37.124938+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:39:37.124938+0000","last_undegraded":"2024-03-24T12:39:37.124938+0000","last_fullsized":"2024-03-24T12:39:37.124938+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:37:27.793250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:27.793250+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":421,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.113738+0000","last_change":"2024-03-24T12:35:58.712757+0000","last_active":"2024-03-24T12:39:37.113738+0000","last_peered":"2024-03-24T12:39:37.113738+0000","last_clean":"2024-03-24T12:35:54.677408+0000","last_became_active":"2024-03-24T12:35:58.712757+0000","last_became_peered":"2024-03-24T12:35:58.712757+0000","last_unstale":"2024-03-24T12:39:37.113738+0000","last_undegraded":"2024-03-24T12:39:37.113738+0000","last_fullsized":"2024-03-24T12:35:58.700636+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:01.913287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:01.913287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"93'9","reported_seq":596,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:38.370436+0000","last_change":"2024-03-24T12:37:32.841507+0000","last_active":"2024-03-24T12:39:38.370436+0000","last_peered":"2024-03-24T12:39:38.370436+0000","last_clean":"2024-03-24T12:39:38.370436+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:39:38.370436+0000","last_undegraded":"2024-03-24T12:39:38.370436+0000","last_fullsized":"2024-03-24T12:39:38.370436+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'9","last_scrub_stamp":"2024-03-24T12:37:32.840824+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:37:32.840824+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":438,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.118171+0000","last_change":"2024-03-24T12:37:31.597078+0000","last_active":"2024-03-24T12:39:37.118171+0000","last_peered":"2024-03-24T12:39:37.118171+0000","last_clean":"2024-03-24T12:39:37.118171+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:39:37.118171+0000","last_undegraded":"2024-03-24T12:39:37.118171+0000","last_fullsized":"2024-03-24T12:39:37.118171+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:31.596956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:31.596956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968608+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"207'19","reported_seq":193,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:37.120969+0000","last_change":"2024-03-24T12:35:58.751467+0000","last_active":"2024-03-24T12:39:37.120969+0000","last_peered":"2024-03-24T12:39:37.120969+0000","last_clean":"2024-03-24T12:27:16.240812+0000","last_became_active":"2024-03-24T12:35:58.751467+0000","last_became_peered":"2024-03-24T12:35:58.751467+0000","last_unstale":"2024-03-24T12:39:37.120969+0000","last_undegraded":"2024-03-24T12:35:58.700376+0000","last_fullsized":"2024-03-24T12:35:58.700265+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:35:08.002032+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.002032+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":434,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.142947+0000","last_change":"2024-03-24T12:37:44.451721+0000","last_active":"2024-03-24T12:39:37.142947+0000","last_peered":"2024-03-24T12:39:37.142947+0000","last_clean":"2024-03-24T12:39:37.142947+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:39:37.142947+0000","last_undegraded":"2024-03-24T12:39:37.142947+0000","last_fullsized":"2024-03-24T12:39:37.142947+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:44.451553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:37:44.451553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":422,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.131003+0000","last_change":"2024-03-24T12:37:48.933827+0000","last_active":"2024-03-24T12:39:37.131003+0000","last_peered":"2024-03-24T12:39:37.131003+0000","last_clean":"2024-03-24T12:39:37.131003+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:39:37.131003+0000","last_undegraded":"2024-03-24T12:39:37.131003+0000","last_fullsized":"2024-03-24T12:39:37.131003+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:48.933558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:48.933558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":451,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.121574+0000","last_change":"2024-03-24T12:39:34.755937+0000","last_active":"2024-03-24T12:39:37.121574+0000","last_peered":"2024-03-24T12:39:37.121574+0000","last_clean":"2024-03-24T12:39:37.121574+0000","last_became_active":"2024-03-24T12:39:34.755495+0000","last_became_peered":"2024-03-24T12:39:34.755495+0000","last_unstale":"2024-03-24T12:39:37.121574+0000","last_undegraded":"2024-03-24T12:39:37.121574+0000","last_fullsized":"2024-03-24T12:39:37.121574+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:50.800852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:50.800852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:40.219239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"207'15","reported_seq":459,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119353+0000","last_change":"2024-03-24T12:37:14.550395+0000","last_active":"2024-03-24T12:39:37.119353+0000","last_peered":"2024-03-24T12:39:37.119353+0000","last_clean":"2024-03-24T12:39:37.119353+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:39:37.119353+0000","last_undegraded":"2024-03-24T12:39:37.119353+0000","last_fullsized":"2024-03-24T12:39:37.119353+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:37:14.550244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:14.550244+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968513+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":428,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.120722+0000","last_change":"2024-03-24T12:39:35.100588+0000","last_active":"2024-03-24T12:39:37.120722+0000","last_peered":"2024-03-24T12:39:37.120722+0000","last_clean":"2024-03-24T12:39:37.120722+0000","last_became_active":"2024-03-24T12:39:35.100242+0000","last_became_peered":"2024-03-24T12:39:35.100242+0000","last_unstale":"2024-03-24T12:39:37.120722+0000","last_undegraded":"2024-03-24T12:39:37.120722+0000","last_fullsized":"2024-03-24T12:39:37.120722+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:14.149913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:14.149913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:40.219254+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":117,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.126332+0000","last_change":"2024-03-24T12:35:58.714488+0000","last_active":"2024-03-24T12:39:37.126332+0000","last_peered":"2024-03-24T12:39:37.126332+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.714488+0000","last_became_peered":"2024-03-24T12:35:58.714488+0000","last_unstale":"2024-03-24T12:39:37.126332+0000","last_undegraded":"2024-03-24T12:39:37.126332+0000","last_fullsized":"2024-03-24T12:35:58.700893+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.995618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.995618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":393,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.120449+0000","last_change":"2024-03-24T12:39:35.121187+0000","last_active":"2024-03-24T12:39:37.120449+0000","last_peered":"2024-03-24T12:39:37.120449+0000","last_clean":"2024-03-24T12:39:37.120449+0000","last_became_active":"2024-03-24T12:39:35.120926+0000","last_became_peered":"2024-03-24T12:39:35.120926+0000","last_unstale":"2024-03-24T12:39:37.120449+0000","last_undegraded":"2024-03-24T12:39:37.120449+0000","last_fullsized":"2024-03-24T12:39:37.120449+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.188916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.188916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,3],"avail_no_missing":["6","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"207'12","reported_seq":444,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.131503+0000","last_change":"2024-03-24T12:37:25.779784+0000","last_active":"2024-03-24T12:39:37.131503+0000","last_peered":"2024-03-24T12:39:37.131503+0000","last_clean":"2024-03-24T12:39:37.131503+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:39:37.131503+0000","last_undegraded":"2024-03-24T12:39:37.131503+0000","last_fullsized":"2024-03-24T12:39:37.131503+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-24T12:37:25.779285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:25.779285+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.5","version":"258'11195","reported_seq":11293,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:41.250853+0000","last_change":"2024-03-24T12:36:53.934548+0000","last_active":"2024-03-24T12:39:41.250853+0000","last_peered":"2024-03-24T12:39:41.250853+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"2024-03-24T12:36:41.664044+0000","last_became_peered":"2024-03-24T12:36:41.664044+0000","last_unstale":"2024-03-24T12:39:41.250853+0000","last_undegraded":"2024-03-24T12:36:53.926339+0000","last_fullsized":"2024-03-24T12:36:41.661062+0000","mapping_epoch":212,"log_start":"228'5666","ondisk_log_start":"228'5666","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":5529,"log_dups_size":0,"ondisk_log_size":5529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184877056,"num_objects":2821,"num_object_clones":0,"num_object_copies":8463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2821,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11195,"num_write_kb":448512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":2821}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":76,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.121038+0000","last_change":"2024-03-24T12:39:35.104356+0000","last_active":"2024-03-24T12:39:37.121038+0000","last_peered":"2024-03-24T12:39:37.121038+0000","last_clean":"2024-03-24T12:39:37.121038+0000","last_became_active":"2024-03-24T12:39:35.103777+0000","last_became_peered":"2024-03-24T12:39:35.103777+0000","last_unstale":"2024-03-24T12:39:37.121038+0000","last_undegraded":"2024-03-24T12:39:37.121038+0000","last_fullsized":"2024-03-24T12:39:37.121038+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.170473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.170473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,2],"avail_no_missing":["6","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":451,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.120102+0000","last_change":"2024-03-24T12:37:57.274231+0000","last_active":"2024-03-24T12:39:37.120102+0000","last_peered":"2024-03-24T12:39:37.120102+0000","last_clean":"2024-03-24T12:39:37.120102+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:39:37.120102+0000","last_undegraded":"2024-03-24T12:39:37.120102+0000","last_fullsized":"2024-03-24T12:39:37.120102+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:37:57.274142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:57.274142+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"207'27","reported_seq":212,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:42.354863+0000","last_change":"2024-03-24T12:39:42.354983+0000","last_active":"2024-03-24T12:39:42.354863+0000","last_peered":"2024-03-24T12:39:42.354863+0000","last_clean":"2024-03-24T12:39:42.354863+0000","last_became_active":"2024-03-24T12:39:35.103338+0000","last_became_peered":"2024-03-24T12:39:35.103338+0000","last_unstale":"2024-03-24T12:39:42.354863+0000","last_undegraded":"2024-03-24T12:39:42.354863+0000","last_fullsized":"2024-03-24T12:39:42.354863+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-24T12:39:42.354720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:39:42.354720+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:08.243487+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,4],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"258'10","reported_seq":634,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.121718+0000","last_change":"2024-03-24T12:39:34.759270+0000","last_active":"2024-03-24T12:39:37.121718+0000","last_peered":"2024-03-24T12:39:37.121718+0000","last_clean":"2024-03-24T12:39:37.121718+0000","last_became_active":"2024-03-24T12:39:34.753277+0000","last_became_peered":"2024-03-24T12:39:34.753277+0000","last_unstale":"2024-03-24T12:39:37.121718+0000","last_undegraded":"2024-03-24T12:39:37.121718+0000","last_fullsized":"2024-03-24T12:39:37.121718+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:38:01.284422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:38:01.284422+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:40.219316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.4","version":"258'16304","reported_seq":16680,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:44.652679+0000","last_change":"2024-03-24T12:38:41.135820+0000","last_active":"2024-03-24T12:39:44.652679+0000","last_peered":"2024-03-24T12:39:44.652679+0000","last_clean":"2024-03-24T12:39:44.652679+0000","last_became_active":"2024-03-24T12:36:41.667094+0000","last_became_peered":"2024-03-24T12:36:41.667094+0000","last_unstale":"2024-03-24T12:39:44.652679+0000","last_undegraded":"2024-03-24T12:39:44.652679+0000","last_fullsized":"2024-03-24T12:39:44.652679+0000","mapping_epoch":212,"log_start":"238'11683","ondisk_log_start":"238'11683","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"240'12543","last_scrub_stamp":"2024-03-24T12:38:41.135548+0000","last_deep_scrub":"240'12543","last_deep_scrub_stamp":"2024-03-24T12:38:41.135548+0000","last_clean_scrub_stamp":"2024-03-24T12:38:41.135548+0000","objects_scrubbed":6528,"log_size":4621,"log_dups_size":0,"ondisk_log_size":4621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":40,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:45.668099+0000","scrub_duration":39042,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168361984,"num_objects":2569,"num_object_clones":0,"num_object_copies":7707,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2569,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11321,"num_write_kb":444480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":419,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.124354+0000","last_change":"2024-03-24T12:37:51.992994+0000","last_active":"2024-03-24T12:39:37.124354+0000","last_peered":"2024-03-24T12:39:37.124354+0000","last_clean":"2024-03-24T12:39:37.124354+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:39:37.124354+0000","last_undegraded":"2024-03-24T12:39:37.124354+0000","last_fullsized":"2024-03-24T12:39:37.124354+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:51.992860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:51.992860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":431,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119602+0000","last_change":"2024-03-24T12:36:57.986752+0000","last_active":"2024-03-24T12:39:37.119602+0000","last_peered":"2024-03-24T12:39:37.119602+0000","last_clean":"2024-03-24T12:39:37.119602+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:39:37.119602+0000","last_undegraded":"2024-03-24T12:39:37.119602+0000","last_fullsized":"2024-03-24T12:39:37.119602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:57.986675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:36:57.986675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"207'23","reported_seq":480,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.113538+0000","last_change":"2024-03-24T12:37:30.795479+0000","last_active":"2024-03-24T12:39:37.113538+0000","last_peered":"2024-03-24T12:39:37.113538+0000","last_clean":"2024-03-24T12:39:37.113538+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:39:37.113538+0000","last_undegraded":"2024-03-24T12:39:37.113538+0000","last_fullsized":"2024-03-24T12:39:37.113538+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-24T12:37:30.795334+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:37:30.795334+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":439,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.131220+0000","last_change":"2024-03-24T12:39:36.565212+0000","last_active":"2024-03-24T12:39:37.131220+0000","last_peered":"2024-03-24T12:39:37.131220+0000","last_clean":"2024-03-24T12:39:37.131220+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:39:37.131220+0000","last_undegraded":"2024-03-24T12:39:37.131220+0000","last_fullsized":"2024-03-24T12:39:37.131220+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:36.564861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:39:36.564861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:40.949969+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.7","version":"258'11337","reported_seq":11437,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:42.597624+0000","last_change":"2024-03-24T12:36:53.934613+0000","last_active":"2024-03-24T12:39:42.597624+0000","last_peered":"2024-03-24T12:39:42.597624+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"2024-03-24T12:36:41.663634+0000","last_became_peered":"2024-03-24T12:36:41.663634+0000","last_unstale":"2024-03-24T12:39:42.597624+0000","last_undegraded":"2024-03-24T12:36:53.926245+0000","last_fullsized":"2024-03-24T12:36:41.660768+0000","mapping_epoch":212,"log_start":"236'6644","ondisk_log_start":"236'6644","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":4693,"log_dups_size":0,"ondisk_log_size":4693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180289536,"num_objects":2751,"num_object_clones":0,"num_object_copies":8253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2751,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11337,"num_write_kb":450816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":2751}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":427,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.118947+0000","last_change":"2024-03-24T12:37:49.172015+0000","last_active":"2024-03-24T12:39:37.118947+0000","last_peered":"2024-03-24T12:39:37.118947+0000","last_clean":"2024-03-24T12:39:37.118947+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:39:37.118947+0000","last_undegraded":"2024-03-24T12:39:37.118947+0000","last_fullsized":"2024-03-24T12:39:37.118947+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:49.171892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:49.171892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":426,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.132098+0000","last_change":"2024-03-24T12:35:58.711865+0000","last_active":"2024-03-24T12:39:37.132098+0000","last_peered":"2024-03-24T12:39:37.132098+0000","last_clean":"2024-03-24T12:35:54.681353+0000","last_became_active":"2024-03-24T12:35:58.711865+0000","last_became_peered":"2024-03-24T12:35:58.711865+0000","last_unstale":"2024-03-24T12:39:37.132098+0000","last_undegraded":"2024-03-24T12:39:37.132098+0000","last_fullsized":"2024-03-24T12:35:58.701950+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:16.019255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.019255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:41.544554+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":436,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.120975+0000","last_change":"2024-03-24T12:39:35.119622+0000","last_active":"2024-03-24T12:39:37.120975+0000","last_peered":"2024-03-24T12:39:37.120975+0000","last_clean":"2024-03-24T12:39:37.120975+0000","last_became_active":"2024-03-24T12:39:35.119214+0000","last_became_peered":"2024-03-24T12:39:35.119214+0000","last_unstale":"2024-03-24T12:39:37.120975+0000","last_undegraded":"2024-03-24T12:39:37.120975+0000","last_fullsized":"2024-03-24T12:39:37.120975+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:33:34.204829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:40.219222+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,3],"avail_no_missing":["7","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"207'35","reported_seq":458,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.120616+0000","last_change":"2024-03-24T12:39:35.101923+0000","last_active":"2024-03-24T12:39:37.120616+0000","last_peered":"2024-03-24T12:39:37.120616+0000","last_clean":"2024-03-24T12:39:37.120616+0000","last_became_active":"2024-03-24T12:39:35.101391+0000","last_became_peered":"2024-03-24T12:39:35.101391+0000","last_unstale":"2024-03-24T12:39:37.120616+0000","last_undegraded":"2024-03-24T12:39:37.120616+0000","last_fullsized":"2024-03-24T12:39:37.120616+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-24T12:37:09.996370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:09.996370+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"258'11593","reported_seq":11693,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:44.076718+0000","last_change":"2024-03-24T12:36:53.930467+0000","last_active":"2024-03-24T12:39:44.076718+0000","last_peered":"2024-03-24T12:39:44.076718+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"2024-03-24T12:36:41.666978+0000","last_became_peered":"2024-03-24T12:36:41.666978+0000","last_unstale":"2024-03-24T12:39:44.076718+0000","last_undegraded":"2024-03-24T12:36:53.926995+0000","last_fullsized":"2024-03-24T12:36:41.658871+0000","mapping_epoch":212,"log_start":"225'4940","ondisk_log_start":"225'4940","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":6653,"log_dups_size":0,"ondisk_log_size":6653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:47.553965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178192384,"num_objects":2719,"num_object_clones":0,"num_object_copies":8157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11593,"num_write_kb":457984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":2719}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":419,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.133439+0000","last_change":"2024-03-24T12:37:02.653185+0000","last_active":"2024-03-24T12:39:37.133439+0000","last_peered":"2024-03-24T12:39:37.133439+0000","last_clean":"2024-03-24T12:39:37.133439+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:39:37.133439+0000","last_undegraded":"2024-03-24T12:39:37.133439+0000","last_fullsized":"2024-03-24T12:39:37.133439+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:02.652980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:02.652980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968298+0000","scrub_duration":88,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":431,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.145802+0000","last_change":"2024-03-24T12:37:08.191342+0000","last_active":"2024-03-24T12:39:37.145802+0000","last_peered":"2024-03-24T12:39:37.145802+0000","last_clean":"2024-03-24T12:39:37.145802+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:39:37.145802+0000","last_undegraded":"2024-03-24T12:39:37.145802+0000","last_fullsized":"2024-03-24T12:39:37.145802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:08.191222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:37:08.191222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":188,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.125040+0000","last_change":"2024-03-24T12:39:35.119812+0000","last_active":"2024-03-24T12:39:37.125040+0000","last_peered":"2024-03-24T12:39:37.125040+0000","last_clean":"2024-03-24T12:39:37.125040+0000","last_became_active":"2024-03-24T12:39:35.119146+0000","last_became_peered":"2024-03-24T12:39:35.119146+0000","last_unstale":"2024-03-24T12:39:37.125040+0000","last_undegraded":"2024-03-24T12:39:37.125040+0000","last_fullsized":"2024-03-24T12:39:37.125040+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:35:15.994784+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.994784+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"207'15","reported_seq":468,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119628+0000","last_change":"2024-03-24T12:37:23.569527+0000","last_active":"2024-03-24T12:39:37.119628+0000","last_peered":"2024-03-24T12:39:37.119628+0000","last_clean":"2024-03-24T12:39:37.119628+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:39:37.119628+0000","last_undegraded":"2024-03-24T12:39:37.119628+0000","last_fullsized":"2024-03-24T12:39:37.119628+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:37:23.569282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:23.569282+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968480+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"258'11040","reported_seq":11140,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:41.247864+0000","last_change":"2024-03-24T12:36:53.929699+0000","last_active":"2024-03-24T12:39:41.247864+0000","last_peered":"2024-03-24T12:39:41.247864+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"2024-03-24T12:36:41.663349+0000","last_became_peered":"2024-03-24T12:36:41.663349+0000","last_unstale":"2024-03-24T12:39:41.247864+0000","last_undegraded":"2024-03-24T12:36:53.926702+0000","last_fullsized":"2024-03-24T12:36:41.660614+0000","mapping_epoch":212,"log_start":"228'5541","ondisk_log_start":"228'5541","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":5499,"log_dups_size":0,"ondisk_log_size":5499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180879360,"num_objects":2760,"num_object_clones":0,"num_object_copies":8280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2760,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11040,"num_write_kb":441600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":2760}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":419,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.126519+0000","last_change":"2024-03-24T12:37:23.691297+0000","last_active":"2024-03-24T12:39:37.126519+0000","last_peered":"2024-03-24T12:39:37.126519+0000","last_clean":"2024-03-24T12:39:37.126519+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:39:37.126519+0000","last_undegraded":"2024-03-24T12:39:37.126519+0000","last_fullsized":"2024-03-24T12:39:37.126519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:23.691167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:37:23.691167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":156,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.138119+0000","last_change":"2024-03-24T12:39:35.104530+0000","last_active":"2024-03-24T12:39:37.138119+0000","last_peered":"2024-03-24T12:39:37.138119+0000","last_clean":"2024-03-24T12:39:37.138119+0000","last_became_active":"2024-03-24T12:39:35.101653+0000","last_became_peered":"2024-03-24T12:39:35.101653+0000","last_unstale":"2024-03-24T12:39:37.138119+0000","last_undegraded":"2024-03-24T12:39:37.138119+0000","last_fullsized":"2024-03-24T12:39:37.138119+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:35:56.337636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:56.337636+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":451,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.113003+0000","last_change":"2024-03-24T12:39:34.751891+0000","last_active":"2024-03-24T12:39:37.113003+0000","last_peered":"2024-03-24T12:39:37.113003+0000","last_clean":"2024-03-24T12:39:37.113003+0000","last_became_active":"2024-03-24T12:39:34.751547+0000","last_became_peered":"2024-03-24T12:39:34.751547+0000","last_unstale":"2024-03-24T12:39:37.113003+0000","last_undegraded":"2024-03-24T12:39:37.113003+0000","last_fullsized":"2024-03-24T12:39:37.113003+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:17.809955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:17.809955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.857677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"248'51","reported_seq":559,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:38.370196+0000","last_change":"2024-03-24T12:39:07.750748+0000","last_active":"2024-03-24T12:39:38.370196+0000","last_peered":"2024-03-24T12:39:38.370196+0000","last_clean":"2024-03-24T12:39:38.370196+0000","last_became_active":"2024-03-24T12:39:07.750351+0000","last_became_peered":"2024-03-24T12:39:07.750351+0000","last_unstale":"2024-03-24T12:39:38.370196+0000","last_undegraded":"2024-03-24T12:39:38.370196+0000","last_fullsized":"2024-03-24T12:39:38.370196+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'50","last_scrub_stamp":"2024-03-24T12:37:23.072184+0000","last_deep_scrub":"90'38","last_deep_scrub_stamp":"2024-03-24T12:27:31.375629+0000","last_clean_scrub_stamp":"2024-03-24T12:37:23.072184+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":423,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119975+0000","last_change":"2024-03-24T12:36:56.595107+0000","last_active":"2024-03-24T12:39:37.119975+0000","last_peered":"2024-03-24T12:39:37.119975+0000","last_clean":"2024-03-24T12:39:37.119975+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:39:37.119975+0000","last_undegraded":"2024-03-24T12:39:37.119975+0000","last_fullsized":"2024-03-24T12:39:37.119975+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:56.594983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:36:56.594983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968239+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"258'11207","reported_seq":11309,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:42.608206+0000","last_change":"2024-03-24T12:39:07.750159+0000","last_active":"2024-03-24T12:39:42.608206+0000","last_peered":"2024-03-24T12:39:42.608206+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"2024-03-24T12:39:07.750159+0000","last_became_peered":"2024-03-24T12:39:07.750159+0000","last_unstale":"2024-03-24T12:39:42.608206+0000","last_undegraded":"2024-03-24T12:39:07.748227+0000","last_fullsized":"2024-03-24T12:39:07.745545+0000","mapping_epoch":247,"log_start":"235'6483","ondisk_log_start":"235'6483","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":4724,"log_dups_size":0,"ondisk_log_size":4724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":174391296,"num_objects":2661,"num_object_clones":0,"num_object_copies":7983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2661,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11207,"num_write_kb":443776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":2661}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":447,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.121348+0000","last_change":"2024-03-24T12:39:34.756774+0000","last_active":"2024-03-24T12:39:37.121348+0000","last_peered":"2024-03-24T12:39:37.121348+0000","last_clean":"2024-03-24T12:39:37.121348+0000","last_became_active":"2024-03-24T12:39:34.755964+0000","last_became_peered":"2024-03-24T12:39:34.755964+0000","last_unstale":"2024-03-24T12:39:37.121348+0000","last_undegraded":"2024-03-24T12:39:37.121348+0000","last_fullsized":"2024-03-24T12:39:37.121348+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:44.251949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:44.251949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:40.219284+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":432,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.131362+0000","last_change":"2024-03-24T12:39:34.744570+0000","last_active":"2024-03-24T12:39:37.131362+0000","last_peered":"2024-03-24T12:39:37.131362+0000","last_clean":"2024-03-24T12:39:37.131362+0000","last_became_active":"2024-03-24T12:39:34.744167+0000","last_became_peered":"2024-03-24T12:39:34.744167+0000","last_unstale":"2024-03-24T12:39:37.131362+0000","last_undegraded":"2024-03-24T12:39:37.131362+0000","last_fullsized":"2024-03-24T12:39:37.131362+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:25.075119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:25.075119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":462,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:40.561242+0000","last_change":"2024-03-24T12:39:40.561281+0000","last_active":"2024-03-24T12:39:40.561242+0000","last_peered":"2024-03-24T12:39:40.561242+0000","last_clean":"2024-03-24T12:39:40.561242+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:39:40.561242+0000","last_undegraded":"2024-03-24T12:39:40.561242+0000","last_fullsized":"2024-03-24T12:39:40.561242+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:40.561152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:39:40.561152+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:54.061511+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"207'22","reported_seq":468,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:37.128584+0000","last_change":"2024-03-24T12:35:58.709644+0000","last_active":"2024-03-24T12:39:37.128584+0000","last_peered":"2024-03-24T12:39:37.128584+0000","last_clean":"2024-03-24T12:35:54.687814+0000","last_became_active":"2024-03-24T12:35:58.709644+0000","last_became_peered":"2024-03-24T12:35:58.709644+0000","last_unstale":"2024-03-24T12:39:37.128584+0000","last_undegraded":"2024-03-24T12:35:58.700232+0000","last_fullsized":"2024-03-24T12:35:58.699170+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:33:02.509218+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:31:08.237387+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.509218+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.183762+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":426,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.134743+0000","last_change":"2024-03-24T12:35:58.703902+0000","last_active":"2024-03-24T12:39:37.134743+0000","last_peered":"2024-03-24T12:39:37.134743+0000","last_clean":"2024-03-24T12:35:54.680479+0000","last_became_active":"2024-03-24T12:35:58.703902+0000","last_became_peered":"2024-03-24T12:35:58.703902+0000","last_unstale":"2024-03-24T12:39:37.134743+0000","last_undegraded":"2024-03-24T12:39:37.134743+0000","last_fullsized":"2024-03-24T12:35:58.701457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:13.031712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:13.031712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:41.544508+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"258'11542","reported_seq":11642,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:45.012163+0000","last_change":"2024-03-24T12:36:53.924567+0000","last_active":"2024-03-24T12:39:45.012163+0000","last_peered":"2024-03-24T12:39:45.012163+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"2024-03-24T12:36:41.662795+0000","last_became_peered":"2024-03-24T12:36:41.662795+0000","last_unstale":"2024-03-24T12:39:45.012163+0000","last_undegraded":"2024-03-24T12:36:53.923700+0000","last_fullsized":"2024-03-24T12:36:41.656436+0000","mapping_epoch":212,"log_start":"236'7041","ondisk_log_start":"236'7041","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":4501,"log_dups_size":0,"ondisk_log_size":4501,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166199296,"num_objects":2536,"num_object_clones":0,"num_object_copies":7608,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2536,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2536,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11542,"num_write_kb":450496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[{"shards":"0,3","objects":2536}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":116,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.130823+0000","last_change":"2024-03-24T12:35:58.713785+0000","last_active":"2024-03-24T12:39:37.130823+0000","last_peered":"2024-03-24T12:39:37.130823+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.713785+0000","last_became_peered":"2024-03-24T12:35:58.713785+0000","last_unstale":"2024-03-24T12:39:37.130823+0000","last_undegraded":"2024-03-24T12:39:37.130823+0000","last_fullsized":"2024-03-24T12:35:58.700902+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:18.057286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:18.057286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":722,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.136926+0000","last_change":"2024-03-24T12:39:37.136926+0000","last_active":"2024-03-24T12:39:37.136926+0000","last_peered":"2024-03-24T12:39:37.136926+0000","last_clean":"2024-03-24T12:39:37.136926+0000","last_became_active":"2024-03-24T12:39:37.136501+0000","last_became_peered":"2024-03-24T12:39:37.136501+0000","last_unstale":"2024-03-24T12:39:37.136926+0000","last_undegraded":"2024-03-24T12:39:37.136926+0000","last_fullsized":"2024-03-24T12:39:37.136926+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:37:21.191312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:37:21.191312+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":436,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.123187+0000","last_change":"2024-03-24T12:39:35.128714+0000","last_active":"2024-03-24T12:39:37.123187+0000","last_peered":"2024-03-24T12:39:37.123187+0000","last_clean":"2024-03-24T12:39:37.123187+0000","last_became_active":"2024-03-24T12:39:35.128182+0000","last_became_peered":"2024-03-24T12:39:35.128182+0000","last_unstale":"2024-03-24T12:39:37.123187+0000","last_undegraded":"2024-03-24T12:39:37.123187+0000","last_fullsized":"2024-03-24T12:39:37.123187+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:24.345279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:24.345279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"258'30","reported_seq":36,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.595157+0000","last_change":"2024-03-24T12:39:34.768678+0000","last_active":"2024-03-24T12:39:37.595157+0000","last_peered":"2024-03-24T12:39:37.595157+0000","last_clean":"2024-03-24T12:39:37.595157+0000","last_became_active":"2024-03-24T12:39:34.753078+0000","last_became_peered":"2024-03-24T12:39:34.753078+0000","last_unstale":"2024-03-24T12:39:37.595157+0000","last_undegraded":"2024-03-24T12:39:37.595157+0000","last_fullsized":"2024-03-24T12:39:37.595157+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'28","last_scrub_stamp":"2024-03-24T12:37:25.085444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:25.085444+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":418,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.119526+0000","last_change":"2024-03-24T12:35:58.711216+0000","last_active":"2024-03-24T12:39:37.119526+0000","last_peered":"2024-03-24T12:39:37.119526+0000","last_clean":"2024-03-24T12:35:54.688099+0000","last_became_active":"2024-03-24T12:35:58.711216+0000","last_became_peered":"2024-03-24T12:35:58.711216+0000","last_unstale":"2024-03-24T12:39:37.119526+0000","last_undegraded":"2024-03-24T12:39:37.119526+0000","last_fullsized":"2024-03-24T12:35:58.699124+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.802609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.802609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"258'11421","reported_seq":24263,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:41.259027+0000","last_change":"2024-03-24T12:39:35.135407+0000","last_active":"2024-03-24T12:39:41.259027+0000","last_peered":"2024-03-24T12:39:41.259027+0000","last_clean":"2024-03-24T12:39:41.259027+0000","last_became_active":"2024-03-24T12:39:35.104380+0000","last_became_peered":"2024-03-24T12:39:35.104380+0000","last_unstale":"2024-03-24T12:39:41.259027+0000","last_undegraded":"2024-03-24T12:39:41.259027+0000","last_fullsized":"2024-03-24T12:39:41.259027+0000","mapping_epoch":255,"log_start":"228'5966","ondisk_log_start":"228'5966","created":212,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":5455,"log_dups_size":0,"ondisk_log_size":5455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190251008,"num_objects":2903,"num_object_clones":0,"num_object_copies":8709,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2903,"num_objects_unfound":0,"num_objects_dirty":2903,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11421,"num_write_kb":458368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6365,"num_bytes_recovered":417136640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,2],"avail_no_missing":["0","2","6"],"object_location_counts":[{"shards":"0,2,6","objects":2903}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":430,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.133612+0000","last_change":"2024-03-24T12:35:58.717519+0000","last_active":"2024-03-24T12:39:37.133612+0000","last_peered":"2024-03-24T12:39:37.133612+0000","last_clean":"2024-03-24T12:35:54.687071+0000","last_became_active":"2024-03-24T12:35:58.717519+0000","last_became_peered":"2024-03-24T12:35:58.717519+0000","last_unstale":"2024-03-24T12:39:37.133612+0000","last_undegraded":"2024-03-24T12:39:37.133612+0000","last_fullsized":"2024-03-24T12:35:58.696848+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:59.974279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:59.974279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.183796+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":439,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.133203+0000","last_change":"2024-03-24T12:37:25.546135+0000","last_active":"2024-03-24T12:39:37.133203+0000","last_peered":"2024-03-24T12:39:37.133203+0000","last_clean":"2024-03-24T12:39:37.133203+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:39:37.133203+0000","last_undegraded":"2024-03-24T12:39:37.133203+0000","last_fullsized":"2024-03-24T12:39:37.133203+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:25.546026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:13.629968+0000","last_clean_scrub_stamp":"2024-03-24T12:37:25.546026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1269,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:41.423175+0000","last_change":"2024-03-24T12:37:35.137487+0000","last_active":"2024-03-24T12:39:41.423175+0000","last_peered":"2024-03-24T12:39:41.423175+0000","last_clean":"2024-03-24T12:39:41.423175+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:39:41.423175+0000","last_undegraded":"2024-03-24T12:39:41.423175+0000","last_fullsized":"2024-03-24T12:39:41.423175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:37:35.137356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:35.137356+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":812,"num_read_kb":807,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"207'24","reported_seq":463,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119000+0000","last_change":"2024-03-24T12:36:55.592574+0000","last_active":"2024-03-24T12:39:37.119000+0000","last_peered":"2024-03-24T12:39:37.119000+0000","last_clean":"2024-03-24T12:39:37.119000+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:39:37.119000+0000","last_undegraded":"2024-03-24T12:39:37.119000+0000","last_fullsized":"2024-03-24T12:39:37.119000+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-24T12:36:55.592359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:36:55.592359+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968331+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":117,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.128884+0000","last_change":"2024-03-24T12:35:58.708969+0000","last_active":"2024-03-24T12:39:37.128884+0000","last_peered":"2024-03-24T12:39:37.128884+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.708969+0000","last_became_peered":"2024-03-24T12:35:58.708969+0000","last_unstale":"2024-03-24T12:39:37.128884+0000","last_undegraded":"2024-03-24T12:39:37.128884+0000","last_fullsized":"2024-03-24T12:35:58.699657+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:52.136918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:52.136918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.183804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.e","version":"258'11353","reported_seq":11526,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:42.603689+0000","last_change":"2024-03-24T12:38:04.875897+0000","last_active":"2024-03-24T12:39:42.603689+0000","last_peered":"2024-03-24T12:39:42.603689+0000","last_clean":"2024-03-24T12:39:42.603689+0000","last_became_active":"2024-03-24T12:36:41.668641+0000","last_became_peered":"2024-03-24T12:36:41.668641+0000","last_unstale":"2024-03-24T12:39:42.603689+0000","last_undegraded":"2024-03-24T12:39:42.603689+0000","last_fullsized":"2024-03-24T12:39:42.603689+0000","mapping_epoch":212,"log_start":"236'6664","ondisk_log_start":"236'6664","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'6301","last_scrub_stamp":"2024-03-24T12:38:04.875538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:38:04.875538+0000","objects_scrubbed":6241,"log_size":4689,"log_dups_size":0,"ondisk_log_size":4689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179240960,"num_objects":2735,"num_object_clones":0,"num_object_copies":8205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2735,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11353,"num_write_kb":450816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":447,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.130204+0000","last_change":"2024-03-24T12:39:34.753121+0000","last_active":"2024-03-24T12:39:37.130204+0000","last_peered":"2024-03-24T12:39:37.130204+0000","last_clean":"2024-03-24T12:39:37.130204+0000","last_became_active":"2024-03-24T12:39:34.751275+0000","last_became_peered":"2024-03-24T12:39:34.751275+0000","last_unstale":"2024-03-24T12:39:37.130204+0000","last_undegraded":"2024-03-24T12:39:37.130204+0000","last_fullsized":"2024-03-24T12:39:37.130204+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:49.010733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:49.010733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":349,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.142413+0000","last_change":"2024-03-24T12:39:07.747764+0000","last_active":"2024-03-24T12:39:37.142413+0000","last_peered":"2024-03-24T12:39:37.142413+0000","last_clean":"2024-03-24T12:35:54.687290+0000","last_became_active":"2024-03-24T12:39:07.747764+0000","last_became_peered":"2024-03-24T12:39:07.747764+0000","last_unstale":"2024-03-24T12:39:37.142413+0000","last_undegraded":"2024-03-24T12:39:37.142413+0000","last_fullsized":"2024-03-24T12:39:07.743810+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:54.953570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.953570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:41.544540+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"207'37","reported_seq":481,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:37.119729+0000","last_change":"2024-03-24T12:35:58.711927+0000","last_active":"2024-03-24T12:39:37.119729+0000","last_peered":"2024-03-24T12:39:37.119729+0000","last_clean":"2024-03-24T12:35:54.682985+0000","last_became_active":"2024-03-24T12:35:58.711927+0000","last_became_peered":"2024-03-24T12:35:58.711927+0000","last_unstale":"2024-03-24T12:39:37.119729+0000","last_undegraded":"2024-03-24T12:35:58.703907+0000","last_fullsized":"2024-03-24T12:35:58.703783+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'27","last_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_deep_scrub":"195'27","last_deep_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.981148+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.960965+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":386,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.120755+0000","last_change":"2024-03-24T12:39:35.100139+0000","last_active":"2024-03-24T12:39:37.120755+0000","last_peered":"2024-03-24T12:39:37.120755+0000","last_clean":"2024-03-24T12:39:37.120755+0000","last_became_active":"2024-03-24T12:39:35.099711+0000","last_became_peered":"2024-03-24T12:39:35.099711+0000","last_unstale":"2024-03-24T12:39:37.120755+0000","last_undegraded":"2024-03-24T12:39:37.120755+0000","last_fullsized":"2024-03-24T12:39:37.120755+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:44.164114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.164114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,2],"avail_no_missing":["6","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"258'11367","reported_seq":11467,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:44.673080+0000","last_change":"2024-03-24T12:36:53.926582+0000","last_active":"2024-03-24T12:39:44.673080+0000","last_peered":"2024-03-24T12:39:44.673080+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"2024-03-24T12:36:41.668418+0000","last_became_peered":"2024-03-24T12:36:41.668418+0000","last_unstale":"2024-03-24T12:39:44.673080+0000","last_undegraded":"2024-03-24T12:36:53.925951+0000","last_fullsized":"2024-03-24T12:36:41.665242+0000","mapping_epoch":212,"log_start":"236'6778","ondisk_log_start":"236'6778","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":4589,"log_dups_size":0,"ondisk_log_size":4589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175702016,"num_objects":2681,"num_object_clones":0,"num_object_copies":8043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2681,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2681,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11367,"num_write_kb":449536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":2681}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":419,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.118102+0000","last_change":"2024-03-24T12:37:12.563433+0000","last_active":"2024-03-24T12:39:37.118102+0000","last_peered":"2024-03-24T12:39:37.118102+0000","last_clean":"2024-03-24T12:39:37.118102+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:39:37.118102+0000","last_undegraded":"2024-03-24T12:39:37.118102+0000","last_fullsized":"2024-03-24T12:39:37.118102+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:12.563308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:12.563308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968392+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":116,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.125257+0000","last_change":"2024-03-24T12:35:58.711061+0000","last_active":"2024-03-24T12:39:37.125257+0000","last_peered":"2024-03-24T12:39:37.125257+0000","last_clean":"2024-03-24T12:26:02.143860+0000","last_became_active":"2024-03-24T12:35:58.711061+0000","last_became_peered":"2024-03-24T12:35:58.711061+0000","last_unstale":"2024-03-24T12:39:37.125257+0000","last_undegraded":"2024-03-24T12:39:37.125257+0000","last_fullsized":"2024-03-24T12:35:58.704709+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:00.924106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.924106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"207'25","reported_seq":465,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:37.138640+0000","last_change":"2024-03-24T12:35:58.704200+0000","last_active":"2024-03-24T12:39:37.138640+0000","last_peered":"2024-03-24T12:39:37.138640+0000","last_clean":"2024-03-24T12:35:54.687582+0000","last_became_active":"2024-03-24T12:35:58.704200+0000","last_became_peered":"2024-03-24T12:35:58.704200+0000","last_unstale":"2024-03-24T12:39:37.138640+0000","last_undegraded":"2024-03-24T12:35:58.697842+0000","last_fullsized":"2024-03-24T12:35:58.697725+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:33:07.518368+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.518368+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.183751+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"258'13","reported_seq":615,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:38.373507+0000","last_change":"2024-03-24T12:39:36.908835+0000","last_active":"2024-03-24T12:39:38.373507+0000","last_peered":"2024-03-24T12:39:38.373507+0000","last_clean":"2024-03-24T12:39:38.373507+0000","last_became_active":"2024-03-24T12:39:35.135410+0000","last_became_peered":"2024-03-24T12:39:35.135410+0000","last_unstale":"2024-03-24T12:39:38.373507+0000","last_undegraded":"2024-03-24T12:39:38.373507+0000","last_fullsized":"2024-03-24T12:39:38.373507+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"235'11","last_scrub_stamp":"2024-03-24T12:39:36.908601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:39:36.908601+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:40:39.596979+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,3],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"258'11270","reported_seq":11434,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:44.980612+0000","last_change":"2024-03-24T12:37:52.438386+0000","last_active":"2024-03-24T12:39:44.980612+0000","last_peered":"2024-03-24T12:39:44.980612+0000","last_clean":"2024-03-24T12:39:44.980612+0000","last_became_active":"2024-03-24T12:36:41.662343+0000","last_became_peered":"2024-03-24T12:36:41.662343+0000","last_unstale":"2024-03-24T12:39:44.980612+0000","last_undegraded":"2024-03-24T12:39:44.980612+0000","last_fullsized":"2024-03-24T12:39:44.980612+0000","mapping_epoch":212,"log_start":"236'6804","ondisk_log_start":"236'6804","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"227'5341","last_scrub_stamp":"2024-03-24T12:37:52.438085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:37:52.438085+0000","objects_scrubbed":5265,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168165376,"num_objects":2566,"num_object_clones":0,"num_object_copies":7698,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2566,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11270,"num_write_kb":442752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":439,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119979+0000","last_change":"2024-03-24T12:39:34.756815+0000","last_active":"2024-03-24T12:39:37.119979+0000","last_peered":"2024-03-24T12:39:37.119979+0000","last_clean":"2024-03-24T12:39:37.119979+0000","last_became_active":"2024-03-24T12:39:34.756439+0000","last_became_peered":"2024-03-24T12:39:34.756439+0000","last_unstale":"2024-03-24T12:39:37.119979+0000","last_undegraded":"2024-03-24T12:39:37.119979+0000","last_fullsized":"2024-03-24T12:39:37.119979+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:34.173056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:34.173056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:40.219299+0000","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":440,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.119144+0000","last_change":"2024-03-24T12:37:49.519836+0000","last_active":"2024-03-24T12:39:37.119144+0000","last_peered":"2024-03-24T12:39:37.119144+0000","last_clean":"2024-03-24T12:39:37.119144+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:39:37.119144+0000","last_undegraded":"2024-03-24T12:39:37.119144+0000","last_fullsized":"2024-03-24T12:39:37.119144+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:37:49.519707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:49.519707+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:37.968577+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"207'35","reported_seq":505,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.112504+0000","last_change":"2024-03-24T12:39:35.103764+0000","last_active":"2024-03-24T12:39:37.112504+0000","last_peered":"2024-03-24T12:39:37.112504+0000","last_clean":"2024-03-24T12:39:37.112504+0000","last_became_active":"2024-03-24T12:39:35.103040+0000","last_became_peered":"2024-03-24T12:39:35.103040+0000","last_unstale":"2024-03-24T12:39:37.112504+0000","last_undegraded":"2024-03-24T12:39:37.112504+0000","last_fullsized":"2024-03-24T12:39:37.112504+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'21","last_scrub_stamp":"2024-03-24T12:35:14.829466+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:35:14.829466+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":596,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:38.370944+0000","last_change":"2024-03-24T12:37:10.779677+0000","last_active":"2024-03-24T12:39:38.370944+0000","last_peered":"2024-03-24T12:39:38.370944+0000","last_clean":"2024-03-24T12:39:38.370944+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:39:38.370944+0000","last_undegraded":"2024-03-24T12:39:38.370944+0000","last_fullsized":"2024-03-24T12:39:38.370944+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:37:10.779558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:10.779558+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.d","version":"258'11198","reported_seq":11299,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:42.603849+0000","last_change":"2024-03-24T12:36:53.927149+0000","last_active":"2024-03-24T12:39:42.603849+0000","last_peered":"2024-03-24T12:39:42.603849+0000","last_clean":"2024-03-24T12:36:40.645298+0000","last_became_active":"2024-03-24T12:36:41.668693+0000","last_became_peered":"2024-03-24T12:36:41.668693+0000","last_unstale":"2024-03-24T12:39:42.603849+0000","last_undegraded":"2024-03-24T12:36:53.926928+0000","last_fullsized":"2024-03-24T12:36:41.660349+0000","mapping_epoch":212,"log_start":"236'6510","ondisk_log_start":"236'6510","created":212,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":173735960,"num_objects":2652,"num_object_clones":0,"num_object_copies":7956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2652,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2652,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11198,"num_write_kb":443137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":2652}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":427,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.112563+0000","last_change":"2024-03-24T12:37:29.822038+0000","last_active":"2024-03-24T12:39:37.112563+0000","last_peered":"2024-03-24T12:39:37.112563+0000","last_clean":"2024-03-24T12:39:37.112563+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:39:37.112563+0000","last_undegraded":"2024-03-24T12:39:37.112563+0000","last_fullsized":"2024-03-24T12:39:37.112563+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:29.821915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:29.821915+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":431,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.130675+0000","last_change":"2024-03-24T12:37:30.870003+0000","last_active":"2024-03-24T12:39:37.130675+0000","last_peered":"2024-03-24T12:39:37.130675+0000","last_clean":"2024-03-24T12:39:37.130675+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:39:37.130675+0000","last_undegraded":"2024-03-24T12:39:37.130675+0000","last_fullsized":"2024-03-24T12:39:37.130675+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:30.869794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:30.869794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"207'30","reported_seq":485,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.124045+0000","last_change":"2024-03-24T12:37:54.840702+0000","last_active":"2024-03-24T12:39:37.124045+0000","last_peered":"2024-03-24T12:39:37.124045+0000","last_clean":"2024-03-24T12:39:37.124045+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:39:37.124045+0000","last_undegraded":"2024-03-24T12:39:37.124045+0000","last_fullsized":"2024-03-24T12:39:37.124045+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'30","last_scrub_stamp":"2024-03-24T12:37:54.840559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:54.840559+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":439,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.112375+0000","last_change":"2024-03-24T12:37:48.720907+0000","last_active":"2024-03-24T12:39:37.112375+0000","last_peered":"2024-03-24T12:39:37.112375+0000","last_clean":"2024-03-24T12:39:37.112375+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:39:37.112375+0000","last_undegraded":"2024-03-24T12:39:37.112375+0000","last_fullsized":"2024-03-24T12:39:37.112375+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:48.720761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:48.720761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"258'11482","reported_seq":11585,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:45.012891+0000","last_change":"2024-03-24T12:36:41.664129+0000","last_active":"2024-03-24T12:39:45.012891+0000","last_peered":"2024-03-24T12:39:45.012891+0000","last_clean":"2024-03-24T12:39:45.012891+0000","last_became_active":"2024-03-24T12:36:41.663814+0000","last_became_peered":"2024-03-24T12:36:41.663814+0000","last_unstale":"2024-03-24T12:39:45.012891+0000","last_undegraded":"2024-03-24T12:39:45.012891+0000","last_fullsized":"2024-03-24T12:39:45.012891+0000","mapping_epoch":212,"log_start":"236'7017","ondisk_log_start":"236'7017","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:36:40.645298+0000","objects_scrubbed":0,"log_size":4465,"log_dups_size":0,"ondisk_log_size":4465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163184640,"num_objects":2490,"num_object_clones":0,"num_object_copies":7470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11482,"num_write_kb":447104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":116,"reported_epoch":258,"state":"active+undersized","last_fresh":"2024-03-24T12:39:37.131766+0000","last_change":"2024-03-24T12:35:58.711067+0000","last_active":"2024-03-24T12:39:37.131766+0000","last_peered":"2024-03-24T12:39:37.131766+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.711067+0000","last_became_peered":"2024-03-24T12:35:58.711067+0000","last_unstale":"2024-03-24T12:39:37.131766+0000","last_undegraded":"2024-03-24T12:39:37.131766+0000","last_fullsized":"2024-03-24T12:35:58.696416+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:36.210440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.210440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.183787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"205'10","reported_seq":600,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:37.543685+0000","last_change":"2024-03-24T12:35:58.712862+0000","last_active":"2024-03-24T12:39:37.543685+0000","last_peered":"2024-03-24T12:39:37.543685+0000","last_clean":"2024-03-24T12:35:54.678342+0000","last_became_active":"2024-03-24T12:35:58.712862+0000","last_became_peered":"2024-03-24T12:35:58.712862+0000","last_unstale":"2024-03-24T12:39:37.543685+0000","last_undegraded":"2024-03-24T12:35:58.700712+0000","last_fullsized":"2024-03-24T12:35:58.700587+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'9","last_scrub_stamp":"2024-03-24T12:34:44.703379+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.703379+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":431,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.123960+0000","last_change":"2024-03-24T12:37:37.893086+0000","last_active":"2024-03-24T12:39:37.123960+0000","last_peered":"2024-03-24T12:39:37.123960+0000","last_clean":"2024-03-24T12:39:37.123960+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:39:37.123960+0000","last_undegraded":"2024-03-24T12:39:37.123960+0000","last_fullsized":"2024-03-24T12:39:37.123960+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:37.892950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:37.892950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"207'34","reported_seq":467,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.130745+0000","last_change":"2024-03-24T12:37:31.267002+0000","last_active":"2024-03-24T12:39:37.130745+0000","last_peered":"2024-03-24T12:39:37.130745+0000","last_clean":"2024-03-24T12:39:37.130745+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:39:37.130745+0000","last_undegraded":"2024-03-24T12:39:37.130745+0000","last_fullsized":"2024-03-24T12:39:37.130745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-24T12:37:31.266835+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:37:31.266835+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3","version":"258'11189","reported_seq":11362,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:42.512261+0000","last_change":"2024-03-24T12:38:07.067256+0000","last_active":"2024-03-24T12:39:42.512261+0000","last_peered":"2024-03-24T12:39:42.512261+0000","last_clean":"2024-03-24T12:39:42.512261+0000","last_became_active":"2024-03-24T12:36:41.667403+0000","last_became_peered":"2024-03-24T12:36:41.667403+0000","last_unstale":"2024-03-24T12:39:42.512261+0000","last_undegraded":"2024-03-24T12:39:42.512261+0000","last_fullsized":"2024-03-24T12:39:42.512261+0000","mapping_epoch":212,"log_start":"231'6475","ondisk_log_start":"231'6475","created":212,"last_epoch_clean":213,"parent":"0.0","parent_split_bits":0,"last_scrub":"230'6286","last_scrub_stamp":"2024-03-24T12:38:07.067066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:38:07.067066+0000","objects_scrubbed":6248,"log_size":4714,"log_dups_size":0,"ondisk_log_size":4714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1632,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180944896,"num_objects":2761,"num_object_clones":0,"num_object_copies":8283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11189,"num_write_kb":446400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":416,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.124840+0000","last_change":"2024-03-24T12:39:35.130227+0000","last_active":"2024-03-24T12:39:37.124840+0000","last_peered":"2024-03-24T12:39:37.124840+0000","last_clean":"2024-03-24T12:39:37.124840+0000","last_became_active":"2024-03-24T12:39:35.129779+0000","last_became_peered":"2024-03-24T12:39:35.129779+0000","last_unstale":"2024-03-24T12:39:37.124840+0000","last_undegraded":"2024-03-24T12:39:37.124840+0000","last_fullsized":"2024-03-24T12:39:37.124840+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:19.376894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:19.376894+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":127,"reported_epoch":258,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:39:37.113549+0000","last_change":"2024-03-24T12:39:35.107270+0000","last_active":"2024-03-24T12:39:37.113549+0000","last_peered":"2024-03-24T12:39:37.113549+0000","last_clean":"2024-03-24T12:39:37.113549+0000","last_became_active":"2024-03-24T12:39:35.106802+0000","last_became_peered":"2024-03-24T12:39:35.106802+0000","last_unstale":"2024-03-24T12:39:37.113549+0000","last_undegraded":"2024-03-24T12:39:37.113549+0000","last_fullsized":"2024-03-24T12:39:37.113549+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:15.004831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:15.004831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,2],"avail_no_missing":["4","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":439,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.112408+0000","last_change":"2024-03-24T12:37:23.800546+0000","last_active":"2024-03-24T12:39:37.112408+0000","last_peered":"2024-03-24T12:39:37.112408+0000","last_clean":"2024-03-24T12:39:37.112408+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:39:37.112408+0000","last_undegraded":"2024-03-24T12:39:37.112408+0000","last_fullsized":"2024-03-24T12:39:37.112408+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:23.800421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:23.800421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"207'19","reported_seq":502,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.113066+0000","last_change":"2024-03-24T12:39:34.767256+0000","last_active":"2024-03-24T12:39:37.113066+0000","last_peered":"2024-03-24T12:39:37.113066+0000","last_clean":"2024-03-24T12:39:37.113066+0000","last_became_active":"2024-03-24T12:39:34.752219+0000","last_became_peered":"2024-03-24T12:39:34.752219+0000","last_unstale":"2024-03-24T12:39:37.113066+0000","last_undegraded":"2024-03-24T12:39:37.113066+0000","last_fullsized":"2024-03-24T12:39:37.113066+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:37:21.827708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:21.827708+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.857695+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"258'11287","reported_seq":26069,"reported_epoch":258,"state":"active+remapped+backfilling","last_fresh":"2024-03-24T12:39:44.714364+0000","last_change":"2024-03-24T12:39:35.208219+0000","last_active":"2024-03-24T12:39:44.714364+0000","last_peered":"2024-03-24T12:39:44.714364+0000","last_clean":"2024-03-24T12:39:33.689744+0000","last_became_active":"2024-03-24T12:39:35.171954+0000","last_became_peered":"2024-03-24T12:39:35.171954+0000","last_unstale":"2024-03-24T12:39:44.714364+0000","last_undegraded":"2024-03-24T12:39:44.714364+0000","last_fullsized":"2024-03-24T12:39:44.714364+0000","mapping_epoch":255,"log_start":"236'6716","ondisk_log_start":"236'6716","created":212,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'5891","last_scrub_stamp":"2024-03-24T12:37:59.634014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:37:59.634014+0000","objects_scrubbed":5822,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2682,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172294144,"num_objects":2629,"num_object_clones":0,"num_object_copies":7887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1346,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11287,"num_write_kb":445312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7297,"num_bytes_recovered":478216192,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,2,0],"avail_no_missing":["1","0","2"],"object_location_counts":[{"shards":"0,1,2","objects":2629}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":435,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.131022+0000","last_change":"2024-03-24T12:37:34.273853+0000","last_active":"2024-03-24T12:39:37.131022+0000","last_peered":"2024-03-24T12:39:37.131022+0000","last_clean":"2024-03-24T12:39:37.131022+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:39:37.131022+0000","last_undegraded":"2024-03-24T12:39:37.131022+0000","last_fullsized":"2024-03-24T12:39:37.131022+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:34.273732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:34.273732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":32,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.113029+0000","last_change":"2024-03-24T12:39:07.753836+0000","last_active":"2024-03-24T12:39:37.113029+0000","last_peered":"2024-03-24T12:39:37.113029+0000","last_clean":"2024-03-24T12:39:37.113029+0000","last_became_active":"2024-03-24T12:39:07.753551+0000","last_became_peered":"2024-03-24T12:39:07.753551+0000","last_unstale":"2024-03-24T12:39:37.113029+0000","last_undegraded":"2024-03-24T12:39:37.113029+0000","last_fullsized":"2024-03-24T12:39:37.113029+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:36:56.908268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:36:56.908268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.120151+0000","last_change":"2024-03-24T12:39:34.745715+0000","last_active":"2024-03-24T12:39:37.120151+0000","last_peered":"2024-03-24T12:39:37.120151+0000","last_clean":"2024-03-24T12:39:37.120151+0000","last_became_active":"2024-03-24T12:39:34.745353+0000","last_became_peered":"2024-03-24T12:39:34.745353+0000","last_unstale":"2024-03-24T12:39:37.120151+0000","last_undegraded":"2024-03-24T12:39:37.120151+0000","last_fullsized":"2024-03-24T12:39:37.120151+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:15.126519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:15.126519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"207'39","reported_seq":492,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.113502+0000","last_change":"2024-03-24T12:37:38.795066+0000","last_active":"2024-03-24T12:39:37.113502+0000","last_peered":"2024-03-24T12:39:37.113502+0000","last_clean":"2024-03-24T12:39:37.113502+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:39:37.113502+0000","last_undegraded":"2024-03-24T12:39:37.113502+0000","last_fullsized":"2024-03-24T12:39:37.113502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-24T12:37:38.794952+0000","last_deep_scrub":"207'39","last_deep_scrub_stamp":"2024-03-24T12:36:22.669686+0000","last_clean_scrub_stamp":"2024-03-24T12:37:38.794952+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1","version":"258'11459","reported_seq":27595,"reported_epoch":258,"state":"active+remapped+backfilling","last_fresh":"2024-03-24T12:39:45.019857+0000","last_change":"2024-03-24T12:39:35.223193+0000","last_active":"2024-03-24T12:39:45.019857+0000","last_peered":"2024-03-24T12:39:45.019857+0000","last_clean":"2024-03-24T12:39:33.702220+0000","last_became_active":"2024-03-24T12:39:35.187275+0000","last_became_peered":"2024-03-24T12:39:35.187275+0000","last_unstale":"2024-03-24T12:39:45.019857+0000","last_undegraded":"2024-03-24T12:39:45.019857+0000","last_fullsized":"2024-03-24T12:39:45.019857+0000","mapping_epoch":255,"log_start":"236'7007","ondisk_log_start":"236'7007","created":212,"last_epoch_clean":244,"parent":"0.0","parent_split_bits":0,"last_scrub":"228'5692","last_scrub_stamp":"2024-03-24T12:37:56.032939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:40.645298+0000","last_clean_scrub_stamp":"2024-03-24T12:37:56.032939+0000","objects_scrubbed":5632,"log_size":4452,"log_dups_size":0,"ondisk_log_size":4452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"no scrub is scheduled","scrub_duration":2179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172556288,"num_objects":2633,"num_object_clones":0,"num_object_copies":7899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1268,"num_objects_unfound":0,"num_objects_dirty":2633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11459,"num_write_kb":450944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7974,"num_bytes_recovered":522584064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,7,4],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":2633}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":435,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.113538+0000","last_change":"2024-03-24T12:37:28.891971+0000","last_active":"2024-03-24T12:39:37.113538+0000","last_peered":"2024-03-24T12:39:37.113538+0000","last_clean":"2024-03-24T12:39:37.113538+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:39:37.113538+0000","last_undegraded":"2024-03-24T12:39:37.113538+0000","last_fullsized":"2024-03-24T12:39:37.113538+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:28.891832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:28.891832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"205'9","reported_seq":600,"reported_epoch":258,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:39:43.372090+0000","last_change":"2024-03-24T12:35:58.752262+0000","last_active":"2024-03-24T12:39:43.372090+0000","last_peered":"2024-03-24T12:39:43.372090+0000","last_clean":"2024-03-24T12:35:54.684037+0000","last_became_active":"2024-03-24T12:35:58.752262+0000","last_became_peered":"2024-03-24T12:35:58.752262+0000","last_unstale":"2024-03-24T12:39:43.372090+0000","last_undegraded":"2024-03-24T12:35:58.701740+0000","last_fullsized":"2024-03-24T12:35:58.701602+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:35:00.998713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.998713+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:47.553908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":431,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.123869+0000","last_change":"2024-03-24T12:37:38.861709+0000","last_active":"2024-03-24T12:39:37.123869+0000","last_peered":"2024-03-24T12:39:37.123869+0000","last_clean":"2024-03-24T12:39:37.123869+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:39:37.123869+0000","last_undegraded":"2024-03-24T12:39:37.123869+0000","last_fullsized":"2024-03-24T12:39:37.123869+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:38.861597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:38.861597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"207'20","reported_seq":509,"reported_epoch":258,"state":"active+clean","last_fresh":"2024-03-24T12:39:37.113200+0000","last_change":"2024-03-24T12:39:34.767616+0000","last_active":"2024-03-24T12:39:37.113200+0000","last_peered":"2024-03-24T12:39:37.113200+0000","last_clean":"2024-03-24T12:39:37.113200+0000","last_became_active":"2024-03-24T12:39:34.759881+0000","last_became_peered":"2024-03-24T12:39:34.759881+0000","last_unstale":"2024-03-24T12:39:37.113200+0000","last_undegraded":"2024-03-24T12:39:37.113200+0000","last_fullsized":"2024-03-24T12:39:37.113200+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:37:35.816046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:35.816046+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:39:39.857731+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2809266200,"num_objects":42867,"num_object_clones":0,"num_object_copies":128601,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21581,"num_objects_misplaced":5517,"num_objects_unfound":0,"num_objects_dirty":42867,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":181261,"num_write_kb":7172033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21636,"num_bytes_recovered":1417936896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7145660416,"data_stored":7145652272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78309,"ondisk_log_size":78309,"up":39,"acting":40,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":812,"num_read_kb":807,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":90,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":23,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1123,"num_read_kb":880,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":753,"ondisk_log_size":753,"up":88,"acting":92,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":83,"acting":86,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":82,"acting":86,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037928,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2043732,"kb_used_data":1575244,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91701420,"statfs":{"total":95995035648,"available":93902254080,"internally_reserved":0,"allocated":1613049856,"data_stored":1612448200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":479721930},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332270,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":672376,"kb_used_data":308016,"kb_used_omap":1,"kb_used_meta":364350,"kb_avail":93072776,"statfs":{"total":95995035648,"available":95306522624,"internally_reserved":0,"allocated":315408384,"data_stored":314834788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":373094855},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1977968,"kb_used_data":1509416,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":91767184,"statfs":{"total":95995035648,"available":93969596416,"internally_reserved":0,"allocated":1545641984,"data_stored":1545012377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738573,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1419676,"kb_used_data":915092,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92325476,"statfs":{"total":95995035648,"available":94541287424,"internally_reserved":0,"allocated":937054208,"data_stored":936443919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411523,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1154420,"kb_used_data":690092,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":92590732,"statfs":{"total":95995035648,"available":94812909568,"internally_reserved":0,"allocated":706654208,"data_stored":706083624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":475462085},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117180,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1852508,"kb_used_data":1325588,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91892644,"statfs":{"total":95995035648,"available":94098067456,"internally_reserved":0,"allocated":1357402112,"data_stored":1356792012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":539425223},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920952,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1062224,"kb_used_data":710344,"kb_used_omap":1,"kb_used_meta":351870,"kb_avail":92682928,"statfs":{"total":95995035648,"available":94907318272,"internally_reserved":0,"allocated":727392256,"data_stored":726793380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":360315333},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626566,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957988,"kb_used_data":688028,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92787164,"statfs":{"total":95995035648,"available":95014055936,"internally_reserved":0,"allocated":704540672,"data_stored":704173829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":926875648,"data_stored":926875648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1533022208,"data_stored":1533018136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":695664640,"data_stored":695664640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1352794112,"data_stored":1352794112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":713883648,"data_stored":713883648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":323026944,"data_stored":323026944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1600393216,"data_stored":1600389144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:40:06.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-24T12:40:06.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:05 smithi190 ceph-mon[33748]: pgmap v585: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.7 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 651 op/s; 13912/85437 objects degraded (16.283%); 1847/85437 objects misplaced (2.162%) 2024-03-24T12:40:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:05 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 13912/85437 objects degraded (16.283%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:06.397 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:06.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:06.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:06.600 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:06.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:06.606 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:06.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:06.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:06.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:06.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:06.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:06.861 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:06.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:06.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:06.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:06.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:06.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:07.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:07.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:07.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:07.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:07.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:07.248 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:07.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:07.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:07.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:07.375 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:07.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:07.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:06 smithi190 ceph-mon[33748]: 3.1 scrub starts 2024-03-24T12:40:07.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:06 smithi190 ceph-mon[33748]: 3.1 scrub ok 2024-03-24T12:40:07.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:06 smithi190 ceph-mon[33748]: 5.2 scrub starts 2024-03-24T12:40:07.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:06 smithi190 ceph-mon[33748]: 5.2 scrub ok 2024-03-24T12:40:07.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:06 smithi029 ceph-mon[29919]: 3.1 scrub starts 2024-03-24T12:40:07.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:06 smithi029 ceph-mon[29919]: 3.1 scrub ok 2024-03-24T12:40:07.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:06 smithi029 ceph-mon[29919]: 5.2 scrub starts 2024-03-24T12:40:07.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:06 smithi029 ceph-mon[29919]: 5.2 scrub ok 2024-03-24T12:40:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:06 smithi029 ceph-mon[41500]: 3.1 scrub starts 2024-03-24T12:40:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:06 smithi029 ceph-mon[41500]: 3.1 scrub ok 2024-03-24T12:40:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:06 smithi029 ceph-mon[41500]: 5.2 scrub starts 2024-03-24T12:40:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:06 smithi029 ceph-mon[41500]: 5.2 scrub ok 2024-03-24T12:40:07.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:07.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:07.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:07.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:07.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:07.633 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:07.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:07.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:07.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:07.763 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:07.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:07.864 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:07.893 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:07.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:07.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:40:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:40:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:40:07.994 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:08.021 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:08.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:08.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:08.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:08.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:08.183 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:08.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:08.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:08.286 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:07 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:40:08.286 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:07 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:40:08.286 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:07 smithi029 ceph-mon[29919]: pgmap v586: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 949 op/s; 12622/75660 objects degraded (16.683%); 1485/75660 objects misplaced (1.963%) 2024-03-24T12:40:08.286 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:07 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:40:08.287 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:07 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:40:08.287 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:07 smithi029 ceph-mon[41500]: pgmap v586: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 949 op/s; 12622/75660 objects degraded (16.683%); 1485/75660 objects misplaced (1.963%) 2024-03-24T12:40:08.313 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:08.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:08.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:40:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:40:08.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:07 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:40:08.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:07 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:40:08.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:07 smithi190 ceph-mon[33748]: pgmap v586: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.5 GiB data, 8.0 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 949 op/s; 12622/75660 objects degraded (16.683%); 1485/75660 objects misplaced (1.963%) 2024-03-24T12:40:08.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:08.453 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:08.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:08.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:08.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:08.582 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:08.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:08.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:08.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:08.712 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:08.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:08.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:08.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:08.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:08.998 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:08.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:08.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:09.083 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:40:09.084 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:40:09.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:09.111 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:40:09.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:09.144 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:09.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:09.159 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:40:09.197 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:40:09.197 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:08 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:40:09.197 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:08 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:40:09.197 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:08 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:40:09.197 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:08 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:40:09.231 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:08 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:40:09.231 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:08 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:40:09.236 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:40:09.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:09.273 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:40:09.273 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:40:09.273 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:40:09.273 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:40:09.273 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:40:09.273 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:40:09.273 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:40:09.273 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:40:09.274 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:40:09.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:09.275 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:09.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:09.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:09.403 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:09.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:09.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:09.542 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:09.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:09.543 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:09.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:09.672 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:09.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:09.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:09.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:09.819 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:09.820 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:09.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:09.962 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:09.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:09.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:10.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:40:10.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:10.101 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:10.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:10.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:10.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:10.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:09 smithi190 ceph-mon[33748]: pgmap v587: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 952 op/s; 11609/70470 objects degraded (16.474%); 1485/70470 objects misplaced (2.107%) 2024-03-24T12:40:10.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:09 smithi190 ceph-mon[33748]: 3.10 scrub starts 2024-03-24T12:40:10.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:09 smithi190 ceph-mon[33748]: 3.10 scrub ok 2024-03-24T12:40:10.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:09 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1917300434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:10.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:10.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:10.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:10.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:10.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:10.403 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:10.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:09 smithi029 ceph-mon[29919]: pgmap v587: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 952 op/s; 11609/70470 objects degraded (16.474%); 1485/70470 objects misplaced (2.107%) 2024-03-24T12:40:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:09 smithi029 ceph-mon[29919]: 3.10 scrub starts 2024-03-24T12:40:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:09 smithi029 ceph-mon[29919]: 3.10 scrub ok 2024-03-24T12:40:10.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:09 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1917300434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:10.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:10.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:09 smithi029 ceph-mon[41500]: pgmap v587: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 952 op/s; 11609/70470 objects degraded (16.474%); 1485/70470 objects misplaced (2.107%) 2024-03-24T12:40:10.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:09 smithi029 ceph-mon[41500]: 3.10 scrub starts 2024-03-24T12:40:10.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:09 smithi029 ceph-mon[41500]: 3.10 scrub ok 2024-03-24T12:40:10.421 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:09 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1917300434' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:10.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:10.543 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:10.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:10.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:10.683 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:10.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:10.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:10.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:10.811 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:10.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:10.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:10.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:10.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:10.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:11.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:11.081 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:11.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:11.171 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:11.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:11.239 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:11.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:11.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:11.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:10 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:40:11.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:10 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:40:11.297 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:10 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 11609/70470 objects degraded (16.474%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:11.297 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:10 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:40:11.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:10 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:40:11.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:10 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 11609/70470 objects degraded (16.474%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:11.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:10 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:40:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:10 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:40:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:10 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 11609/70470 objects degraded (16.474%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:11.387 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:11.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:11.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:11.517 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:11.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:11.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:11.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:11.662 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:11.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:11.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:11.764 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:11.806 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:11.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:11.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:11.908 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:11.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:11.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:12.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:12.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:12.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:12.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:12.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:11 smithi029 ceph-mon[41500]: 4.b scrub starts 2024-03-24T12:40:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:11 smithi029 ceph-mon[41500]: 4.b scrub ok 2024-03-24T12:40:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:11 smithi029 ceph-mon[41500]: pgmap v588: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 968 B/s rd, 0 B/s wr, 995 op/s; 10468/61146 objects degraded (17.120%); 1485/61146 objects misplaced (2.429%) 2024-03-24T12:40:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:11 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:40:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:11 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:40:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:11 smithi029 ceph-mon[29919]: 4.b scrub starts 2024-03-24T12:40:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:11 smithi029 ceph-mon[29919]: 4.b scrub ok 2024-03-24T12:40:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:11 smithi029 ceph-mon[29919]: pgmap v588: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 968 B/s rd, 0 B/s wr, 995 op/s; 10468/61146 objects degraded (17.120%); 1485/61146 objects misplaced (2.429%) 2024-03-24T12:40:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:11 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:40:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:11 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:40:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:12.199 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:12.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:40:12.298 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:40:12.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:12.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:12.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:12.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:11 smithi190 ceph-mon[33748]: 4.b scrub starts 2024-03-24T12:40:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:11 smithi190 ceph-mon[33748]: 4.b scrub ok 2024-03-24T12:40:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:11 smithi190 ceph-mon[33748]: pgmap v588: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.2 GiB data, 7.3 GiB used, 708 GiB / 715 GiB avail; 968 B/s rd, 0 B/s wr, 995 op/s; 10468/61146 objects degraded (17.120%); 1485/61146 objects misplaced (2.429%) 2024-03-24T12:40:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:11 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:40:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:11 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:40:12.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:12.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:12.462 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:12.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:12.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:12.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:12.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:12.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:12.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:12.729 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:12.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:12.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:12.858 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:12.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:12.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:12.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:13.000 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:13.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:13.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:13.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:13.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:13.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:13.252 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:13.261 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:13.279 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:13.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:13.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:13.381 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:12 smithi190 ceph-mon[33748]: 9.2 scrub starts 2024-03-24T12:40:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:12 smithi190 ceph-mon[33748]: 9.2 scrub ok 2024-03-24T12:40:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:13.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:13.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:13.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:12 smithi029 ceph-mon[41500]: 9.2 scrub starts 2024-03-24T12:40:13.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:12 smithi029 ceph-mon[41500]: 9.2 scrub ok 2024-03-24T12:40:13.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:12 smithi029 ceph-mon[29919]: 9.2 scrub starts 2024-03-24T12:40:13.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:12 smithi029 ceph-mon[29919]: 9.2 scrub ok 2024-03-24T12:40:13.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:13.541 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:13.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:13.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:13.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:13.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:13.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:13.769 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:13.817 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:13.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:13.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:13.919 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:13.946 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:13.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:14.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:14.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:14.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:14.110 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.24989.0:210819 9.8 e266/212 9:12e69779:::benchmark_data_smithi029_134815_object98753:head v 266'13216, mlcod=266'13214)", 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:40:14.110566+0000", 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0028668579999999999, 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0029037680000000001, 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110566+0000", 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:14.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110566+0000", 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110567+0000", 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4499999999999997e-07 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:14.112 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110572+0000", 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4739999999999996e-06 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110580+0000", 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8620000000000003e-06 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110586+0000", 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6180000000000003e-06 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.113 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110655+0000", 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9562999999999998e-05 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110677+0000", 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1871000000000001e-05 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:210816 9.5 9:ac4cc104:::benchmark_data_smithi029_134815_object98750:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-03-24T12:40:14.114 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:40:14.106825+0000", 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0066083180000000002, 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0067296680000000003, 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 210816 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106825+0000", 2024-03-24T12:40:14.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106825+0000", 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106826+0000", 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.097e-06 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:14.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106830+0000", 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6909999999999996e-06 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106842+0000", 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1805e-05 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106851+0000", 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.541e-06 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:14.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106907+0000", 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6363999999999998e-05 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.107074+0000", 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000167388 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 7", 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.107183+0000", 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000108414 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:14.118 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:210817 9.5 9.d3718ad5 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e266)", 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:40:14.106873+0000", 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0065598059999999996, 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.006765526, 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 210817 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.119 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106873+0000", 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106873+0000", 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106874+0000", 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0100000000000005e-07 2024-03-24T12:40:14.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106877+0000", 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0019999999999998e-06 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106885+0000", 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8820000000000001e-06 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.106891+0000", 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4300000000000003e-06 2024-03-24T12:40:14.121 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.24989.0:210816 9.5 e266/212)", 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:40:14.110181+0000", 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.003252203, 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0035181449999999999, 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:14.122 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110181+0000", 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110181+0000", 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110181+0000", 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.13e-07 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:14.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110185+0000", 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6289999999999999e-06 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110193+0000", 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1580000000000005e-06 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:40:14.110199+0000", 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8710000000000002e-06 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:14.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:14.125 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:14.125 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:40:14.125 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 4 2024-03-24T12:40:14.125 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:40:14.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:13 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:40:14.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:13 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:40:14.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:13 smithi029 ceph-mon[29919]: 4.10 scrub starts 2024-03-24T12:40:14.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:13 smithi029 ceph-mon[29919]: 4.10 scrub ok 2024-03-24T12:40:14.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:13 smithi029 ceph-mon[29919]: pgmap v589: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 8531/51798 objects degraded (16.470%); 1087/51798 objects misplaced (2.099%) 2024-03-24T12:40:14.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:13 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:40:14.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:13 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:40:14.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:13 smithi029 ceph-mon[41500]: 4.10 scrub starts 2024-03-24T12:40:14.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:13 smithi029 ceph-mon[41500]: 4.10 scrub ok 2024-03-24T12:40:14.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:13 smithi029 ceph-mon[41500]: pgmap v589: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 8531/51798 objects degraded (16.470%); 1087/51798 objects misplaced (2.099%) 2024-03-24T12:40:14.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:14.220 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:14.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:14.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:14.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:40:14.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:14.348 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:14.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:14.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:13 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:40:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:13 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:40:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:13 smithi190 ceph-mon[33748]: 4.10 scrub starts 2024-03-24T12:40:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:13 smithi190 ceph-mon[33748]: 4.10 scrub ok 2024-03-24T12:40:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:13 smithi190 ceph-mon[33748]: pgmap v589: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s; 8531/51798 objects degraded (16.470%); 1087/51798 objects misplaced (2.099%) 2024-03-24T12:40:14.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:14.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:14.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:14.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:14.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:14.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:14.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:14.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:14.778 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:14.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:14.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:14.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:14.911 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:14.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:15.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:15.038 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:15.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:15.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:14 smithi029 ceph-mon[29919]: 5.1a scrub starts 2024-03-24T12:40:15.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:14 smithi029 ceph-mon[29919]: 5.1a scrub ok 2024-03-24T12:40:15.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:14 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:40:15.105 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:14 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:40:15.105 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:14 smithi029 ceph-mon[41500]: 5.1a scrub starts 2024-03-24T12:40:15.105 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:14 smithi029 ceph-mon[41500]: 5.1a scrub ok 2024-03-24T12:40:15.105 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:14 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:40:15.105 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:14 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:40:15.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:15.179 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:15.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:15.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:15.281 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:15.307 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:15.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:15.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:14 smithi190 ceph-mon[33748]: 5.1a scrub starts 2024-03-24T12:40:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:14 smithi190 ceph-mon[33748]: 5.1a scrub ok 2024-03-24T12:40:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:14 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:40:15.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:14 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:40:15.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:15.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:15.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:15.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:15.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:15.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:15.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:15.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:15.690 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:15.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:15.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:15.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:15.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:15.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:15.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:15.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:15.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:16.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:16.102 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:16.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:16.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:16.229 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:16.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:16.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:16.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:16.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:16 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:40:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:16 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:40:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:16 smithi190 ceph-mon[33748]: pgmap v590: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 992 op/s; 8112/50541 objects degraded (16.050%); 1087/50541 objects misplaced (2.151%) 2024-03-24T12:40:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:16 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 8112/50541 objects degraded (16.050%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:16.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:16 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:40:16.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:16 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:40:16.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:16 smithi029 ceph-mon[29919]: pgmap v590: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 992 op/s; 8112/50541 objects degraded (16.050%); 1087/50541 objects misplaced (2.151%) 2024-03-24T12:40:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:16 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 8112/50541 objects degraded (16.050%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:16.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:16 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:40:16.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:16 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:40:16.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:16 smithi029 ceph-mon[41500]: pgmap v590: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 1.0 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 992 op/s; 8112/50541 objects degraded (16.050%); 1087/50541 objects misplaced (2.151%) 2024-03-24T12:40:16.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:16 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 8112/50541 objects degraded (16.050%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:16.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:16.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:16.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:16.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:16.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:16.629 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:16.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:16.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:16.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T12:40:16.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:16.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:16.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:16.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:16.919 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:16.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:16.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:17.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:17.074 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:17.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:17.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:17.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:17.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:17.352 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:17.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:17.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:17 smithi190 ceph-mon[33748]: 6.5 deep-scrub starts 2024-03-24T12:40:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:17 smithi190 ceph-mon[33748]: 6.5 deep-scrub ok 2024-03-24T12:40:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:17 smithi190 ceph-mon[33748]: 5.1d deep-scrub starts 2024-03-24T12:40:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:17 smithi190 ceph-mon[33748]: 5.1d deep-scrub ok 2024-03-24T12:40:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:17 smithi190 ceph-mon[33748]: 4.d scrub starts 2024-03-24T12:40:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:17 smithi190 ceph-mon[33748]: 4.d scrub ok 2024-03-24T12:40:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:17 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:40:17.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:17 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:40:17.400 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[41500]: 6.5 deep-scrub starts 2024-03-24T12:40:17.400 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[41500]: 6.5 deep-scrub ok 2024-03-24T12:40:17.400 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[41500]: 5.1d deep-scrub starts 2024-03-24T12:40:17.400 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[41500]: 5.1d deep-scrub ok 2024-03-24T12:40:17.400 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[41500]: 4.d scrub starts 2024-03-24T12:40:17.400 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[41500]: 4.d scrub ok 2024-03-24T12:40:17.401 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:40:17.401 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:40:17.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[29919]: 6.5 deep-scrub starts 2024-03-24T12:40:17.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[29919]: 6.5 deep-scrub ok 2024-03-24T12:40:17.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[29919]: 5.1d deep-scrub starts 2024-03-24T12:40:17.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[29919]: 5.1d deep-scrub ok 2024-03-24T12:40:17.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[29919]: 4.d scrub starts 2024-03-24T12:40:17.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[29919]: 4.d scrub ok 2024-03-24T12:40:17.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:40:17.402 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:40:17.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:17.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:17.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:17.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:17.630 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:17.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:17.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:40:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:40:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:40:17.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:17.784 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:17.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:17.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:17.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:17.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:17.928 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:17.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:18.029 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:18.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:18.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:18.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:18.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[41500]: 5.0 scrub starts 2024-03-24T12:40:18.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[41500]: 5.0 scrub ok 2024-03-24T12:40:18.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[41500]: 5.1e scrub starts 2024-03-24T12:40:18.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[41500]: 5.1e scrub ok 2024-03-24T12:40:18.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[41500]: pgmap v591: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 807 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 6595/39312 objects degraded (16.776%); 706/39312 objects misplaced (1.796%) 2024-03-24T12:40:18.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:40:18.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:40:18.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:18.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[29919]: 5.0 scrub starts 2024-03-24T12:40:18.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[29919]: 5.0 scrub ok 2024-03-24T12:40:18.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[29919]: 5.1e scrub starts 2024-03-24T12:40:18.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[29919]: 5.1e scrub ok 2024-03-24T12:40:18.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[29919]: pgmap v591: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 807 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 6595/39312 objects degraded (16.776%); 706/39312 objects misplaced (1.796%) 2024-03-24T12:40:18.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:40:18.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:40:18.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:18.200 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:18.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:18.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:18.339 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:18.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:18.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:18.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:40:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:40:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:18 smithi190 ceph-mon[33748]: 5.0 scrub starts 2024-03-24T12:40:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:18 smithi190 ceph-mon[33748]: 5.0 scrub ok 2024-03-24T12:40:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:18 smithi190 ceph-mon[33748]: 5.1e scrub starts 2024-03-24T12:40:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:18 smithi190 ceph-mon[33748]: 5.1e scrub ok 2024-03-24T12:40:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:18 smithi190 ceph-mon[33748]: pgmap v591: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 807 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 6595/39312 objects degraded (16.776%); 706/39312 objects misplaced (1.796%) 2024-03-24T12:40:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:18 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:40:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:18 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:40:18.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:18.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:18.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:18.507 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:18.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:18.509 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:18.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:18.650 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:18.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:18.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:18.840 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:18.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:18.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:18.968 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:18.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:18.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:19.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:19.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:19 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:40:19.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:19 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:40:19.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:19 smithi029 ceph-mon[41500]: 5.18 deep-scrub starts 2024-03-24T12:40:19.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:19 smithi029 ceph-mon[41500]: 5.18 deep-scrub ok 2024-03-24T12:40:19.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:19 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:40:19.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:19 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:40:19.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:19 smithi029 ceph-mon[29919]: 5.18 deep-scrub starts 2024-03-24T12:40:19.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:19 smithi029 ceph-mon[29919]: 5.18 deep-scrub ok 2024-03-24T12:40:19.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:19.120 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:19.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:19.221 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:19.248 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:19.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:19.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:19.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:19.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:19 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:40:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:19 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:40:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:19 smithi190 ceph-mon[33748]: 5.18 deep-scrub starts 2024-03-24T12:40:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:19 smithi190 ceph-mon[33748]: 5.18 deep-scrub ok 2024-03-24T12:40:19.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:19.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:19.509 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:19.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:19.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:19.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:19.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:19.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:19.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:19.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:19.889 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:19.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:19.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:20.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:20.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:20.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:20.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:20.193 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:20.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:20.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:20.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:20.322 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:20.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:20 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:20 smithi190 ceph-mon[33748]: pgmap v592: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 692 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 5500/33795 objects degraded (16.275%); 706/33795 objects misplaced (2.089%) 2024-03-24T12:40:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:20 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:40:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:20 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:40:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:20 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2385445088' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-24T12:40:20.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:20 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:20.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:20 smithi029 ceph-mon[41500]: pgmap v592: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 692 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 5500/33795 objects degraded (16.275%); 706/33795 objects misplaced (2.089%) 2024-03-24T12:40:20.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:20 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:40:20.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:20 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:40:20.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:20 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2385445088' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-24T12:40:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:20 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:20 smithi029 ceph-mon[29919]: pgmap v592: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 692 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.16k op/s; 5500/33795 objects degraded (16.275%); 706/33795 objects misplaced (2.089%) 2024-03-24T12:40:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:20 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:40:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:20 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:40:20.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:20 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2385445088' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-24T12:40:20.419 INFO:teuthology.orchestra.run.smithi029.stderr:set pool 1 pgp_num to 1 2024-03-24T12:40:20.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:20.450 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:20.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:20.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:20.590 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:20.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:20.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:20.730 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:20.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:20.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:20.833 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:20.870 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:20.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:20.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:20.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:21.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:21.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:21.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:21.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:21.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:21.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:21.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:21.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:21.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: 9.0 scrub starts 2024-03-24T12:40:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:40:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: 9.0 scrub ok 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2385445088' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: osdmap e267: 8 total, 7 up, 8 in 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 5500/33795 objects degraded (16.275%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: osdmap e268: 8 total, 7 up, 8 in 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4084504403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:40:21.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:21 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:40:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:21.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:21.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: 9.0 scrub starts 2024-03-24T12:40:21.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:40:21.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:40:21.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: 9.0 scrub ok 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2385445088' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: osdmap e267: 8 total, 7 up, 8 in 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 5500/33795 objects degraded (16.275%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: osdmap e268: 8 total, 7 up, 8 in 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4084504403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:40:21.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: 9.0 scrub starts 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: 9.0 scrub ok 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2385445088' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: osdmap e267: 8 total, 7 up, 8 in 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 5500/33795 objects degraded (16.275%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: osdmap e268: 8 total, 7 up, 8 in 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4084504403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:40:21.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:40:21.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:21 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:40:21.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:21.527 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:21.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:21.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:21.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:21.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:21.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:21.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:21.794 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:21.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:21.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:21.926 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:21.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:22.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:22.069 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:22.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:22.172 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:22.211 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:22.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:40:22.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[29919]: 3.1e scrub starts 2024-03-24T12:40:22.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[29919]: 3.1e scrub ok 2024-03-24T12:40:22.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[29919]: 9.4 deep-scrub starts 2024-03-24T12:40:22.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[29919]: pgmap v595: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 504 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s; 4322/24774 objects degraded (17.446%); 706/24774 objects misplaced (2.850%) 2024-03-24T12:40:22.290 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[29919]: osdmap e269: 8 total, 7 up, 8 in 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[41500]: 3.1e scrub starts 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[41500]: 3.1e scrub ok 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[41500]: 9.4 deep-scrub starts 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[41500]: pgmap v595: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 504 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s; 4322/24774 objects degraded (17.446%); 706/24774 objects misplaced (2.850%) 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:40:22.291 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:40:22.292 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:22 smithi029 ceph-mon[41500]: osdmap e269: 8 total, 7 up, 8 in 2024-03-24T12:40:22.314 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:22.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:22.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:22 smithi190 ceph-mon[33748]: 3.1e scrub starts 2024-03-24T12:40:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:22 smithi190 ceph-mon[33748]: 3.1e scrub ok 2024-03-24T12:40:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:22 smithi190 ceph-mon[33748]: 9.4 deep-scrub starts 2024-03-24T12:40:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:22 smithi190 ceph-mon[33748]: pgmap v595: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 504 MiB data, 5.4 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.12k op/s; 4322/24774 objects degraded (17.446%); 706/24774 objects misplaced (2.850%) 2024-03-24T12:40:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:22 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:40:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:22 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:40:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:22 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:40:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:22 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:40:22.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:22 smithi190 ceph-mon[33748]: osdmap e269: 8 total, 7 up, 8 in 2024-03-24T12:40:22.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:22.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:22.483 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:22.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:22.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:22.628 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:22.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:22.729 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:22.757 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:22.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:22.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:22.884 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:22.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:22.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:22.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:23.014 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:23.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:23.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:23.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:23.155 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:23.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:23.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:23.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:23.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:23.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:23.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:23 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:40:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:23 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:40:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:23 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:40:23.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:23 smithi190 ceph-mon[33748]: osdmap e270: 8 total, 7 up, 8 in 2024-03-24T12:40:23.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:23.416 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:23.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:23.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:23.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:23 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:40:23.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:23 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:40:23.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:23 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:40:23.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:23 smithi029 ceph-mon[41500]: osdmap e270: 8 total, 7 up, 8 in 2024-03-24T12:40:23.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:23 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:40:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:23 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:40:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:23 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:40:23.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:23 smithi029 ceph-mon[29919]: osdmap e270: 8 total, 7 up, 8 in 2024-03-24T12:40:23.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:23.557 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:23.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:23.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:23.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:23.685 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:23.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:23.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:23.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:23.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:23.812 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:23.812 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:23.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:23.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:23.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:24.029 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:24.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:24.068 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:24.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:24.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:24.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:24.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:24.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:24.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:24.325 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:24.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:24 smithi190 ceph-mon[33748]: 9.4 deep-scrub ok 2024-03-24T12:40:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:24 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:40:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:24 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:40:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:24 smithi190 ceph-mon[33748]: pgmap v598: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 341 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 2697/16935 objects degraded (15.926%); 350/16935 objects misplaced (2.067%) 2024-03-24T12:40:24.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:24 smithi190 ceph-mon[33748]: osdmap e271: 8 total, 7 up, 8 in 2024-03-24T12:40:24.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:24 smithi029 ceph-mon[29919]: 9.4 deep-scrub ok 2024-03-24T12:40:24.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:24 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:40:24.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:24 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:40:24.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:24 smithi029 ceph-mon[29919]: pgmap v598: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 341 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 2697/16935 objects degraded (15.926%); 350/16935 objects misplaced (2.067%) 2024-03-24T12:40:24.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:24 smithi029 ceph-mon[29919]: osdmap e271: 8 total, 7 up, 8 in 2024-03-24T12:40:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:24 smithi029 ceph-mon[41500]: 9.4 deep-scrub ok 2024-03-24T12:40:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:24 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:40:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:24 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:40:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:24 smithi029 ceph-mon[41500]: pgmap v598: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 341 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 2697/16935 objects degraded (15.926%); 350/16935 objects misplaced (2.067%) 2024-03-24T12:40:24.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:24 smithi029 ceph-mon[41500]: osdmap e271: 8 total, 7 up, 8 in 2024-03-24T12:40:24.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:24.453 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:24.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:24.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:24.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:24.598 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:24.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:24.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:24.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:24.740 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:24.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:24.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:24.879 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:24.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:24.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:24.980 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:25.019 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:25.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:25.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:25.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:25.149 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:25.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:25.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:25.251 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:25.292 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:25.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:25.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:25.393 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:25.418 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:25.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:25.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:25.551 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:25.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:25.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:25.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:25.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:25.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:25.761 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Removed 112063 objects 2024-03-24T12:40:25.761 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Clean up completed and total clean up time :121.826 2024-03-24T12:40:25.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:25.803 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:25.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:25.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:25.928 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:25.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:26.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:26.094 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:26.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:26.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:26.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:26.220 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:26.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:26.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:26.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:26.346 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:26.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:26 smithi190 ceph-mon[33748]: pgmap v600: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 320 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 962 op/s; 2359/15921 objects degraded (14.817%); 350/15921 objects misplaced (2.198%) 2024-03-24T12:40:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:26 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 2359/15921 objects degraded (14.817%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:26 smithi190 ceph-mon[33748]: osdmap e272: 8 total, 7 up, 8 in 2024-03-24T12:40:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:26 smithi029 ceph-mon[29919]: pgmap v600: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 320 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 962 op/s; 2359/15921 objects degraded (14.817%); 350/15921 objects misplaced (2.198%) 2024-03-24T12:40:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:26 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 2359/15921 objects degraded (14.817%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:26 smithi029 ceph-mon[29919]: osdmap e272: 8 total, 7 up, 8 in 2024-03-24T12:40:26.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:26 smithi029 ceph-mon[41500]: pgmap v600: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 320 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 962 op/s; 2359/15921 objects degraded (14.817%); 350/15921 objects misplaced (2.198%) 2024-03-24T12:40:26.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:26 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 2359/15921 objects degraded (14.817%), 16 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:26.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:26 smithi029 ceph-mon[41500]: osdmap e272: 8 total, 7 up, 8 in 2024-03-24T12:40:26.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:26.472 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:26.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:26.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:26.596 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:26.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:26.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:26.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:26.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:26.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:40:26.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:26.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:26.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:26.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:26.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:26.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:27.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:27.103 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:27.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:27.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:27.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:27.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:27.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:27.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:27.335 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:27.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:27.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:27.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:27.490 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:27.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:27.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:27.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:27.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:27.619 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:27.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:27.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:27.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:27.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:27.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:27.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:27.878 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:27.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:27.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:40:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:40:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:40:27.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:28.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:28.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:28.108 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:28.136 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:28.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:28.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:28.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:28.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:28.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:28.367 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:28.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:28 smithi190 ceph-mon[33748]: pgmap v602: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 114 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 1.12k op/s; 1061/6045 objects degraded (17.552%); 26/6045 objects misplaced (0.430%) 2024-03-24T12:40:28.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:40:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:40:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:28.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:28.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:28.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:28 smithi029 ceph-mon[29919]: pgmap v602: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 114 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 1.12k op/s; 1061/6045 objects degraded (17.552%); 26/6045 objects misplaced (0.430%) 2024-03-24T12:40:28.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:28 smithi029 ceph-mon[41500]: pgmap v602: 148 pgs: 18 active+clean+remapped, 92 active+clean, 22 active+undersized, 16 active+undersized+degraded; 114 MiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 1.12k op/s; 1061/6045 objects degraded (17.552%); 26/6045 objects misplaced (0.430%) 2024-03-24T12:40:28.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:28.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:28.523 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:28.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:28.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:28.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:28.651 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:28.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:28.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:28.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:28.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:28.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:28.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:28.906 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:28.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:28.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:29.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:29.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:29.036 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:29.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:29.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:29.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:29.161 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:29.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:29.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:29.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:29.392 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:29.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:29.416 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:29.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:29.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:29.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:29.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:29.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:29.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:29.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:29.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:29.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:29.798 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:29.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:29.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:29.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:29.927 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:29.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:29.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:30.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:30.056 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:30.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:30.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:30.158 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:30.184 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:30.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:30.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:30.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:30.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:30.315 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:30.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:30 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:30 smithi190 ceph-mon[33748]: pgmap v603: 148 pgs: 18 active+clean+remapped, 92 active+clean, 23 active+undersized, 15 active+undersized+degraded; 37 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 984 op/s; 316/2361 objects degraded (13.384%); 26/2361 objects misplaced (1.101%) 2024-03-24T12:40:30.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:30.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:30 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:30.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:30 smithi029 ceph-mon[41500]: pgmap v603: 148 pgs: 18 active+clean+remapped, 92 active+clean, 23 active+undersized, 15 active+undersized+degraded; 37 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 984 op/s; 316/2361 objects degraded (13.384%); 26/2361 objects misplaced (1.101%) 2024-03-24T12:40:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:30 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:30 smithi029 ceph-mon[29919]: pgmap v603: 148 pgs: 18 active+clean+remapped, 92 active+clean, 23 active+undersized, 15 active+undersized+degraded; 37 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 984 op/s; 316/2361 objects degraded (13.384%); 26/2361 objects misplaced (1.101%) 2024-03-24T12:40:30.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:30.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:30.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:30.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:30.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:30.674 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:30.683 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-24T12:40:30.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-24T12:40:30.702 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:30.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:30.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:30.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:30.827 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:30.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:30.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:30.953 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:30.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:31.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:31.079 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:31.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:31.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:31.184 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:31 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 316/2361 objects degraded (13.384%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:31.184 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:31 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 316/2361 objects degraded (13.384%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:31.206 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:31.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:31.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:31.331 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:31.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:31.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:31.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:31 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 316/2361 objects degraded (13.384%), 15 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:31.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:31.458 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:31.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:31.559 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:31.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:31.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:31.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:31.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:31.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:31.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:31.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:31.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:31.969 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:31.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:32.030 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:32.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:32.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:32.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:32.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:32.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:32.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:32.227 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:32.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:32.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:32 smithi029 ceph-mon[41500]: pgmap v604: 148 pgs: 18 active+clean+remapped, 92 active+clean, 26 active+undersized, 12 active+undersized+degraded; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 681 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:32.299 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:32 smithi029 ceph-mon[29919]: pgmap v604: 148 pgs: 18 active+clean+remapped, 92 active+clean, 26 active+undersized, 12 active+undersized+degraded; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 681 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:32.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:32.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:32.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:32 smithi190 ceph-mon[33748]: pgmap v604: 148 pgs: 18 active+clean+remapped, 92 active+clean, 26 active+undersized, 12 active+undersized+degraded; 582 KiB data, 4.2 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 681 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:32.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:32.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:32.484 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:32.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:32.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:32.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:32.610 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:32.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:32.711 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:32.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:32.739 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:32.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:32.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:32.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:32.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:32.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:32.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:32.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:33.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:33.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:33.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:33.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:33.222 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:33.250 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:33.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:33.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:33.352 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:33.379 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:33.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:33.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:33.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:33.510 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:33.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:33.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:33.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:33.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:33.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:33.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:33.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:33.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:33.812 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:40:33.831 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:40:33.851 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:40:33.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:33.882 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:40:33.896 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:33.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:33.902 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:40:33.925 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:40:33.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:33 smithi029 ceph-mon[29919]: pgmap v605: 148 pgs: 18 active+clean+remapped, 92 active+clean, 26 active+undersized, 12 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 570 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:33.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:40:33.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:33.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:33 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3611757386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:33.926 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:33 smithi029 ceph-mon[41500]: pgmap v605: 148 pgs: 18 active+clean+remapped, 92 active+clean, 26 active+undersized, 12 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 570 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:33.926 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:40:33.926 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:33.926 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:33 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3611757386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:33.945 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:40:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:40:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:40:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:40:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:40:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:40:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:40:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:40:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:40:33.998 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:34.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:34.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:34.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:34.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:34.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:34.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:34.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:34.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:34.280 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:34.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:34.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:33 smithi190 ceph-mon[33748]: pgmap v605: 148 pgs: 18 active+clean+remapped, 92 active+clean, 26 active+undersized, 12 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 570 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:40:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:34.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:33 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3611757386' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:34.407 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:34.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:34.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:34.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:34.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:34.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:34.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:34.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:34.659 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:34.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:34.760 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:34.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:34.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:34.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:34.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:34.915 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:34.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:35.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:35.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:35.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:35.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:35.174 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:35.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:35.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:35.295 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:40:35.295 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:40:35.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:35.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:35.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:35.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:35.429 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:35.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:35.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:35.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:35.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:35.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:35.684 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:40:35.684 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:40:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:35.685 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:35.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:35.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:35.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:35.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:35.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:35.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:35.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:35.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:35.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:36.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:36.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:36.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:35 smithi029 ceph-mon[29919]: pgmap v606: 148 pgs: 18 active+clean+remapped, 92 active+clean, 26 active+undersized, 12 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 511 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:35 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 26/594 objects degraded (4.377%), 12 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:35 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:40:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:35 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:40:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:35 smithi029 ceph-mon[41500]: pgmap v606: 148 pgs: 18 active+clean+remapped, 92 active+clean, 26 active+undersized, 12 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 511 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:35 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 26/594 objects degraded (4.377%), 12 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:35 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:40:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:35 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:40:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:36.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:36.200 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:36.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:36.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:36.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:36.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:36.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:35 smithi190 ceph-mon[33748]: pgmap v606: 148 pgs: 18 active+clean+remapped, 92 active+clean, 26 active+undersized, 12 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 511 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:35 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 26/594 objects degraded (4.377%), 12 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:35 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:40:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:35 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:40:36.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:36.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:36.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:36.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:36.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:36.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:36.582 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:36.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:40:36.644 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:40:36.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:36.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:36.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:36.836 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:36.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:36.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:36.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:36.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:36.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:36.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:37.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:37.092 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:37.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:37.118 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:40:37.128 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:37 smithi029 ceph-mon[41500]: osdmap e273: 8 total, 7 up, 8 in 2024-03-24T12:40:37.129 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:37 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4119742939' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:40:37.129 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:37 smithi029 ceph-mon[29919]: osdmap e273: 8 total, 7 up, 8 in 2024-03-24T12:40:37.129 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:37 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4119742939' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:40:37.193 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:37.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:37.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:37.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T12:40:37.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:37.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:37.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:37 smithi190 ceph-mon[33748]: osdmap e273: 8 total, 7 up, 8 in 2024-03-24T12:40:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:37 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4119742939' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:40:37.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:37.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:37.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:37.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:37.604 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:37.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:37.705 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:37.734 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:37.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:37.836 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:37.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:37.864 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:37.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:37.909 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:40:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:40:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:40:37.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:37.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:37.989 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:37.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:38.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:38.119 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:38.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:38.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:38 smithi029 ceph-mon[41500]: pgmap v608: 148 pgs: 18 active+clean+remapped, 92 active+clean, 28 active+undersized, 10 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 182 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:38.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:38 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:40:38.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:38 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4119742939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:40:38.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:38 smithi029 ceph-mon[41500]: osdmap e274: 8 total, 7 up, 8 in 2024-03-24T12:40:38.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:38 smithi029 ceph-mon[29919]: pgmap v608: 148 pgs: 18 active+clean+remapped, 92 active+clean, 28 active+undersized, 10 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 182 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:38.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:38 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:40:38.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:38 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4119742939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:40:38.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:38 smithi029 ceph-mon[29919]: osdmap e274: 8 total, 7 up, 8 in 2024-03-24T12:40:38.220 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:38.249 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:38.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:38.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:38.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:38.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:38 smithi190 ceph-mon[33748]: pgmap v608: 148 pgs: 18 active+clean+remapped, 92 active+clean, 28 active+undersized, 10 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 182 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:38 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:40:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:38 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4119742939' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:40:38.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:38 smithi190 ceph-mon[33748]: osdmap e274: 8 total, 7 up, 8 in 2024-03-24T12:40:38.383 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:40:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:40:38.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:38.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:38.505 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:38.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:38.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:38.631 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:38.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:38.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:40:38.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:38.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:38.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:38.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:38.886 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:38.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:38.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:39.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:39.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:39.043 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:39.062 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:39.116 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:39.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:39.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:39.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:39.272 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:39.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:39.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:39.399 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:39.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:39.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:39.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:39.526 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:39.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:39.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:39.653 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:39.653 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:39.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:39.779 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:39.779 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:39.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:39.905 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:39.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:39.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:39.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:40:40.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:40.033 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:40.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'unique_pool_2' removed 2024-03-24T12:40:40.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:40.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:40 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:40.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:40 smithi029 ceph-mon[41500]: pgmap v610: 148 pgs: 18 active+clean+remapped, 92 active+clean, 30 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:40.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:40 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2316433044' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:40:40.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:40 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 26/594 objects degraded (4.377%), 8 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:40.136 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:40 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2316433044' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:40:40.162 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:40.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:40.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:40.290 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:40.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:40.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:40.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:40 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:40 smithi190 ceph-mon[33748]: pgmap v610: 148 pgs: 18 active+clean+remapped, 92 active+clean, 30 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:40 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2316433044' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:40:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:40 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 26/594 objects degraded (4.377%), 8 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:40.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:40 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2316433044' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:40:40.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:40 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:40 smithi029 ceph-mon[29919]: pgmap v610: 148 pgs: 18 active+clean+remapped, 92 active+clean, 30 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:40 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2316433044' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:40:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:40 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 26/594 objects degraded (4.377%), 8 pgs degraded, 38 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:40 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2316433044' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:40:40.418 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:40.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:40.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:40.519 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:40.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:40.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:40.645 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:40.668 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:40.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:40.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:40.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:40.795 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:40.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:40.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:40.896 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:40.920 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:40.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:40.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:41.022 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:41.045 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:41.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:41.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:41.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:41.172 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:41.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:41.256 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T12:40:41.257 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T12:40:41.257 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-24T12:40:41.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-24T12:40:41.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:41.296 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:41.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:41.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:41 smithi029 ceph-mon[41500]: osdmap e275: 8 total, 7 up, 8 in 2024-03-24T12:40:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:41 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:40:41.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:41 smithi029 ceph-mon[29919]: osdmap e275: 8 total, 7 up, 8 in 2024-03-24T12:40:41.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:41 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:40:41.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:41.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:41.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:41.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:41 smithi190 ceph-mon[33748]: osdmap e275: 8 total, 7 up, 8 in 2024-03-24T12:40:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:41 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:40:41.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:41.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:41.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:41.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:41.800 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:41.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:41.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:41.926 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:41.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:41.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:42.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:42.052 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:42.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:42.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:42.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:42.177 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:42.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:42.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:42.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:42.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:42.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:42.382 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:42 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:40:42.382 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:42 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:40:42.382 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:42 smithi029 ceph-mon[41500]: pgmap v612: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:42.382 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:42 smithi029 ceph-mon[41500]: osdmap e276: 8 total, 7 up, 8 in 2024-03-24T12:40:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:42 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:40:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:42 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:40:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:42 smithi029 ceph-mon[29919]: pgmap v612: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:42.383 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:42 smithi029 ceph-mon[29919]: osdmap e276: 8 total, 7 up, 8 in 2024-03-24T12:40:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:42 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:40:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:42 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:40:42.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:42 smithi190 ceph-mon[33748]: pgmap v612: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:42.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:42 smithi190 ceph-mon[33748]: osdmap e276: 8 total, 7 up, 8 in 2024-03-24T12:40:42.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:42.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:42.431 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:42.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:42.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:42.556 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:42.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:42.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:42.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:42.682 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:42.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:42.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:42.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:42.809 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:42.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:42.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:42.936 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:42.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:43.038 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:43.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:43.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:43.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:43.166 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:43.192 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:43.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:43.266 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:43.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:43.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:43.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:43.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:43.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:43.485 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:43.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:43.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:43.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:43.611 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:43.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:43.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:43.619 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:40:43.619 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:40:43.619 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:40:43.619 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:40:43.619 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:40:43.619 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:40:43.631 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:40:43.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:43.732 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:40:43.732 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:40:43.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:40:43.732 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:40:43.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.732 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:150960 7.18 7:1d604460:::benchmark_data_smithi029_84282_object50197:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e137)", 2024-03-24T12:40:43.732 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:08.190147+0000", 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 575.43219089199999, 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.076245219, 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 150960 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190147+0000", 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190147+0000", 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190149+0000", 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7570000000000001e-06 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190155+0000", 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1290000000000001e-06 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190167+0000", 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2293000000000001e-05 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.190176+0000", 2024-03-24T12:40:43.735 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3340000000000002e-06 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.196962+0000", 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0067861099999999997 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:08.196971+0000", 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9940000000000006e-06 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.258125+0000", 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0611542940000001 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for missing object", 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.258536+0000", 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041083200000000003 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.266309+0000", 2024-03-24T12:40:43.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0077733589999999997 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:09.266392+0000", 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3116999999999997e-05 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:163980 7.13 7:cef35ea1:::benchmark_data_smithi029_84282_object63217:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:40:43.738 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.071712+0000", 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 555.55062576499995, 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.14239947, 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 163980 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071712+0000", 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071712+0000", 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071714+0000", 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.669e-06 2024-03-24T12:40:43.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071720+0000", 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0719999999999996e-06 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071737+0000", 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6583999999999999e-05 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071747+0000", 2024-03-24T12:40:43.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0244999999999999e-05 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071844+0000", 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6825999999999998e-05 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071853+0000", 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3409999999999992e-06 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214027+0000", 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1421742610000001 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214112+0000", 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4472e-05 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.743 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:163984 7.13 7:cd9b75c3:::benchmark_data_smithi029_84282_object63221:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.071773+0000", 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 555.55056496999998, 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1428663079999999, 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 163984 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.744 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071773+0000", 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071773+0000", 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071774+0000", 2024-03-24T12:40:43.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0900000000000002e-07 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071781+0000", 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2450000000000001e-06 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071788+0000", 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6000000000000003e-06 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071793+0000", 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7049999999999996e-06 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071881+0000", 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7682999999999994e-05 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.071887+0000", 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7699999999999998e-06 2024-03-24T12:40:43.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214557+0000", 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1426706010000001 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214639+0000", 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2194999999999995e-05 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164027 7.13 7:cd84f892:::benchmark_data_smithi029_84282_object63264:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.107232+0000", 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 555.51510596699995, 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1125308970000001, 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164027 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107232+0000", 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107232+0000", 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107233+0000", 2024-03-24T12:40:43.750 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9199999999999999e-07 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107237+0000", 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3449999999999999e-06 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107248+0000", 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0278000000000001e-05 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107255+0000", 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8080000000000005e-06 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107313+0000", 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7259999999999997e-05 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.107321+0000", 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0020000000000006e-06 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214680+0000", 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.107358882 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.214776+0000", 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6278000000000002e-05 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:40:43.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.215116+0000", 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000340012 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219615+0000", 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0044993300000000002 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219629+0000", 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3993e-05 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219666+0000", 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7017000000000002e-05 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219716+0000", 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9487000000000002e-05 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.219763+0000", 2024-03-24T12:40:43.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7312999999999999e-05 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.14799.0:164078 7.13 7:ccf4d89e:::benchmark_data_smithi029_84282_object63315:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e143)", 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:31:28.157889+0000", 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 555.464449296, 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0645090349999999, 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.14799", 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4285448008", 2024-03-24T12:40:43.756 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 164078 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157889+0000", 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157889+0000", 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.757 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157890+0000", 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.387e-06 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157895+0000", 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4939999999999997e-06 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157908+0000", 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3327e-05 2024-03-24T12:40:43.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157921+0000", 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2719e-05 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157980+0000", 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9626999999999997e-05 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:28.157989+0000", 2024-03-24T12:40:43.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5120000000000008e-06 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.217363+0000", 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0593739980000001 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.217463+0000", 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010005600000000001 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,7", 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.217715+0000", 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025247600000000001 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222212+0000", 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0044967549999999998 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222265+0000", 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2377999999999999e-05 2024-03-24T12:40:43.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222303+0000", 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8455999999999999e-05 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222354+0000", 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1387000000000001e-05 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:31:29.222398+0000", 2024-03-24T12:40:43.763 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3463e-05 2024-03-24T12:40:43.763 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.763 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.763 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.763 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.763 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104312 9.b 9:d3e4c742:::benchmark_data_smithi029_134815_object104311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:40:43.763 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.099394+0000", 2024-03-24T12:40:43.763 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.522943973, 2024-03-24T12:40:43.763 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1656896990000001, 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104312 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099394+0000", 2024-03-24T12:40:43.764 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099394+0000", 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099395+0000", 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2249999999999999e-06 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099446+0000", 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0138e-05 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099455+0000", 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8849999999999994e-06 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099464+0000", 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7309999999999999e-06 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.766 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099570+0000", 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105568 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099579+0000", 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1190000000000003e-06 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073282+0000", 2024-03-24T12:40:43.767 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.97370271399999997 2024-03-24T12:40:43.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.768 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073293+0000", 2024-03-24T12:40:43.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1156e-05 2024-03-24T12:40:43.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.768 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.225494+0000", 2024-03-24T12:40:43.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1522010009999999 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.225625+0000", 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013125300000000001 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.226198+0000", 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00057334900000000002 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.241089+0000", 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.014890944 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250249+0000", 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0091596839999999995 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.264890+0000", 2024-03-24T12:40:43.770 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.014640604 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.264958+0000", 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8332999999999998e-05 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265084+0000", 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000125995 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.771 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104318 9.b 9:d43bf6ba:::benchmark_data_smithi029_134815_object104317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.109135+0000", 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.51320353400001, 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1563207370000002, 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.772 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104318 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109135+0000", 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109135+0000", 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109135+0000", 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4700000000000001e-07 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109181+0000", 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5513999999999998e-05 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.774 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109188+0000", 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.644e-06 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109195+0000", 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2400000000000004e-06 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.111000+0000", 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0018054429999999999 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.111011+0000", 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.134e-05 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073314+0000", 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.96230284899999996 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073321+0000", 2024-03-24T12:40:43.776 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.002e-06 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.241134+0000", 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.167813244 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.241273+0000", 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013887400000000001 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.241450+0000", 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000176226 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250310+0000", 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0088607200000000007 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257152+0000", 2024-03-24T12:40:43.778 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.006841387 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265168+0000", 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0080164239999999994 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265349+0000", 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018142200000000001 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265455+0000", 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010586099999999999 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104334 9.b 9:df2efe4b:::benchmark_data_smithi029_134815_object104333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.128443+0000", 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.49389493699999, 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1374394969999999, 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.780 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104334 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128443+0000", 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128443+0000", 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128445+0000", 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.516e-06 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.782 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128495+0000", 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9907999999999997e-05 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128504+0000", 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3510000000000008e-06 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128511+0000", 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3889999999999999e-06 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128580+0000", 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9028000000000004e-05 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128591+0000", 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0794e-05 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073342+0000", 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94475095399999998 2024-03-24T12:40:43.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073349+0000", 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5220000000000004e-06 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.244214+0000", 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.170865115 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.244346+0000", 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000131777 2024-03-24T12:40:43.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.244514+0000", 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016814999999999999 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250351+0000", 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0058376089999999997 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.786 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257198+0000", 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0068467659999999998 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265468+0000", 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0082702160000000004 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265777+0000", 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030915599999999999 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265883+0000", 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105246 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104336 9.b 9:d1900f74:::benchmark_data_smithi029_134815_object104335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.129851+0000", 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.492487141, 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1363790969999998, 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.788 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104336 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129851+0000", 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.789 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.790 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129851+0000", 2024-03-24T12:40:43.790 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.790 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.790 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.790 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129852+0000", 2024-03-24T12:40:43.790 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.009e-06 2024-03-24T12:40:43.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.790 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.790 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129902+0000", 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9784000000000003e-05 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129922+0000", 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0543e-05 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129936+0000", 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3386e-05 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.130013+0000", 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7442999999999997e-05 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.130023+0000", 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8519999999999999e-06 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073367+0000", 2024-03-24T12:40:43.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94334440600000002 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073374+0000", 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3500000000000002e-06 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.247267+0000", 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.17389327 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.793 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.247413+0000", 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000146064 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.247597+0000", 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183914 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250393+0000", 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027959180000000001 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.794 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257238+0000", 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0068453170000000001 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265896+0000", 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086582129999999997 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.266117+0000", 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00022031700000000001 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.266230+0000", 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113311 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104365 9.b 9:d74e8c62:::benchmark_data_smithi029_134815_object104364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.178308+0000", 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.44403014299999, 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.089454752, 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.796 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104365 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178308+0000", 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178308+0000", 2024-03-24T12:40:43.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.798 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178310+0000", 2024-03-24T12:40:43.798 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.795e-06 2024-03-24T12:40:43.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.798 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178334+0000", 2024-03-24T12:40:43.798 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3892000000000001e-05 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178355+0000", 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1787e-05 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178366+0000", 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0705e-05 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178440+0000", 2024-03-24T12:40:43.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4006999999999998e-05 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178452+0000", 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1494999999999999e-05 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073397+0000", 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.89494537900000004 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073404+0000", 2024-03-24T12:40:43.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5069999999999997e-06 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250424+0000", 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1770203910000001 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250552+0000", 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012809099999999999 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250738+0000", 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018606100000000001 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257280+0000", 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0065419579999999996 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257325+0000", 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5188999999999997e-05 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.266239+0000", 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0089141340000000006 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.267645+0000", 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0014058460000000001 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.267763+0000", 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000117515 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.803 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104372 9.b 9:d40eb526:::benchmark_data_smithi029_134815_object104371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.195928+0000", 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 153.42640988599999, 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0720185619999998, 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104372 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195928+0000", 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195928+0000", 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195929+0000", 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2359999999999999e-06 2024-03-24T12:40:43.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195950+0000", 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0792e-05 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195962+0000", 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1226999999999999e-05 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195970+0000", 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7239999999999998e-06 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.196035+0000", 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4597999999999994e-05 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.196047+0000", 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1941e-05 2024-03-24T12:40:43.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073425+0000", 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.87737854400000004 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073432+0000", 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6429999999999998e-06 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.253708+0000", 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1802758440000001 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.253849+0000", 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014084199999999999 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.254020+0000", 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000171261 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257368+0000", 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0033482170000000001 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257415+0000", 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6925000000000002e-05 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.267778+0000", 2024-03-24T12:40:43.810 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010363140999999999 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.267833+0000", 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4313e-05 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.267947+0000", 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000114314 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.811 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172139 9.b 9:dc246dd6:::benchmark_data_smithi029_134815_object60073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.674435+0000", 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.947903503000006, 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4605824350000001, 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172139 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.812 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674435+0000", 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674435+0000", 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674436+0000", 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5900000000000005e-07 2024-03-24T12:40:43.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674440+0000", 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0860000000000004e-06 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674453+0000", 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3131e-05 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674461+0000", 2024-03-24T12:40:43.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4849999999999992e-06 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680515+0000", 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060537869999999997 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680647+0000", 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000131624 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.681361+0000", 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00071422300000000005 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.691102+0000", 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0097411960000000006 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.691182+0000", 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0073000000000004e-05 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745633+0000", 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.054451039999999999 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745655+0000", 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1627000000000001e-05 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750438+0000", 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0047831860000001 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750458+0000", 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0426000000000001e-05 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.134965+0000", 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.384506811 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.135017+0000", 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1780999999999998e-05 2024-03-24T12:40:43.818 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172143 9.b 9:d403d143:::benchmark_data_smithi029_134815_object60077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.680416+0000", 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.941921863000005, 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4554223509999999, 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.819 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172143 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680416+0000", 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680416+0000", 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680417+0000", 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0240000000000001e-06 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680421+0000", 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9299999999999996e-06 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680453+0000", 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1365999999999998e-05 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680463+0000", 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0066e-05 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.684801+0000", 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.004338639 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.684935+0000", 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013339999999999999 2024-03-24T12:40:43.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.685063+0000", 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000128718 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.691380+0000", 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0063161479999999997 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.691418+0000", 2024-03-24T12:40:43.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8167999999999998e-05 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745728+0000", 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.054309907999999997 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745737+0000", 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6549999999999999e-06 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750567+0000", 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.004829897 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750579+0000", 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1317e-05 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.135741+0000", 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38516228699999999 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.135839+0000", 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7827999999999999e-05 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172149 9.b 9:d2e9c91b:::benchmark_data_smithi029_134815_object60083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.682707+0000", 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.939630851000004, 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.453752683, 2024-03-24T12:40:43.826 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172149 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682707+0000", 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.827 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682707+0000", 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682708+0000", 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8500000000000002e-07 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682710+0000", 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4210000000000002e-06 2024-03-24T12:40:43.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682716+0000", 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8699999999999997e-06 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682721+0000", 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0830000000000003e-06 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.692301+0000", 2024-03-24T12:40:43.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.009580224 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.692458+0000", 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015688200000000001 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.692594+0000", 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013530700000000001 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.695140+0000", 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002546338 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.695394+0000", 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025376799999999997 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745799+0000", 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.050405028999999997 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745824+0000", 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4890000000000001e-05 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750612+0000", 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0047884199999999 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750621+0000", 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3330000000000003e-06 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136400+0000", 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38577870399999997 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136460+0000", 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9929000000000002e-05 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172172 9.b 9:d5266a5d:::benchmark_data_smithi029_134815_object60106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.697248+0000", 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.925090267000002, 2024-03-24T12:40:43.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4392927250000001, 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172172 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697248+0000", 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.834 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697248+0000", 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697249+0000", 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.46e-06 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697254+0000", 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0259999999999998e-06 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697270+0000", 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5804e-05 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697278+0000", 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1620000000000008e-06 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697424+0000", 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014525700000000001 2024-03-24T12:40:43.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697569+0000", 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014558799999999999 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697705+0000", 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013550699999999999 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.700575+0000", 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028701170000000002 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.701307+0000", 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073196299999999997 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745878+0000", 2024-03-24T12:40:43.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.044571350000000003 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745889+0000", 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0577999999999999e-05 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750652+0000", 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.004763383 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750661+0000", 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1179999999999998e-06 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136490+0000", 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38582929500000002 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136541+0000", 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0117000000000001e-05 2024-03-24T12:40:43.840 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172194 9.b 9:d9b9c02c:::benchmark_data_smithi029_134815_object60128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.718569+0000", 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.903768701000004, 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4180509909999999, 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.841 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172194 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718569+0000", 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718569+0000", 2024-03-24T12:40:43.842 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718570+0000", 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9800000000000002e-07 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718575+0000", 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8670000000000002e-06 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718586+0000", 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0492000000000001e-05 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718592+0000", 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.711e-06 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718636+0000", 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4610999999999997e-05 2024-03-24T12:40:43.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718740+0000", 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010396899999999999 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718811+0000", 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0983999999999997e-05 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.721173+0000", 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0023618150000000002 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.721297+0000", 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000124263 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745952+0000", 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.024654850999999998 2024-03-24T12:40:43.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745963+0000", 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0806e-05 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750691+0000", 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.004728445 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750700+0000", 2024-03-24T12:40:43.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0659999999999993e-06 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136572+0000", 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38587177099999997 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136620+0000", 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8341999999999999e-05 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172198 9.b 9:d32e9228:::benchmark_data_smithi029_134815_object60132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.725029+0000", 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.897308938999998, 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.41167083, 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172198 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725029+0000", 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725029+0000", 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725030+0000", 2024-03-24T12:40:43.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.850000000000001e-07 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725035+0000", 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.459e-06 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725045+0000", 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0522e-05 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725053+0000", 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9270000000000005e-06 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725103+0000", 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0380999999999998e-05 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725193+0000", 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9495999999999998e-05 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725270+0000", 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7470000000000002e-05 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.727388+0000", 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002118048 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.727403+0000", 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4909000000000001e-05 2024-03-24T12:40:43.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.746084+0000", 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.018681039999999999 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.746095+0000", 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0424999999999999e-05 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750731+0000", 2024-03-24T12:40:43.854 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.004635827 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750740+0000", 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0299999999999999e-06 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136651+0000", 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38591176799999999 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.855 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136700+0000", 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8542999999999999e-05 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172205 9.b 9:d50f1c31:::benchmark_data_smithi029_134815_object60139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.794173+0000", 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.828164845000003, 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3523102300000001, 2024-03-24T12:40:43.856 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172205 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794173+0000", 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794173+0000", 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794175+0000", 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2359999999999999e-06 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794179+0000", 2024-03-24T12:40:43.858 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6829999999999999e-06 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794190+0000", 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.046e-05 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794198+0000", 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5459999999999992e-06 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.859 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794253+0000", 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5228e-05 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794263+0000", 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0016e-05 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750765+0000", 2024-03-24T12:40:43.860 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.95650199599999997 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750773+0000", 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9689999999999994e-06 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136726+0000", 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38595213699999997 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.861 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136829+0000", 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010315900000000001 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136948+0000", 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000119073 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146237+0000", 2024-03-24T12:40:43.862 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0092890230000000004 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146250+0000", 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3419e-05 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146339+0000", 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8263999999999994e-05 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.863 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.864 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146386+0000", 2024-03-24T12:40:43.864 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7568000000000003e-05 2024-03-24T12:40:43.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.864 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.864 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.864 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146484+0000", 2024-03-24T12:40:43.864 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7453000000000004e-05 2024-03-24T12:40:43.864 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.864 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.864 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172221 9.b 9:df06907a:::benchmark_data_smithi029_134815_object60155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.872329+0000", 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.750009546000001, 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2750098759999999, 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172221 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.865 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872329+0000", 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872329+0000", 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.866 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872330+0000", 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.807e-06 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872338+0000", 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5460000000000003e-06 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872355+0000", 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6626999999999999e-05 2024-03-24T12:40:43.867 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872365+0000", 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0359e-05 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872441+0000", 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6285000000000006e-05 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.868 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872454+0000", 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2812e-05 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750796+0000", 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.87834196399999998 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750804+0000", 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8630000000000008e-06 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.869 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.139760+0000", 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38895642000000002 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.139863+0000", 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010245 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.870 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.139975+0000", 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000112643 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146256+0000", 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0062804260000000004 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146292+0000", 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6346000000000001e-05 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.871 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147145+0000", 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00085249000000000002 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147213+0000", 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8195999999999996e-05 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147338+0000", 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000125642 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.872 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172223 9.b 9:dc9df42b:::benchmark_data_smithi029_134815_object60157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.874452+0000", 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 69.747885830000001, 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.273117195, 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:40:43.873 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172223 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874452+0000", 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874452+0000", 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874454+0000", 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4699999999999999e-06 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874461+0000", 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8129999999999998e-06 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874479+0000", 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8745000000000001e-05 2024-03-24T12:40:43.875 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874490+0000", 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0209e-05 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874624+0000", 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013484099999999999 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874638+0000", 2024-03-24T12:40:43.876 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3506000000000001e-05 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.751235+0000", 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.87659726599999999 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.751246+0000", 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0856000000000001e-05 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.877 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.142969+0000", 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.39172287300000003 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.143104+0000", 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013542599999999999 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.143213+0000", 2024-03-24T12:40:43.878 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000108628 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146305+0000", 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0030918769999999998 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146613+0000", 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030804699999999999 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.879 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147399+0000", 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00078603199999999999 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147459+0000", 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9757999999999999e-05 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147570+0000", 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011084800000000001 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.880 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.881 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.881 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:40:43.881 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:40:43.881 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:40:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:43.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:43.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:43.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:44.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:44.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:44.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:44.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:44.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:44.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:44.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:44.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:44.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:44.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:44.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:44.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:44 smithi029 ceph-mon[29919]: pgmap v614: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:44.316 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:40:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:40:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:40:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:44 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3778698718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:44.317 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:44 smithi029 ceph-mon[41500]: pgmap v614: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:44.317 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:40:44.317 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:40:44.317 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:40:44.317 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:44 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3778698718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:44.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:44.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:44.394 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:44.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:44.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:44.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:44.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:44.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:44.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:44.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:44 smithi190 ceph-mon[33748]: pgmap v614: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:44.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:40:44.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:40:44.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:40:44.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:44 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3778698718' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:44.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:44.649 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:44.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:44.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:44.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:44.775 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:44.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:44.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:44.902 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:44.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:45.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:45.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:45.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:45.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:45.161 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:45.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:45.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:45.286 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:45.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:45.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:45.373 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:45 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 26/594 objects degraded (4.377%), 8 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:45.374 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:45 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 26/594 objects degraded (4.377%), 8 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:45.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:45.416 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:45.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:45.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:45.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:45.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:45.543 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:45.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:45.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:45 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 26/594 objects degraded (4.377%), 8 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:40:45.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:45.668 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:45.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:45.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:45.679 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T12:40:45.683 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:45.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:45.798 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:45.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:45.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:45.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:45.927 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:45.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:46.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:46.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:46.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:46.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:46.124 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:40:46.124 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:40:46.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:46.182 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:46.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:46.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:46.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:46.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:46.309 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:46.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:46.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:46.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:46 smithi029 ceph-mon[41500]: pgmap v615: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:46.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:46 smithi029 ceph-mon[29919]: pgmap v615: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:46.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:46.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:46.559 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:46.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:46.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:46.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:46 smithi190 ceph-mon[33748]: pgmap v615: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:46.661 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:46.689 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:46.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:46.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:46.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:46.817 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:46.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:46.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:46.942 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:46.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:46.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:47.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:47.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:47.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:47.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:47.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:47.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:47.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:47.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:47.324 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:47.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:47.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:47.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:47 smithi029 ceph-mon[29919]: pgmap v616: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:47.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:47 smithi029 ceph-mon[41500]: pgmap v616: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:47.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:47.449 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:47.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:47.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:47.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:47.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:47.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:47.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:47.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:47 smithi190 ceph-mon[33748]: pgmap v616: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:47.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:47.708 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:47.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:47.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:47.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:47.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:47.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:47.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:47.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:40:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:40:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:40:47.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:47.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:47.963 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:47.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:48.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:48.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:48.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:48.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:40:48.154 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:40:48.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:48.222 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:48.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:48.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:48.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:48.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:48.351 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:48.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:48.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:48.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:48 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4181657693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:40:48.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:48.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:48 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4181657693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:40:48.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:48.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:48.476 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:48.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:48.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:48.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:48.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:48.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:40:48.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:48 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4181657693' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:40:48.633 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:40:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:40:48.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:48.731 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:48.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:48.833 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:48.858 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:48.859 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:48.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:48.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:48.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:48.984 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:48.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:48.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:49.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:49.115 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:49.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:49.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:49.242 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:49.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:49.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:49.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:49.368 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:49.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:49.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:49.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:40:49.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4181657693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:40:49.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[41500]: osdmap e277: 8 total, 7 up, 8 in 2024-03-24T12:40:49.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:49.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[41500]: 5.a scrub starts 2024-03-24T12:40:49.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[41500]: 5.a scrub ok 2024-03-24T12:40:49.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[41500]: pgmap v618: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:49.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:40:49.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4181657693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:40:49.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[29919]: osdmap e277: 8 total, 7 up, 8 in 2024-03-24T12:40:49.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:49.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[29919]: 5.a scrub starts 2024-03-24T12:40:49.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[29919]: 5.a scrub ok 2024-03-24T12:40:49.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:49 smithi029 ceph-mon[29919]: pgmap v618: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:49.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:49.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:49.495 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:49.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:49.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:49.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:49.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:49 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:40:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:49 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4181657693' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:40:49.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:49 smithi190 ceph-mon[33748]: osdmap e277: 8 total, 7 up, 8 in 2024-03-24T12:40:49.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:49.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:49 smithi190 ceph-mon[33748]: 5.a scrub starts 2024-03-24T12:40:49.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:49 smithi190 ceph-mon[33748]: 5.a scrub ok 2024-03-24T12:40:49.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:49 smithi190 ceph-mon[33748]: pgmap v618: 132 pgs: 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:49.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:49.752 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:49.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:49.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:49.877 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:49.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:49.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:50.002 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:50.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:50.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:50.117 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'unique_pool_3' created 2024-03-24T12:40:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:50.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:50.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:50.267 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:50.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:50.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:50.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:50.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:50.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:50 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2250478416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T12:40:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:50 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2250478416' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-24T12:40:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:50 smithi029 ceph-mon[29919]: osdmap e278: 8 total, 7 up, 8 in 2024-03-24T12:40:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:50 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2250478416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T12:40:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:50 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2250478416' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-24T12:40:50.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:50 smithi029 ceph-mon[41500]: osdmap e278: 8 total, 7 up, 8 in 2024-03-24T12:40:50.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:50.523 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:50.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:50.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:50 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2250478416' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-24T12:40:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:50 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2250478416' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-24T12:40:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:50 smithi190 ceph-mon[33748]: osdmap e278: 8 total, 7 up, 8 in 2024-03-24T12:40:50.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:50.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:50.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:50.778 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:50.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:50.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:50.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:40:50.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:50.905 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:50.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:51.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:51.032 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:51.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:51.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:51.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:51.157 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:51.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:51.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:51.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:51.287 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:51.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:51.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:51.417 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:51.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:51.418 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:51.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-24T12:40:51.519 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:51.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:51.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:51.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:51.673 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:51.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:51.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:51.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:51.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:51.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:51.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:51.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:51.988 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:51.993 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:40:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:40:51.993 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-03-24T12:40:51.993 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:40:52.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:52.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:52.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:52.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:52.176 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:52.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:52.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:52.251 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:52 smithi029 ceph-mon[29919]: pgmap v620: 148 pgs: 16 unknown, 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:52.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:52 smithi029 ceph-mon[29919]: osdmap e279: 8 total, 7 up, 8 in 2024-03-24T12:40:52.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:52 smithi029 ceph-mon[29919]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:40:52.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:52 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:40:52.252 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:52 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:40:52.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:52 smithi029 ceph-mon[41500]: pgmap v620: 148 pgs: 16 unknown, 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:52.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:52 smithi029 ceph-mon[41500]: osdmap e279: 8 total, 7 up, 8 in 2024-03-24T12:40:52.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:52 smithi029 ceph-mon[41500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:40:52.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:52 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:40:52.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:52 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:40:52.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:52.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:52.302 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:52.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:52.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:52 smithi190 ceph-mon[33748]: pgmap v620: 148 pgs: 16 unknown, 17 active+clean+remapped, 85 active+clean, 22 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:52.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:52 smithi190 ceph-mon[33748]: osdmap e279: 8 total, 7 up, 8 in 2024-03-24T12:40:52.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:52 smithi190 ceph-mon[33748]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:40:52.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:52 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:40:52.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:52 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:40:52.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:52.426 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:52.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:52.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:52.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:52.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:52.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:52.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:52.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:52.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:52.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:52.811 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:52.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:52.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:52.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T12:40:52.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:52.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:52.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:53.040 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:53.066 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:53.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:53.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:53.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:53.194 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:53.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:53.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:53.321 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:53.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:53.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:53.374 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:53.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:53 smithi029 ceph-mon[29919]: osdmap e280: 8 total, 7 up, 8 in 2024-03-24T12:40:53.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:53 smithi029 ceph-mon[41500]: osdmap e280: 8 total, 7 up, 8 in 2024-03-24T12:40:53.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:53.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:53.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:53.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:53.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:53.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:53.579 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:53.579 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:53 smithi190 ceph-mon[33748]: osdmap e280: 8 total, 7 up, 8 in 2024-03-24T12:40:53.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:53.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:53.705 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:53.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:53.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:53.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:53.831 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:53.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:53.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:53.962 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:53.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:54.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:54.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:54.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:54.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:54.217 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:54.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:54.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:54.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:54.344 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:54.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:54.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:54 smithi029 ceph-mon[29919]: pgmap v623: 148 pgs: 1 creating+activating, 4 unknown, 17 active+clean+remapped, 93 active+clean, 25 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:54.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:54 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:40:54.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:54 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:40:54.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:54 smithi029 ceph-mon[29919]: 3.12 scrub starts 2024-03-24T12:40:54.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:54 smithi029 ceph-mon[29919]: 3.12 scrub ok 2024-03-24T12:40:54.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:54 smithi029 ceph-mon[41500]: pgmap v623: 148 pgs: 1 creating+activating, 4 unknown, 17 active+clean+remapped, 93 active+clean, 25 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:54.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:54 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:40:54.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:54 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:40:54.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:54 smithi029 ceph-mon[41500]: 3.12 scrub starts 2024-03-24T12:40:54.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:54 smithi029 ceph-mon[41500]: 3.12 scrub ok 2024-03-24T12:40:54.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:54.469 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:54.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:54.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:54.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:54.594 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:54.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:54.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:54 smithi190 ceph-mon[33748]: pgmap v623: 148 pgs: 1 creating+activating, 4 unknown, 17 active+clean+remapped, 93 active+clean, 25 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:54.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:54 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:40:54.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:54 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:40:54.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:54 smithi190 ceph-mon[33748]: 3.12 scrub starts 2024-03-24T12:40:54.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:54 smithi190 ceph-mon[33748]: 3.12 scrub ok 2024-03-24T12:40:54.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:54.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:54.722 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:54.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:54.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:54.846 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:54.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:54.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:54.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:54.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:55.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:55.102 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:55.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:55.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:55.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:55.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:55.228 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:55.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:55.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:55.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:55.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:55.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:55.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:55 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:40:55.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:55 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:40:55.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:55 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:40:55.409 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:55 smithi029 ceph-mon[41500]: 4.11 scrub ok 2024-03-24T12:40:55.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:55 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:40:55.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:55 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:40:55.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:55 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:40:55.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:55 smithi029 ceph-mon[29919]: 4.11 scrub ok 2024-03-24T12:40:55.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:55.486 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:55.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:55.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:55.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:55.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:55.612 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:55.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:55 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:40:55.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:55 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:40:55.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:55 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:40:55.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:55 smithi190 ceph-mon[33748]: 4.11 scrub ok 2024-03-24T12:40:55.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:55.737 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:55.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:55.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:55.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:55.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:55.863 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:55.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:55.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:55.989 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:55.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:55.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:56.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:56.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:56.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:56.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:56 smithi029 ceph-mon[29919]: pgmap v624: 148 pgs: 1 creating+activating, 1 unknown, 17 active+clean+remapped, 94 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:56 smithi029 ceph-mon[29919]: 6.12 scrub starts 2024-03-24T12:40:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:56 smithi029 ceph-mon[29919]: 6.12 scrub ok 2024-03-24T12:40:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:56 smithi029 ceph-mon[41500]: pgmap v624: 148 pgs: 1 creating+activating, 1 unknown, 17 active+clean+remapped, 94 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:56 smithi029 ceph-mon[41500]: 6.12 scrub starts 2024-03-24T12:40:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:56 smithi029 ceph-mon[41500]: 6.12 scrub ok 2024-03-24T12:40:56.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:56.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:56.246 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:56.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:56.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:56.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:56.371 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:56.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:56.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:56.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:56.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:56.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:56.622 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:56.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:56 smithi190 ceph-mon[33748]: pgmap v624: 148 pgs: 1 creating+activating, 1 unknown, 17 active+clean+remapped, 94 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:56.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:56 smithi190 ceph-mon[33748]: 6.12 scrub starts 2024-03-24T12:40:56.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:56 smithi190 ceph-mon[33748]: 6.12 scrub ok 2024-03-24T12:40:56.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:56.748 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:56.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:56.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:56.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:56.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:56.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:56.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:57.004 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:57.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:57.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:57.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:57.129 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:57.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:57.152 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:40:57.156 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:40:57.156 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:40:57.156 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:40:57.156 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:40:57.157 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:40:57.163 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:40:57.181 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:40:57.202 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:40:57.202 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:40:57.202 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:40:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:40:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:40:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:40:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:40:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:40:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:40:57.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:57.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:57.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:57.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:57.386 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:57.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:57.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[29919]: 3.9 scrub starts 2024-03-24T12:40:57.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[29919]: 3.9 scrub ok 2024-03-24T12:40:57.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:40:57.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:40:57.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[29919]: pgmap v625: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 775 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:57.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:40:57.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[29919]: 3.13 scrub ok 2024-03-24T12:40:57.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3814098288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:57.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1388590783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:40:57.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[41500]: 3.9 scrub starts 2024-03-24T12:40:57.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[41500]: 3.9 scrub ok 2024-03-24T12:40:57.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:40:57.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:40:57.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[41500]: pgmap v625: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 775 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:57.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:40:57.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[41500]: 3.13 scrub ok 2024-03-24T12:40:57.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3814098288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:57.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1388590783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:40:57.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:57.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:57.511 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:57.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:57.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:57.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:57 smithi190 ceph-mon[33748]: 3.9 scrub starts 2024-03-24T12:40:57.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:57 smithi190 ceph-mon[33748]: 3.9 scrub ok 2024-03-24T12:40:57.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:57 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:40:57.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:57 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:40:57.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:57 smithi190 ceph-mon[33748]: pgmap v625: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 775 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:57.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:57 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:40:57.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:57 smithi190 ceph-mon[33748]: 3.13 scrub ok 2024-03-24T12:40:57.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:57 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3814098288' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:40:57.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:57 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1388590783' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:40:57.636 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:57.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:57.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:57.737 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:57.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:57.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:57.767 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:57.808 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:40:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:40:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:40:57.869 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:57.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:57.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:58.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:58.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:58.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:58.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:58.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:58.153 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:58.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:58.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:58.284 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:58.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:58.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:58.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:58.411 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:58.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:58.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:58 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:40:58.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:58 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1388590783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:40:58.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:58 smithi029 ceph-mon[29919]: osdmap e281: 8 total, 7 up, 8 in 2024-03-24T12:40:58.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:58 smithi029 ceph-mon[29919]: 4.19 scrub starts 2024-03-24T12:40:58.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:58 smithi029 ceph-mon[29919]: 4.19 scrub ok 2024-03-24T12:40:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:58 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:40:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:58 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1388590783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:40:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:58 smithi029 ceph-mon[41500]: osdmap e281: 8 total, 7 up, 8 in 2024-03-24T12:40:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:58 smithi029 ceph-mon[41500]: 4.19 scrub starts 2024-03-24T12:40:58.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:58 smithi029 ceph-mon[41500]: 4.19 scrub ok 2024-03-24T12:40:58.512 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:58.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:58 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:40:58.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:58 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1388590783' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:40:58.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:58 smithi190 ceph-mon[33748]: osdmap e281: 8 total, 7 up, 8 in 2024-03-24T12:40:58.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:58 smithi190 ceph-mon[33748]: 4.19 scrub starts 2024-03-24T12:40:58.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:58 smithi190 ceph-mon[33748]: 4.19 scrub ok 2024-03-24T12:40:58.534 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:40:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:40:58.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:58.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:58.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:58.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:40:58.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:58.667 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:58.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:40:58.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:58.796 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:58.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:58.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:58.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:58.923 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:58.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:59.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:40:59.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:59.048 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:59.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:40:59.149 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:59.177 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:59.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:59.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:59.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:40:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:59.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:40:59.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:40:59.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[29919]: 5.1 scrub starts 2024-03-24T12:40:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[29919]: 5.1 scrub ok 2024-03-24T12:40:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[29919]: 3.1b scrub starts 2024-03-24T12:40:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[29919]: 3.1b scrub ok 2024-03-24T12:40:59.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[29919]: pgmap v627: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[41500]: 5.1 scrub starts 2024-03-24T12:40:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[41500]: 5.1 scrub ok 2024-03-24T12:40:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[41500]: 3.1b scrub starts 2024-03-24T12:40:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[41500]: 3.1b scrub ok 2024-03-24T12:40:59.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:40:59 smithi029 ceph-mon[41500]: pgmap v627: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:59.437 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:59.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:59.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:40:59.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:40:59.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:59.563 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:40:59.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:40:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:59 smithi190 ceph-mon[33748]: 5.1 scrub starts 2024-03-24T12:40:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:59 smithi190 ceph-mon[33748]: 5.1 scrub ok 2024-03-24T12:40:59.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:40:59.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:59 smithi190 ceph-mon[33748]: 3.1b scrub starts 2024-03-24T12:40:59.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:59 smithi190 ceph-mon[33748]: 3.1b scrub ok 2024-03-24T12:40:59.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:40:59 smithi190 ceph-mon[33748]: pgmap v627: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:40:59.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:40:59.692 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:59.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:59.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:40:59.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:59.824 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:59.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:40:59.885 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:40:59.886 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:40:59.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:40:59.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:40:59.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:40:59.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:00.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:00.078 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:00.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:00.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:00.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:00.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:00.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:00.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:00.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:00.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:00.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:00 smithi029 ceph-mon[29919]: 6.3 scrub starts 2024-03-24T12:41:00.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:00 smithi029 ceph-mon[29919]: 6.3 scrub ok 2024-03-24T12:41:00.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:00 smithi029 ceph-mon[29919]: osdmap e282: 8 total, 7 up, 8 in 2024-03-24T12:41:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:00 smithi029 ceph-mon[41500]: 6.3 scrub starts 2024-03-24T12:41:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:00 smithi029 ceph-mon[41500]: 6.3 scrub ok 2024-03-24T12:41:00.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:00 smithi029 ceph-mon[41500]: osdmap e282: 8 total, 7 up, 8 in 2024-03-24T12:41:00.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:00.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:00.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:00.591 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:00.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:00 smithi190 ceph-mon[33748]: 6.3 scrub starts 2024-03-24T12:41:00.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:00 smithi190 ceph-mon[33748]: 6.3 scrub ok 2024-03-24T12:41:00.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:00 smithi190 ceph-mon[33748]: osdmap e282: 8 total, 7 up, 8 in 2024-03-24T12:41:00.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:00.719 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:00.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:00.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:00.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:00.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:00.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:00.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:00.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:01.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:01.099 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:01.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:01.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:01.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:01.232 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:01.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:01.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:41:01.286 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:01 smithi029 ceph-mon[41500]: pgmap v629: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:41:01.286 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:01 smithi029 ceph-mon[29919]: pgmap v629: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:41:01.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:01.356 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:01.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:01.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:01.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:01.481 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:01.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:01.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:01.578 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:41:01.579 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:41:01.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:01.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:01.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:01.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:01 smithi190 ceph-mon[33748]: pgmap v629: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:41:01.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:01.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:01.739 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:01.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:01.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:01.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:01.867 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:01.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:01.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:41:01.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:01.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:01.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:02.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:02.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:02.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:02.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:02.169 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-24T12:41:02.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:02.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:02.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:02.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:02.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:02 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3907062050' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:41:02.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:02 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:41:02.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:02 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:41:02.372 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:02 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:41:02.373 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:02 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3907062050' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:41:02.373 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:02 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:41:02.373 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:02 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:41:02.373 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:02 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:41:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:02.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:02.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:02.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T12:41:02.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:02.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:02.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:02.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:02 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3907062050' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:41:02.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:02 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:41:02.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:02 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:41:02.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:02 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:41:02.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:02.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:02.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:02.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:02.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:02.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:02.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:02.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:02.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:02.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:03.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:03.030 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:03.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:03.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:03.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:03.158 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:03.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:03.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:03.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:03.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:03.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:41:03.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[29919]: osdmap e283: 8 total, 7 up, 8 in 2024-03-24T12:41:03.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:41:03.304 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[29919]: pgmap v631: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[29919]: 5.13 scrub starts 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[29919]: 5.13 scrub ok 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[41500]: osdmap e283: 8 total, 7 up, 8 in 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[41500]: pgmap v631: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[41500]: 5.13 scrub starts 2024-03-24T12:41:03.305 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[41500]: 5.13 scrub ok 2024-03-24T12:41:03.306 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:41:03.306 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:03.385 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:03.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:03.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:03.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:03.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:03.540 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:03.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:03 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:41:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:03 smithi190 ceph-mon[33748]: osdmap e283: 8 total, 7 up, 8 in 2024-03-24T12:41:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:03 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:41:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:03 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:41:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:03 smithi190 ceph-mon[33748]: pgmap v631: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:41:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:03 smithi190 ceph-mon[33748]: 5.13 scrub starts 2024-03-24T12:41:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:03 smithi190 ceph-mon[33748]: 5.13 scrub ok 2024-03-24T12:41:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:41:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:03.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:03.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:03.667 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:03.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:03.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:03.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:03.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:03.858 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:03.858 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:03.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:03.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:03.984 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:03.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:04.051 DEBUG:teuthology.orchestra.run.smithi029:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-24T12:41:04.052 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-03-24T12:41:04.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:41:04.055 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T12:41:04.055 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T12:41:04.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:04.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:04.115 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:04.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:04.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:04.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:04.243 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:04.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:04.274 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:hints = 1 2024-03-24T12:41:04.274 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T12:41:04.274 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Object prefix: benchmark_data_smithi029_160664 2024-03-24T12:41:04.275 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:41:04.275 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 0 0 0 0 0 0 - 0 2024-03-24T12:41:04.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:04.379 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:04.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:04.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:04 smithi029 ceph-mon[29919]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:41:04.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:04 smithi029 ceph-mon[29919]: osdmap e284: 8 total, 7 up, 8 in 2024-03-24T12:41:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:04 smithi029 ceph-mon[41500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:41:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:04 smithi029 ceph-mon[41500]: osdmap e284: 8 total, 7 up, 8 in 2024-03-24T12:41:04.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:04.523 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:04.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:04.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:04.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:04 smithi190 ceph-mon[33748]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:41:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:04 smithi190 ceph-mon[33748]: osdmap e284: 8 total, 7 up, 8 in 2024-03-24T12:41:04.669 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:04.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:04.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:04.814 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:04.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:04.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:04.916 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:04.950 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:04.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:05.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:05.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:05.095 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:05.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:05.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:05.229 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:05.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:05.275 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 1 16 3673 3657 228.55 228.562 0.00212983 0.00436178 2024-03-24T12:41:05.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:05.356 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:05.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:05.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:05 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:41:05.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:05 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:41:05.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:05 smithi029 ceph-mon[41500]: 5.3 scrub starts 2024-03-24T12:41:05.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:05 smithi029 ceph-mon[41500]: 5.3 scrub ok 2024-03-24T12:41:05.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:05 smithi029 ceph-mon[41500]: pgmap v633: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:41:05.396 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:05 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:41:05.396 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:05 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:41:05.396 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:05 smithi029 ceph-mon[29919]: 5.3 scrub starts 2024-03-24T12:41:05.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:05 smithi029 ceph-mon[29919]: 5.3 scrub ok 2024-03-24T12:41:05.397 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:05 smithi029 ceph-mon[29919]: pgmap v633: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:41:05.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:05.495 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:05.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:05.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:05.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:05.631 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:05.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:05.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:05 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:41:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:05 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:41:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:05 smithi190 ceph-mon[33748]: 5.3 scrub starts 2024-03-24T12:41:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:05 smithi190 ceph-mon[33748]: 5.3 scrub ok 2024-03-24T12:41:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:05 smithi190 ceph-mon[33748]: pgmap v633: 148 pgs: 17 active+clean+remapped, 96 active+clean, 27 active+undersized, 8 active+undersized+degraded; 582 KiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 26/594 objects degraded (4.377%); 26/594 objects misplaced (4.377%) 2024-03-24T12:41:05.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:05.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:05.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:05.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:05.912 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:05.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:05.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:06.014 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:06.056 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:06.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:06.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:06.200 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:06.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:06.275 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 2 16 6708 6692 209.109 189.688 0.00215231 0.00477002 2024-03-24T12:41:06.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:06.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:06.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:06.438 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:06 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:41:06.439 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:06 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:41:06.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:06 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:41:06.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:06 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:41:06.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:06.495 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:06.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:06.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:06 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:41:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:06 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:41:06.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:06.637 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:06.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:06.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:06.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:06.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:06.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:06.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:06.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:07.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:07.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:07.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:07.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:07.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:07.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:07.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:07.203 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:41:07.275 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 3 15 10016 10001 208.336 206.812 0.00654097 0.0047929 2024-03-24T12:41:07.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:07.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:07.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:07.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:07.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:07 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3468698533' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:41:07.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:07 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:41:07.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:07 smithi029 ceph-mon[41500]: pgmap v634: 148 pgs: 17 active+clean+remapped, 96 active+clean, 26 active+undersized, 9 active+undersized+degraded; 90 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 775 B/s rd, 14 MiB/s wr, 217 op/s; 472/4896 objects degraded (9.641%); 26/4896 objects misplaced (0.531%) 2024-03-24T12:41:07.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:07 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3468698533' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:41:07.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:07 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:41:07.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:07 smithi029 ceph-mon[29919]: pgmap v634: 148 pgs: 17 active+clean+remapped, 96 active+clean, 26 active+undersized, 9 active+undersized+degraded; 90 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 775 B/s rd, 14 MiB/s wr, 217 op/s; 472/4896 objects degraded (9.641%); 26/4896 objects misplaced (0.531%) 2024-03-24T12:41:07.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:07.456 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:07.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:07.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:07.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:07.599 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:07.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:07 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3468698533' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:41:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:07 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:41:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:07 smithi190 ceph-mon[33748]: pgmap v634: 148 pgs: 17 active+clean+remapped, 96 active+clean, 26 active+undersized, 9 active+undersized+degraded; 90 MiB data, 4.3 GiB used, 711 GiB / 715 GiB avail; 775 B/s rd, 14 MiB/s wr, 217 op/s; 472/4896 objects degraded (9.641%); 26/4896 objects misplaced (0.531%) 2024-03-24T12:41:07.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:07.739 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:07.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:07.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:07.841 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:07.880 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:07.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:07.906 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:41:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:41:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:41:07.982 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:08.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:08.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:08.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:08.118 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:08.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:08.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:08.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:08.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:08.275 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 4 16 12907 12891 201.369 180.625 0.00442804 0.00495922 2024-03-24T12:41:08.302 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:08.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:08.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:08.438 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:08.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:08.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:08.578 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:08.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:08.602 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:41:08.602 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:41:08.602 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:41:08.602 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 472/4896 objects degraded (9.641%), 9 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:08.602 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:41:08.602 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[41500]: osdmap e285: 8 total, 7 up, 8 in 2024-03-24T12:41:08.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:41:08.603 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:41:08.603 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:41:08.603 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 472/4896 objects degraded (9.641%), 9 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:08.603 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:41:08.603 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:08 smithi029 ceph-mon[29919]: osdmap e285: 8 total, 7 up, 8 in 2024-03-24T12:41:08.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:41:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:41:08.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:08 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:41:08.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:08 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:41:08.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:08 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:41:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:08 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 472/4896 objects degraded (9.641%), 9 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:08 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:41:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:08 smithi190 ceph-mon[33748]: osdmap e285: 8 total, 7 up, 8 in 2024-03-24T12:41:08.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:08.727 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:08.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:08.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:08.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:08.868 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:08.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:08.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:08.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:09.009 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:09.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:09.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:09.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:09.150 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:09.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:09.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:09.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 5 15 15859 15844 198.005 184.562 0.00481153 0.00504446 2024-03-24T12:41:09.287 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:09.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:09.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:09.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:09.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:09.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:09.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:09.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:09.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:09 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:41:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:09 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:41:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:09 smithi190 ceph-mon[33748]: osdmap e286: 8 total, 7 up, 8 in 2024-03-24T12:41:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:09 smithi190 ceph-mon[33748]: pgmap v637: 148 pgs: 17 active+clean+remapped, 96 active+clean, 24 active+undersized, 11 active+undersized+degraded; 470 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s; 1874/23103 objects degraded (8.112%); 26/23103 objects misplaced (0.113%) 2024-03-24T12:41:09.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:09 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:41:09.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:09 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:41:09.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:09.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:09 smithi029 ceph-mon[29919]: osdmap e286: 8 total, 7 up, 8 in 2024-03-24T12:41:09.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:09 smithi029 ceph-mon[29919]: pgmap v637: 148 pgs: 17 active+clean+remapped, 96 active+clean, 24 active+undersized, 11 active+undersized+degraded; 470 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s; 1874/23103 objects degraded (8.112%); 26/23103 objects misplaced (0.113%) 2024-03-24T12:41:09.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:09 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:41:09.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:09 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:41:09.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:09.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:09 smithi029 ceph-mon[41500]: osdmap e286: 8 total, 7 up, 8 in 2024-03-24T12:41:09.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:09 smithi029 ceph-mon[41500]: pgmap v637: 148 pgs: 17 active+clean+remapped, 96 active+clean, 24 active+undersized, 11 active+undersized+degraded; 470 MiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.25k op/s; 1874/23103 objects degraded (8.112%); 26/23103 objects misplaced (0.113%) 2024-03-24T12:41:09.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:09.708 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:09.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:09.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:09.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:09.853 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:09.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:09.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:09.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:09.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:09.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:10.032 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:41:10.071 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:41:10.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:10.109 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:41:10.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:10.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:10.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:10.150 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:41:10.194 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:41:10.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:10.235 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:41:10.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:10.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:10.274 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:41:10.275 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:41:10.275 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:41:10.275 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:41:10.275 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:41:10.275 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:41:10.275 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:41:10.275 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:41:10.275 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:41:10.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 6 16 18503 18487 192.534 165.188 0.00393597 0.00518589 2024-03-24T12:41:10.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:10.402 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:10.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:10.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:10.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:10 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/235181103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:41:10.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:10 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/235181103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:41:10.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:10.537 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:10.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:10.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:10 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/235181103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:41:10.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:10.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:10.670 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:10.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:10.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:10.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:10.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:10.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:10.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:11.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:11.079 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:11.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:11.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:11.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:11.216 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:11.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:11.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:11.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 7 15 20667 20652 184.358 135.312 0.00723125 0.00541825 2024-03-24T12:41:11.317 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:11.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:11.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:11.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:11.491 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:11.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:11.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:11.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:11.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:11.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:11.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:11.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:11 smithi190 ceph-mon[33748]: pgmap v638: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 707 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 677 B/s rd, 94 MiB/s wr, 1.50k op/s; 3753/34509 objects degraded (10.875%); 26/34509 objects misplaced (0.075%) 2024-03-24T12:41:11.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:11 smithi029 ceph-mon[41500]: pgmap v638: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 707 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 677 B/s rd, 94 MiB/s wr, 1.50k op/s; 3753/34509 objects degraded (10.875%); 26/34509 objects misplaced (0.075%) 2024-03-24T12:41:11.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:11 smithi029 ceph-mon[29919]: pgmap v638: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 707 MiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 677 B/s rd, 94 MiB/s wr, 1.50k op/s; 3753/34509 objects degraded (10.875%); 26/34509 objects misplaced (0.075%) 2024-03-24T12:41:11.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:11.762 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:11.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:11.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:11.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:11.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:11.888 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:11.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:11.989 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:12.012 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:12.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:12.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:12.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:12.148 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:12.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:12.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:12.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:12.274 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:12.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:12.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:12.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 8 16 22617 22601 176.54 121.812 0.00263692 0.00560839 2024-03-24T12:41:12.375 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:12.415 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:12.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:12.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:12.517 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:12.542 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:12.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:12.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:12.682 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:12.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:12.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:12.817 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:12.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:12.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:12.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:12 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 6232/63651 objects degraded (9.791%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:12.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:12 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 6232/63651 objects degraded (9.791%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:12.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:12.942 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:12.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:12.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:13.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:13.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:13.086 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:13.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:12 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 6232/63651 objects degraded (9.791%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:13.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:13.214 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:13.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:13.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 9 16 24613 24597 170.784 124.75 0.00222616 0.00583342 2024-03-24T12:41:13.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:13.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:13.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:13.458 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:13.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:13.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:13.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:13.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:13.616 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:13.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:13.718 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:13.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:13.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:13.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:13.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:13.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:13.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:13.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:13.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:14.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:14.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:14.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:13 smithi029 ceph-mon[29919]: pgmap v639: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 164 MiB/s wr, 2.63k op/s; 6232/63651 objects degraded (9.791%); 26/63651 objects misplaced (0.041%) 2024-03-24T12:41:14.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:13 smithi029 ceph-mon[41500]: pgmap v639: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 164 MiB/s wr, 2.63k op/s; 6232/63651 objects degraded (9.791%); 26/63651 objects misplaced (0.041%) 2024-03-24T12:41:14.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:14.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:13 smithi190 ceph-mon[33748]: pgmap v639: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.3 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 164 MiB/s wr, 2.63k op/s; 6232/63651 objects degraded (9.791%); 26/63651 objects misplaced (0.041%) 2024-03-24T12:41:14.164 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:14.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:14.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:14.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 10 16 26828 26812 167.549 138.438 0.00524196 0.005964 2024-03-24T12:41:14.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:14.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:14.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:14.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:14.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:14.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:14.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:14.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:14.565 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:14.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:14.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:14.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:14.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:14.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:14.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:14.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:14.817 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:14.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:14.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:14.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:15.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:15.083 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:15.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:15.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:15.209 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:15.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:15.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 11 16 27895 27879 158.38 66.6875 0.00622312 0.00630796 2024-03-24T12:41:15.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:15.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:15.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:15.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:15.449 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:15.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:15.483 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:15.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:15.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:15.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:15.608 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:15.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:15.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:15.742 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:15.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:15.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:15.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:15.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:15.879 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:15.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:15.980 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:16.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:16.106 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:15 smithi190 ceph-mon[33748]: pgmap v640: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 168 MiB/s wr, 2.70k op/s; 7525/69618 objects degraded (10.809%); 26/69618 objects misplaced (0.037%) 2024-03-24T12:41:16.137 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:16.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:16.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:16.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:15 smithi029 ceph-mon[29919]: pgmap v640: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 168 MiB/s wr, 2.70k op/s; 7525/69618 objects degraded (10.809%); 26/69618 objects misplaced (0.037%) 2024-03-24T12:41:16.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:15 smithi029 ceph-mon[41500]: pgmap v640: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.4 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 168 MiB/s wr, 2.70k op/s; 7525/69618 objects degraded (10.809%); 26/69618 objects misplaced (0.037%) 2024-03-24T12:41:16.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:16.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:16.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:16.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:16.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 12 16 29199 29183 151.973 81.5 0.00286742 0.00653535 2024-03-24T12:41:16.367 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:16.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:16.406 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:16.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:16.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:16.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:16.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:16.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:16.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:16.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:16.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:16.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:16.795 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:16.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:16.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:16.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:16.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:16.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:17.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:17.069 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:17.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:17.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:17.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:17.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:17.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 13 16 30753 30737 147.753 97.125 0.00157622 0.00669923 2024-03-24T12:41:17.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:17.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:17.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:17.436 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:17.469 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:17.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:17.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:17.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:17.606 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:17.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:17.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:17.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:17.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:17.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:17.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:17.866 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:17.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:17 smithi029 ceph-mon[29919]: pgmap v641: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 156 MiB/s wr, 2.50k op/s; 8058/76296 objects degraded (10.561%); 26/76296 objects misplaced (0.034%) 2024-03-24T12:41:17.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:17 smithi029 ceph-mon[41500]: pgmap v641: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 156 MiB/s wr, 2.50k op/s; 8058/76296 objects degraded (10.561%); 26/76296 objects misplaced (0.034%) 2024-03-24T12:41:17.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:41:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:41:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:41:17.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:18.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:18.005 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:18.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:18.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:18.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:18.131 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:18.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:17 smithi190 ceph-mon[33748]: pgmap v641: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.5 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.0 KiB/s rd, 156 MiB/s wr, 2.50k op/s; 8058/76296 objects degraded (10.561%); 26/76296 objects misplaced (0.034%) 2024-03-24T12:41:18.233 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:18.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:18.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:18.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:18.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 14 16 32306 32290 144.132 97.0625 0.0130607 0.00693176 2024-03-24T12:41:18.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:18.402 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:18.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:18.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:18.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:18.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:18.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:41:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:41:18.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:18.685 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:18.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:18.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:18.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:18.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:18.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:18.939 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:18.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:18.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:19.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:19.081 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:19.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:19.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:19.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:19.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:19.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 15 16 33974 33958 141.472 104.25 0.00707945 0.00706334 2024-03-24T12:41:19.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:19.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:19.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:41:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:19.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:41:19.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:19.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:41:19.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:19.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:19.505 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:19.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:19.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:19.607 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:19.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:19.631 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:19.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:19.732 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:19.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:19.757 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:19.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:19.858 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:19.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:19.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:19.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:19.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:20.029 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:20.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:20.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:20.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:20.157 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:20.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:20.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:20.276 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 16 16 35449 35433 138.391 92.1875 0.006301 0.00722036 2024-03-24T12:41:20.296 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:20.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:19 smithi190 ceph-mon[33748]: pgmap v642: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 974 B/s rd, 133 MiB/s wr, 2.13k op/s; 9169/90339 objects degraded (10.150%); 26/90339 objects misplaced (0.029%) 2024-03-24T12:41:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:19 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 9169/90339 objects degraded (10.150%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:20.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:20.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:20.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:19 smithi029 ceph-mon[41500]: pgmap v642: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 974 B/s rd, 133 MiB/s wr, 2.13k op/s; 9169/90339 objects degraded (10.150%); 26/90339 objects misplaced (0.029%) 2024-03-24T12:41:20.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:19 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 9169/90339 objects degraded (10.150%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:19 smithi029 ceph-mon[29919]: pgmap v642: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 974 B/s rd, 133 MiB/s wr, 2.13k op/s; 9169/90339 objects degraded (10.150%); 26/90339 objects misplaced (0.029%) 2024-03-24T12:41:20.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:19 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 9169/90339 objects degraded (10.150%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:20.438 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:20.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:20.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:20.570 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:20.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:20.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:20.713 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:20.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:20.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:20.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:20.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:20.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:20.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:21.001 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:21.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:21.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:21.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:21.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:21.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:21.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:21.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:21.263 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:21.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:21.277 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 17 16 37243 37227 136.846 112.125 0.00180129 0.00724926 2024-03-24T12:41:21.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:21.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:21.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:21.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:21.549 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:21.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:21.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:21.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:21.685 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:21.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:21.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:21.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:21.824 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:21.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:21.925 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:21.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:21.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:22.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:22.086 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:22.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:22.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:22.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:22.227 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:22.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:22.277 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 18 16 39080 39064 135.621 114.812 0.00472267 0.00736855 2024-03-24T12:41:22.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:22.367 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:22.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:22.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:22.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:22 smithi190 ceph-mon[33748]: pgmap v643: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 126 MiB/s wr, 2.01k op/s; 10027/95643 objects degraded (10.484%); 26/95643 objects misplaced (0.027%) 2024-03-24T12:41:22.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:22 smithi029 ceph-mon[41500]: pgmap v643: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 126 MiB/s wr, 2.01k op/s; 10027/95643 objects degraded (10.484%); 26/95643 objects misplaced (0.027%) 2024-03-24T12:41:22.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:22 smithi029 ceph-mon[29919]: pgmap v643: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 1.9 GiB data, 9.4 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 126 MiB/s wr, 2.01k op/s; 10027/95643 objects degraded (10.484%); 26/95643 objects misplaced (0.027%) 2024-03-24T12:41:22.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:22.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:22.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:22.646 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:22.647 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:22.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:22.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:22.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:22.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:22.873 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:22.908 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:22.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:22.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:23.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:23.036 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:23.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:23.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:23.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:23.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:23.177 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:23.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:23.277 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 19 16 40758 40742 134.002 104.875 0.00609116 0.00745744 2024-03-24T12:41:23.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:23.287 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:41:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:23.317 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:23.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:23.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:23.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:23.470 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:23.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:23.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:23.624 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:23.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:23.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:23.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:23.783 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:23.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:23.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:23.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:23.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:24.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:24.078 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:24.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:24.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:24.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:24.219 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:24.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:24.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:24.277 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:41:24.282418+0000 min lat: 0.00126577 max lat: 0.249075 avg lat: 0.0075329 2024-03-24T12:41:24.277 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:41:24.277 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 20 16 42472 42456 132.658 107.125 0.00492675 0.0075329 2024-03-24T12:41:24.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:24.340 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:24 smithi029 ceph-mon[41500]: pgmap v644: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s; 11592/114171 objects degraded (10.153%); 26/114171 objects misplaced (0.023%) 2024-03-24T12:41:24.341 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:24 smithi029 ceph-mon[29919]: pgmap v644: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s; 11592/114171 objects degraded (10.153%); 26/114171 objects misplaced (0.023%) 2024-03-24T12:41:24.368 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:24.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:24 smithi190 ceph-mon[33748]: pgmap v644: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.21k op/s; 11592/114171 objects degraded (10.153%); 26/114171 objects misplaced (0.023%) 2024-03-24T12:41:24.453 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:41:24.453 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:41:24.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:132118 8.3 8:c934d450:::benchmark_data_smithi029_108893_object18258:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:34:45.999360+0000", 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 398.45310195399998, 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.44550312199999997, 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:41:24.454 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 132118 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999360+0000", 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999360+0000", 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999361+0000", 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.189e-06 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999367+0000", 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6570000000000002e-06 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999378+0000", 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0954e-05 2024-03-24T12:41:24.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:45.999386+0000", 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2630000000000003e-06 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.010021+0000", 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010634727 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.457 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.010155+0000", 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000134232 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 5,6", 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.010275+0000", 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000120445 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.013680+0000", 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0034047539999999999 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.458 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.439925+0000", 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42624483899999999 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.444736+0000", 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0048109759999999998 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.444797+0000", 2024-03-24T12:41:24.459 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0776999999999998e-05 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.444863+0000", 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6309000000000005e-05 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:132138 8.a 8:583ccc1f:::benchmark_data_smithi029_108893_object18278:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e189)", 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:34:46.016245+0000", 2024-03-24T12:41:24.460 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 398.43621694699999, 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.436156192, 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 132138 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016245+0000", 2024-03-24T12:41:24.461 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016245+0000", 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016246+0000", 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2279999999999999e-06 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.462 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016250+0000", 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.172e-06 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016259+0000", 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2359999999999993e-06 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.016266+0000", 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1489999999999999e-06 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.463 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.439996+0000", 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42373073500000003 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440135+0000", 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013908400000000001 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440278+0000", 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014265999999999999 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.464 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.449225+0000", 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0089471140000000008 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.449357+0000", 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000131564 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.452245+0000", 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002888467 2024-03-24T12:41:24.465 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.452351+0000", 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105548 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.452401+0000", 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0235000000000003e-05 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.466 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.24815.0:132141 8.f e189/158 8:f6572891:::benchmark_data_smithi029_108893_object18281:head v 189'8315, mlcod=189'8313)", 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:34:46.018414+0000", 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 398.43404734199999, 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42895093499999998, 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.018414+0000", 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.467 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.018414+0000", 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.018415+0000", 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5799999999999999e-07 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.018421+0000", 2024-03-24T12:41:24.468 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2330000000000003e-06 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440292+0000", 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42187096600000001 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440304+0000", 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1215e-05 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440368+0000", 2024-03-24T12:41:24.469 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4675e-05 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.440392+0000", 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3594000000000001e-05 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.447323+0000", 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0069309710000000002 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.470 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:46.447365+0000", 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2623000000000002e-05 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:206534 8.a 8:549f3ac9:::benchmark_data_smithi029_108893_object92674:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:57.688035+0000", 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 326.76442678199999, 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0529943820000001, 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:41:24.471 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.472 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:41:24.472 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:41:24.472 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 206534 2024-03-24T12:41:24.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.472 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.472 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.472 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.472 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688035+0000", 2024-03-24T12:41:24.472 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.472 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688035+0000", 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688037+0000", 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5540000000000001e-06 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.473 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688042+0000", 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9219999999999996e-06 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688058+0000", 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6325e-05 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688067+0000", 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5780000000000005e-06 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.474 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688128+0000", 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0321000000000001e-05 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688265+0000", 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013724000000000001 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.688443+0000", 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000178054 2024-03-24T12:41:24.475 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.708590+0000", 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.020147302999999998 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.708600+0000", 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6330000000000008e-06 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.476 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.740862+0000", 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.032261922 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.741029+0000", 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016752999999999999 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:206542 8.a 8:5adb6f6b:::benchmark_data_smithi029_108893_object92682:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:41:24.477 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:57.711565+0000", 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 326.74089722299999, 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0341805719999999, 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 206542 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.478 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711565+0000", 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711565+0000", 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711566+0000", 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.761e-06 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711571+0000", 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0980000000000001e-06 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711584+0000", 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2625e-05 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711590+0000", 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.93e-06 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.480 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711637+0000", 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6998e-05 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711645+0000", 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7009999999999996e-06 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.741788+0000", 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.030143448 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.482 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.741954+0000", 2024-03-24T12:41:24.482 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016632800000000001 2024-03-24T12:41:24.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2", 2024-03-24T12:41:24.482 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.742106+0000", 2024-03-24T12:41:24.482 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:24.484 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015137899999999999 2024-03-24T12:41:24.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.484 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.484 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.745416+0000", 2024-03-24T12:41:24.484 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0033100439999999998 2024-03-24T12:41:24.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.484 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.484 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.745629+0000", 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021347200000000001 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.745670+0000", 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0614000000000001e-05 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.745745+0000", 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5173999999999995e-05 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.485 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:2089 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 93828934255616 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:58.114923+0000", 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 326.33753918899998, 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.60225106100000003, 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 2089 2024-03-24T12:41:24.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114923+0000", 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114923+0000", 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114924+0000", 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.753e-06 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114930+0000", 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3329999999999996e-06 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114945+0000", 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5656999999999999e-05 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114955+0000", 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7140000000000005e-06 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.115031+0000", 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6396999999999994e-05 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.115044+0000", 2024-03-24T12:41:24.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2275e-05 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.711561+0000", 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.59651754199999996 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.711857+0000", 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029541399999999997 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.490 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.717174+0000", 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0053169760000000002 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104325 9.0 9:05e930a3:::benchmark_data_smithi029_134815_object104324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.117256+0000", 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 194.335205785, 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.081903713, 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.491 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104325 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117256+0000", 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117256+0000", 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117258+0000", 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5710000000000001e-06 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117282+0000", 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4168999999999998e-05 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.493 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117294+0000", 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1813e-05 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117303+0000", 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9670000000000007e-06 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.118795+0000", 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0014927600000000001 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.118807+0000", 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1378e-05 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077574+0000", 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.958767339 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077587+0000", 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3011e-05 2024-03-24T12:41:24.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.176190+0000", 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0986029450000001 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.176338+0000", 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014812700000000001 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2,6", 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.176830+0000", 2024-03-24T12:41:24.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00049169399999999999 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.191858+0000", 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.015028036999999999 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.191903+0000", 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5529999999999999e-05 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.497 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.193131+0000", 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001227552 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.198956+0000", 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0058248479999999997 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199033+0000", 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6946000000000005e-05 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.498 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199160+0000", 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000127026 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104332 9.0 9:0a464280:::benchmark_data_smithi029_134815_object104331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.128520+0000", 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 194.32394213800001, 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0715006140000001, 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104332 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128520+0000", 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.500 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128520+0000", 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128521+0000", 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.815e-06 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128548+0000", 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6149999999999999e-05 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128568+0000", 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0421e-05 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128579+0000", 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0565e-05 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128649+0000", 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0376000000000005e-05 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128660+0000", 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1141999999999999e-05 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077614+0000", 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94895389900000005 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077623+0000", 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1479999999999995e-06 2024-03-24T12:41:24.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.184183+0000", 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1065596470000001 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.184310+0000", 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012726999999999999 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2,6", 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.184474+0000", 2024-03-24T12:41:24.504 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016396 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.191923+0000", 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0074493980000000003 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.192359+0000", 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00043587299999999998 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.505 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.193191+0000", 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083197800000000004 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199218+0000", 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060271320000000001 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199289+0000", 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0358000000000006e-05 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.506 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.200020+0000", 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073148199999999999 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104356 9.0 9:08e6f257:::benchmark_data_smithi029_134815_object104355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.157948+0000", 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 194.29451381800001, 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0423164210000002, 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.507 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104356 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157948+0000", 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.508 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157948+0000", 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157950+0000", 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6619999999999999e-06 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157972+0000", 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2824e-05 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.509 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157987+0000", 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4649e-05 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157995+0000", 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2199999999999992e-06 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.158067+0000", 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1746000000000001e-05 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.158075+0000", 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7980000000000006e-06 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077650+0000", 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.91957491400000002 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077659+0000", 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8459999999999997e-06 2024-03-24T12:41:24.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.189246+0000", 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1115876039999999 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.189379+0000", 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013289700000000001 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2,6", 2024-03-24T12:41:24.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.189577+0000", 2024-03-24T12:41:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019751600000000001 2024-03-24T12:41:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.192372+0000", 2024-03-24T12:41:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027951769999999998 2024-03-24T12:41:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.513 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.193087+0000", 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00071518999999999999 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194000+0000", 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00091256499999999995 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.200074+0000", 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060745529999999999 2024-03-24T12:41:24.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.200135+0000", 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1086000000000004e-05 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.200264+0000", 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012917399999999999 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:114825 9.4 9:2e3464cd:::benchmark_data_smithi029_134815_object2759:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:27.572952+0000", 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 176.87950985699999, 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.447432828, 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 114825 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.572952+0000", 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.572952+0000", 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.572953+0000", 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.271e-06 2024-03-24T12:41:24.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.572958+0000", 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8559999999999998e-06 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.572970+0000", 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1742e-05 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.572978+0000", 2024-03-24T12:41:24.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.337e-06 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.930625+0000", 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.35764721399999999 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.930735+0000", 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000109418 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:41:24.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.930845+0000", 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000110622 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.939058+0000", 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0082126930000000001 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.942261+0000", 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0032024890000000002 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.520 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:28.020271+0000", 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.078010170000000004 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:28.020322+0000", 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1617999999999999e-05 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:28.020385+0000", 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2397999999999994e-05 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.521 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:114837 9.4 9:21992367:::benchmark_data_smithi029_134815_object2771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:27.584197+0000", 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 176.86826432800001, 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.43629191099999998, 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:41:24.522 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 114837 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.584197+0000", 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.584197+0000", 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.584199+0000", 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.054e-06 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.584202+0000", 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6600000000000001e-06 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.584209+0000", 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1860000000000003e-06 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.584216+0000", 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2419999999999997e-06 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.934392+0000", 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.350176184 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.934519+0000", 2024-03-24T12:41:24.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012680600000000001 2024-03-24T12:41:24.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:41:24.527 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.934650+0000", 2024-03-24T12:41:24.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013142300000000001 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.939134+0000", 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0044841330000000004 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:27.946287+0000", 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0071532999999999996 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:28.020396+0000", 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.074108456000000003 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:28.020439+0000", 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3409999999999999e-05 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:28.020489+0000", 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0056999999999999e-05 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.529 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(osd.1.0:103285 9.4 e238/212 ondisk, result = 0)", 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:31.139885+0000", 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.31257630900001, 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.43235553599999998, 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.139885+0000", 2024-03-24T12:41:24.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.139885+0000", 2024-03-24T12:41:24.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.139886+0000", 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2699999999999998e-07 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.139892+0000", 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.524e-06 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.139902+0000", 2024-03-24T12:41:24.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8320000000000001e-06 2024-03-24T12:41:24.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.139921+0000", 2024-03-24T12:41:24.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9126e-05 2024-03-24T12:41:24.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572118+0000", 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.432196727 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572128+0000", 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0492000000000001e-05 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572241+0000", 2024-03-24T12:41:24.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113008 2024-03-24T12:41:24.538 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.538 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.538 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.538 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(osd.1.0:103286 9.4 e238/212 ondisk, result = 0)", 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:31.142630+0000", 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.309831454, 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.429656651, 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.142630+0000", 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.142630+0000", 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.142631+0000", 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5999999999999991e-07 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.142636+0000", 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9370000000000003e-06 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.142646+0000", 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4870000000000008e-06 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.142652+0000", 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5359999999999998e-06 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572268+0000", 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42961529100000001 2024-03-24T12:41:24.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572275+0000", 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5129999999999999e-06 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572287+0000", 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1926999999999999e-05 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.543 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(osd.1.0:103285 9.4 e238/212 ondisk, result = 0)", 2024-03-24T12:41:24.543 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:31.144650+0000", 2024-03-24T12:41:24.543 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.30781178800001, 2024-03-24T12:41:24.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42775347499999999, 2024-03-24T12:41:24.543 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.543 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:41:24.543 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144650+0000", 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144650+0000", 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144651+0000", 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0800000000000004e-07 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144654+0000", 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.743e-06 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144666+0000", 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1559000000000001e-05 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144672+0000", 2024-03-24T12:41:24.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3350000000000003e-06 2024-03-24T12:41:24.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572309+0000", 2024-03-24T12:41:24.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42763689199999999 2024-03-24T12:41:24.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572316+0000", 2024-03-24T12:41:24.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5350000000000001e-06 2024-03-24T12:41:24.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572403+0000", 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7702999999999997e-05 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(osd.1.0:103286 9.4 e238/212 ondisk, result = 0)", 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:31.144928+0000", 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.30753416600001, 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.43130395399999999, 2024-03-24T12:41:24.548 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.549 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:41:24.549 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.549 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144928+0000", 2024-03-24T12:41:24.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144928+0000", 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144928+0000", 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9299999999999999e-07 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144931+0000", 2024-03-24T12:41:24.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.791e-06 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144938+0000", 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0380000000000002e-06 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.144943+0000", 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1200000000000001e-06 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572491+0000", 2024-03-24T12:41:24.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42754842599999998 2024-03-24T12:41:24.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.552 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.552 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572498+0000", 2024-03-24T12:41:24.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8190000000000003e-06 2024-03-24T12:41:24.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.576232+0000", 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003733367 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(osd.1.0:103288 9.4 e238/212 ondisk, result = 0)", 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:31.149953+0000", 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 173.30250923700001, 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42250303900000002, 2024-03-24T12:41:24.553 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.149953+0000", 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.149953+0000", 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.149953+0000", 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7899999999999997e-07 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.149962+0000", 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.037e-06 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.149973+0000", 2024-03-24T12:41:24.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0882e-05 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.149981+0000", 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4680000000000003e-06 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572437+0000", 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42245610300000003 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572444+0000", 2024-03-24T12:41:24.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8499999999999996e-06 2024-03-24T12:41:24.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:31.572456+0000", 2024-03-24T12:41:24.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1919999999999999e-05 2024-03-24T12:41:24.557 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.557 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.557 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:117661 9.4 9:265558ae:::benchmark_data_smithi029_134815_object5595:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:33.157396+0000", 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 171.29506580399999, 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.479367667, 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 117661 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.558 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.157396+0000", 2024-03-24T12:41:24.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.157396+0000", 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.157397+0000", 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7199999999999998e-07 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.157402+0000", 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8470000000000003e-06 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.157412+0000", 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0198e-05 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.157420+0000", 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0830000000000004e-06 2024-03-24T12:41:24.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.220400+0000", 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.062980098999999998 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.220642+0000", 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024217299999999999 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.220795+0000", 2024-03-24T12:41:24.562 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015297499999999999 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.224337+0000", 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0035417000000000001 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.232779+0000", 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084422639999999997 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.563 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.636650+0000", 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40387086300000002 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.636704+0000", 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4323999999999997e-05 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:33.636764+0000", 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9369e-05 2024-03-24T12:41:24.564 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172155 9.0 9:07da41af:::benchmark_data_smithi029_134815_object60089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.686738+0000", 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 110.765723935, 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.508465065, 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:41:24.565 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172155 2024-03-24T12:41:24.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.566 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.566 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.566 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686738+0000", 2024-03-24T12:41:24.566 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.566 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686738+0000", 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686739+0000", 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5209999999999999e-06 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686744+0000", 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7589999999999997e-06 2024-03-24T12:41:24.567 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.568 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686755+0000", 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0711e-05 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686763+0000", 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8010000000000004e-06 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.569 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686822+0000", 2024-03-24T12:41:24.570 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9033000000000002e-05 2024-03-24T12:41:24.570 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.570 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.570 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.570 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686972+0000", 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000149894 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.687082+0000", 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000110303 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.689792+0000", 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027105240000000002 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.571 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.692544+0000", 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027517980000000002 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.751725+0000", 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.059180952000000002 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.751741+0000", 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5579000000000001e-05 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.572 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738213+0000", 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.98647265900000003 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738227+0000", 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3541e-05 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.195103+0000", 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45687653900000003 2024-03-24T12:41:24.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.574 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.574 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.574 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.195203+0000", 2024-03-24T12:41:24.574 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9450999999999994e-05 2024-03-24T12:41:24.574 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.574 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172200 9.0 9:061b9bea:::benchmark_data_smithi029_134815_object60134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:41:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.785371+0000", 2024-03-24T12:41:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 110.66709107299999, 2024-03-24T12:41:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.419709179, 2024-03-24T12:41:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.575 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.576 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.576 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:41:24.576 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:41:24.576 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172200 2024-03-24T12:41:24.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.576 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.576 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785371+0000", 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785371+0000", 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785372+0000", 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4789999999999999e-06 2024-03-24T12:41:24.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785378+0000", 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2260000000000002e-06 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785390+0000", 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1697e-05 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785399+0000", 2024-03-24T12:41:24.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1880000000000008e-06 2024-03-24T12:41:24.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785495+0000", 2024-03-24T12:41:24.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6033999999999996e-05 2024-03-24T12:41:24.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785502+0000", 2024-03-24T12:41:24.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5849999999999996e-06 2024-03-24T12:41:24.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738257+0000", 2024-03-24T12:41:24.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.95275513099999998 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738266+0000", 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0219999999999993e-06 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.196663+0000", 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45839733799999999 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.196781+0000", 2024-03-24T12:41:24.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000118023 2024-03-24T12:41:24.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2,6", 2024-03-24T12:41:24.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.196907+0000", 2024-03-24T12:41:24.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012570600000000001 2024-03-24T12:41:24.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.583 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202007+0000", 2024-03-24T12:41:24.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0050995379999999998 2024-03-24T12:41:24.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.583 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:24.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:24.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:24.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202150+0000", 2024-03-24T12:41:24.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000142955 2024-03-24T12:41:24.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202201+0000", 2024-03-24T12:41:24.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1745e-05 2024-03-24T12:41:24.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.586 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.586 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.204877+0000", 2024-03-24T12:41:24.586 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026757270000000001 2024-03-24T12:41:24.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.586 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.586 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.204947+0000", 2024-03-24T12:41:24.586 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9937000000000005e-05 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.205080+0000", 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000132848 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172230 9.0 9:0020a4bd:::benchmark_data_smithi029_134815_object60164:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:41:24.587 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.879008+0000", 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 110.573453324, 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3297065889999999, 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172230 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:41:24.589 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.879008+0000", 2024-03-24T12:41:24.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.589 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:41:24.589 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.879008+0000", 2024-03-24T12:41:24.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:41:24.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.879010+0000", 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.618e-06 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.879015+0000", 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0919999999999997e-06 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.879028+0000", 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2933e-05 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.879036+0000", 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.123e-06 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.879108+0000", 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1291999999999994e-05 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.879137+0000", 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9269999999999999e-05 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738292+0000", 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.85915495600000003 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738300+0000", 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2060000000000007e-06 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.199372+0000", 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.46107213200000002 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.199522+0000", 2024-03-24T12:41:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014944200000000001 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2,6", 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.199669+0000", 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014723899999999999 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202437+0000", 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027687110000000001 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202485+0000", 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7503999999999998e-05 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.208335+0000", 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0058503549999999998 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.208544+0000", 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020880000000000001 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.208600+0000", 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5860999999999999e-05 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.208715+0000", 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115055 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:41:24.596 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:41:24.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:24.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:24.718 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:24.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:24.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:24.850 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:24.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:24.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:24.995 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:24.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:25.097 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:25.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:25.177 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:41:25.179 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:41:25.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:25.277 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 21 16 44152 44136 131.341 105 0.00691439 0.00760941 2024-03-24T12:41:25.278 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:25.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:25.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:25.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:25 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 11592/114171 objects degraded (10.153%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:25.415 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:25.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:25.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:25.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:25 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 11592/114171 objects degraded (10.153%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:25.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:25 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 11592/114171 objects degraded (10.153%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:25.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:25.559 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:25.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:25.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:25.701 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:25.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:25.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:25.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:25.841 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:25.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:25.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:25.983 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:25.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:26.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:26.115 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:26.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:26.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:26.146 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:41:26.146 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:41:26.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:26.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:26.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:26.277 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 22 16 46043 46027 130.742 118.188 0.0087391 0.0076434 2024-03-24T12:41:26.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:26.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:26.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:26.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:26 smithi029 ceph-mon[29919]: pgmap v645: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s; 12639/118923 objects degraded (10.628%); 26/118923 objects misplaced (0.022%) 2024-03-24T12:41:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:26 smithi029 ceph-mon[41500]: pgmap v645: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s; 12639/118923 objects degraded (10.628%); 26/118923 objects misplaced (0.022%) 2024-03-24T12:41:26.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:26.546 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:26.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:26.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:26.565 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:26.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:26 smithi190 ceph-mon[33748]: pgmap v645: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.4 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 96 MiB/s wr, 1.53k op/s; 12639/118923 objects degraded (10.628%); 26/118923 objects misplaced (0.022%) 2024-03-24T12:41:26.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:26.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:26.697 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:26.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:26.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:26.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:26.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:26.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:26.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:26.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:27.081 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:27.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:27.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:27.218 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:27.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T12:41:27.250 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:27.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:27.279 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 23 16 47883 47867 130.046 115 0.00511445 0.00768411 2024-03-24T12:41:27.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:27.397 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:27.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:27.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:27.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:27 smithi029 ceph-mon[29919]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:41:27.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:27 smithi029 ceph-mon[41500]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:41:27.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:27.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:27.554 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:27.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:27 smithi190 ceph-mon[33748]: from='client.25154 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:41:27.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:27.690 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:27.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:27.791 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:27.825 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:27.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:27.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:41:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:41:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:41:27.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:27.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:27.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:27.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:28.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:28.094 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:28.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:28.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:28.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:41:28.148 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":645,"stamp":"2024-03-24T12:41:24.732113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2585532766,"num_objects":39641,"num_object_clones":0,"num_object_copies":118923,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12639,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":39641,"num_whiteouts":0,"num_read":2264,"num_read_kb":2110,"num_write":40344,"num_write_kb":2526813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":82,"num_bytes_recovered":2953658,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":40425,"ondisk_log_size":40425,"up":392,"acting":409,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":466,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11095392,"kb_used_data":7226720,"kb_used_omap":12,"kb_used_meta":3868595,"kb_avail":738865824,"statfs":{"total":767960285184,"available":756598603776,"internally_reserved":0,"allocated":7400161280,"data_stored":7395846412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":3961441841},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,5,0,2,0,1,12],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1207435264,"num_objects":18424,"num_object_clones":0,"num_object_copies":55272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6407,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18424,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":18424,"num_write_kb":1179136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007061"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":478,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.222268+0000","last_change":"2024-03-24T12:35:58.701806+0000","last_active":"2024-03-24T12:41:08.222268+0000","last_peered":"2024-03-24T12:41:08.222268+0000","last_clean":"2024-03-24T12:35:54.676897+0000","last_became_active":"2024-03-24T12:35:58.701806+0000","last_became_peered":"2024-03-24T12:35:58.701806+0000","last_unstale":"2024-03-24T12:41:08.222268+0000","last_undegraded":"2024-03-24T12:41:08.222268+0000","last_fullsized":"2024-03-24T12:35:58.695795+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:08.859869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.859869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:15.952259+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":397,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.223652+0000","last_change":"2024-03-24T12:39:07.750122+0000","last_active":"2024-03-24T12:41:08.223652+0000","last_peered":"2024-03-24T12:41:08.223652+0000","last_clean":"2024-03-24T12:35:54.687134+0000","last_became_active":"2024-03-24T12:39:07.750122+0000","last_became_peered":"2024-03-24T12:39:07.750122+0000","last_unstale":"2024-03-24T12:41:08.223652+0000","last_undegraded":"2024-03-24T12:41:08.223652+0000","last_fullsized":"2024-03-24T12:39:07.746192+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:48.744937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.744937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":483,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.222381+0000","last_change":"2024-03-24T12:35:58.712452+0000","last_active":"2024-03-24T12:41:08.222381+0000","last_peered":"2024-03-24T12:41:08.222381+0000","last_clean":"2024-03-24T12:35:54.683285+0000","last_became_active":"2024-03-24T12:35:58.712452+0000","last_became_peered":"2024-03-24T12:35:58.712452+0000","last_unstale":"2024-03-24T12:41:08.222381+0000","last_undegraded":"2024-03-24T12:41:08.222381+0000","last_fullsized":"2024-03-24T12:35:58.707789+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:14.966328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:14.966328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.359848+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"207'15","reported_seq":556,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221007+0000","last_change":"2024-03-24T12:40:20.877349+0000","last_active":"2024-03-24T12:41:08.221007+0000","last_peered":"2024-03-24T12:41:08.221007+0000","last_clean":"2024-03-24T12:41:08.221007+0000","last_became_active":"2024-03-24T12:39:34.752258+0000","last_became_peered":"2024-03-24T12:39:34.752258+0000","last_unstale":"2024-03-24T12:41:08.221007+0000","last_undegraded":"2024-03-24T12:41:08.221007+0000","last_fullsized":"2024-03-24T12:41:08.221007+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:40:20.877200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:20.877200+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:27.189243+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.230448+0000","last_change":"2024-03-24T12:39:34.758724+0000","last_active":"2024-03-24T12:41:08.230448+0000","last_peered":"2024-03-24T12:41:08.230448+0000","last_clean":"2024-03-24T12:41:08.230448+0000","last_became_active":"2024-03-24T12:39:34.758425+0000","last_became_peered":"2024-03-24T12:39:34.758425+0000","last_unstale":"2024-03-24T12:41:08.230448+0000","last_undegraded":"2024-03-24T12:41:08.230448+0000","last_fullsized":"2024-03-24T12:41:08.230448+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:24.826030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:24.826030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":507,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221120+0000","last_change":"2024-03-24T12:40:17.828433+0000","last_active":"2024-03-24T12:41:08.221120+0000","last_peered":"2024-03-24T12:41:08.221120+0000","last_clean":"2024-03-24T12:41:08.221120+0000","last_became_active":"2024-03-24T12:39:34.744030+0000","last_became_peered":"2024-03-24T12:39:34.744030+0000","last_unstale":"2024-03-24T12:41:08.221120+0000","last_undegraded":"2024-03-24T12:41:08.221120+0000","last_fullsized":"2024-03-24T12:41:08.221120+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:17.828354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:17.828354+0000","last_clean_scrub_stamp":"2024-03-24T12:40:17.828354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:23.935435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":508,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.230255+0000","last_change":"2024-03-24T12:40:20.094901+0000","last_active":"2024-03-24T12:41:08.230255+0000","last_peered":"2024-03-24T12:41:08.230255+0000","last_clean":"2024-03-24T12:41:08.230255+0000","last_became_active":"2024-03-24T12:39:34.761198+0000","last_became_peered":"2024-03-24T12:39:34.761198+0000","last_unstale":"2024-03-24T12:41:08.230255+0000","last_undegraded":"2024-03-24T12:41:08.230255+0000","last_fullsized":"2024-03-24T12:41:08.230255+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:20.094789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:20.094789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:28.170840+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"207'34","reported_seq":541,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.229144+0000","last_change":"2024-03-24T12:40:57.892747+0000","last_active":"2024-03-24T12:41:08.229144+0000","last_peered":"2024-03-24T12:41:08.229144+0000","last_clean":"2024-03-24T12:41:08.229144+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:41:08.229144+0000","last_undegraded":"2024-03-24T12:41:08.229144+0000","last_fullsized":"2024-03-24T12:41:08.229144+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-24T12:40:57.892649+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:40:57.892649+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:04.459527+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":491,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.218724+0000","last_change":"2024-03-24T12:39:49.245621+0000","last_active":"2024-03-24T12:41:08.218724+0000","last_peered":"2024-03-24T12:41:08.218724+0000","last_clean":"2024-03-24T12:41:08.218724+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:41:08.218724+0000","last_undegraded":"2024-03-24T12:41:08.218724+0000","last_fullsized":"2024-03-24T12:41:08.218724+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:49.245510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:39:49.245510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":84,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":174,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.221038+0000","last_change":"2024-03-24T12:35:58.711151+0000","last_active":"2024-03-24T12:41:08.221038+0000","last_peered":"2024-03-24T12:41:08.221038+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.711151+0000","last_became_peered":"2024-03-24T12:35:58.711151+0000","last_unstale":"2024-03-24T12:41:08.221038+0000","last_undegraded":"2024-03-24T12:41:08.221038+0000","last_fullsized":"2024-03-24T12:35:58.703392+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:44.161505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.161505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.359834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":191,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.221486+0000","last_change":"2024-03-24T12:39:40.790112+0000","last_active":"2024-03-24T12:41:08.221486+0000","last_peered":"2024-03-24T12:41:08.221486+0000","last_clean":"2024-03-24T12:41:08.221486+0000","last_became_active":"2024-03-24T12:39:35.101206+0000","last_became_peered":"2024-03-24T12:39:35.101206+0000","last_unstale":"2024-03-24T12:41:08.221486+0000","last_undegraded":"2024-03-24T12:41:08.221486+0000","last_fullsized":"2024-03-24T12:41:08.221486+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:40.789901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:39:40.789901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"207'15","reported_seq":516,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222334+0000","last_change":"2024-03-24T12:37:47.713933+0000","last_active":"2024-03-24T12:41:08.222334+0000","last_peered":"2024-03-24T12:41:08.222334+0000","last_clean":"2024-03-24T12:41:08.222334+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:41:08.222334+0000","last_undegraded":"2024-03-24T12:41:08.222334+0000","last_fullsized":"2024-03-24T12:41:08.222334+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:37:47.713795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:47.713795+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":173,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.219556+0000","last_change":"2024-03-24T12:35:58.718922+0000","last_active":"2024-03-24T12:41:08.219556+0000","last_peered":"2024-03-24T12:41:08.219556+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.718922+0000","last_became_peered":"2024-03-24T12:35:58.718922+0000","last_unstale":"2024-03-24T12:41:08.219556+0000","last_undegraded":"2024-03-24T12:41:08.219556+0000","last_fullsized":"2024-03-24T12:35:58.702792+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.155345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.155345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:10.331089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":495,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.223338+0000","last_change":"2024-03-24T12:40:13.040964+0000","last_active":"2024-03-24T12:41:08.223338+0000","last_peered":"2024-03-24T12:41:08.223338+0000","last_clean":"2024-03-24T12:41:08.223338+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:41:08.223338+0000","last_undegraded":"2024-03-24T12:41:08.223338+0000","last_fullsized":"2024-03-24T12:41:08.223338+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:13.040851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:40:13.040851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:13.765684+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":173,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.221588+0000","last_change":"2024-03-24T12:35:58.708742+0000","last_active":"2024-03-24T12:41:08.221588+0000","last_peered":"2024-03-24T12:41:08.221588+0000","last_clean":"2024-03-24T12:26:02.143860+0000","last_became_active":"2024-03-24T12:35:58.708742+0000","last_became_peered":"2024-03-24T12:35:58.708742+0000","last_unstale":"2024-03-24T12:41:08.221588+0000","last_undegraded":"2024-03-24T12:41:08.221588+0000","last_fullsized":"2024-03-24T12:35:58.699613+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:23.095706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:33:23.095706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.008147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"207'7","reported_seq":512,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221226+0000","last_change":"2024-03-24T12:39:57.631665+0000","last_active":"2024-03-24T12:41:08.221226+0000","last_peered":"2024-03-24T12:41:08.221226+0000","last_clean":"2024-03-24T12:41:08.221226+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:41:08.221226+0000","last_undegraded":"2024-03-24T12:41:08.221226+0000","last_fullsized":"2024-03-24T12:41:08.221226+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-24T12:39:57.631577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:39:57.631577+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:24.095479+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":497,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.222006+0000","last_change":"2024-03-24T12:39:41.822559+0000","last_active":"2024-03-24T12:41:08.222006+0000","last_peered":"2024-03-24T12:41:08.222006+0000","last_clean":"2024-03-24T12:41:08.222006+0000","last_became_active":"2024-03-24T12:39:35.101660+0000","last_became_peered":"2024-03-24T12:39:35.101660+0000","last_unstale":"2024-03-24T12:41:08.222006+0000","last_undegraded":"2024-03-24T12:41:08.222006+0000","last_fullsized":"2024-03-24T12:41:08.222006+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:41.822379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:39:41.822379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,3],"avail_no_missing":["4","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":511,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.220252+0000","last_change":"2024-03-24T12:40:15.270665+0000","last_active":"2024-03-24T12:41:08.220252+0000","last_peered":"2024-03-24T12:41:08.220252+0000","last_clean":"2024-03-24T12:41:08.220252+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:41:08.220252+0000","last_undegraded":"2024-03-24T12:41:08.220252+0000","last_fullsized":"2024-03-24T12:41:08.220252+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:15.270454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:15.270454+0000","last_clean_scrub_stamp":"2024-03-24T12:40:15.270454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:43.532830+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":503,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221994+0000","last_change":"2024-03-24T12:40:58.334350+0000","last_active":"2024-03-24T12:41:08.221994+0000","last_peered":"2024-03-24T12:41:08.221994+0000","last_clean":"2024-03-24T12:41:08.221994+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:41:08.221994+0000","last_undegraded":"2024-03-24T12:41:08.221994+0000","last_fullsized":"2024-03-24T12:41:08.221994+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:58.334266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:40:58.334266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:21.984874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"207'27","reported_seq":542,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.226040+0000","last_change":"2024-03-24T12:39:42.561825+0000","last_active":"2024-03-24T12:41:08.226040+0000","last_peered":"2024-03-24T12:41:08.226040+0000","last_clean":"2024-03-24T12:41:08.226040+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:41:08.226040+0000","last_undegraded":"2024-03-24T12:41:08.226040+0000","last_fullsized":"2024-03-24T12:41:08.226040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-24T12:39:42.561693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:39:42.561693+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.995842+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":489,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.231228+0000","last_change":"2024-03-24T12:39:59.167526+0000","last_active":"2024-03-24T12:41:08.231228+0000","last_peered":"2024-03-24T12:41:08.231228+0000","last_clean":"2024-03-24T12:41:08.231228+0000","last_became_active":"2024-03-24T12:39:35.129797+0000","last_became_peered":"2024-03-24T12:39:35.129797+0000","last_unstale":"2024-03-24T12:41:08.231228+0000","last_undegraded":"2024-03-24T12:41:08.231228+0000","last_fullsized":"2024-03-24T12:41:08.231228+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:59.167308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:39:59.167308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:22.815435+0000","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":495,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222070+0000","last_change":"2024-03-24T12:39:55.736131+0000","last_active":"2024-03-24T12:41:08.222070+0000","last_peered":"2024-03-24T12:41:08.222070+0000","last_clean":"2024-03-24T12:41:08.222070+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:41:08.222070+0000","last_undegraded":"2024-03-24T12:41:08.222070+0000","last_fullsized":"2024-03-24T12:41:08.222070+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:55.735803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:39:55.735803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":503,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222495+0000","last_change":"2024-03-24T12:40:02.712200+0000","last_active":"2024-03-24T12:41:08.222495+0000","last_peered":"2024-03-24T12:41:08.222495+0000","last_clean":"2024-03-24T12:41:08.222495+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:41:08.222495+0000","last_undegraded":"2024-03-24T12:41:08.222495+0000","last_fullsized":"2024-03-24T12:41:08.222495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:02.712081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:02.712081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:28.366302+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"207'18","reported_seq":515,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.231155+0000","last_change":"2024-03-24T12:37:34.887625+0000","last_active":"2024-03-24T12:41:08.231155+0000","last_peered":"2024-03-24T12:41:08.231155+0000","last_clean":"2024-03-24T12:41:08.231155+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:41:08.231155+0000","last_undegraded":"2024-03-24T12:41:08.231155+0000","last_fullsized":"2024-03-24T12:41:08.231155+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-24T12:37:34.887491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:34.887491+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:13.832694+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":471,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:08.225012+0000","last_change":"2024-03-24T12:35:58.711430+0000","last_active":"2024-03-24T12:41:08.225012+0000","last_peered":"2024-03-24T12:41:08.225012+0000","last_clean":"2024-03-24T12:35:54.688013+0000","last_became_active":"2024-03-24T12:35:58.711430+0000","last_became_peered":"2024-03-24T12:35:58.711430+0000","last_unstale":"2024-03-24T12:41:08.225012+0000","last_undegraded":"2024-03-24T12:35:58.697902+0000","last_fullsized":"2024-03-24T12:35:58.697784+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:34:42.820487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:42.820487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"258'10","reported_seq":106,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:13.388815+0000","last_change":"2024-03-24T12:40:17.661441+0000","last_active":"2024-03-24T12:41:13.388815+0000","last_peered":"2024-03-24T12:41:13.388815+0000","last_clean":"2024-03-24T12:41:13.388815+0000","last_became_active":"2024-03-24T12:39:34.750827+0000","last_became_peered":"2024-03-24T12:39:34.750827+0000","last_unstale":"2024-03-24T12:41:13.388815+0000","last_undegraded":"2024-03-24T12:41:13.388815+0000","last_fullsized":"2024-03-24T12:41:13.388815+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-24T12:40:17.661312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:40:17.661312+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:43.427225+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":512,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.219417+0000","last_change":"2024-03-24T12:40:06.389505+0000","last_active":"2024-03-24T12:41:08.219417+0000","last_peered":"2024-03-24T12:41:08.219417+0000","last_clean":"2024-03-24T12:41:08.219417+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:41:08.219417+0000","last_undegraded":"2024-03-24T12:41:08.219417+0000","last_fullsized":"2024-03-24T12:41:08.219417+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:40:06.381315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:06.381315+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:27.137096+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"207'20","reported_seq":530,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222714+0000","last_change":"2024-03-24T12:40:02.428955+0000","last_active":"2024-03-24T12:41:08.222714+0000","last_peered":"2024-03-24T12:41:08.222714+0000","last_clean":"2024-03-24T12:41:08.222714+0000","last_became_active":"2024-03-24T12:26:05.359370+0000","last_became_peered":"2024-03-24T12:26:05.359370+0000","last_unstale":"2024-03-24T12:41:08.222714+0000","last_undegraded":"2024-03-24T12:41:08.222714+0000","last_fullsized":"2024-03-24T12:41:08.222714+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:40:02.428792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:02.428792+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:12.034836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":392,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.221187+0000","last_change":"2024-03-24T12:39:07.748878+0000","last_active":"2024-03-24T12:41:08.221187+0000","last_peered":"2024-03-24T12:41:08.221187+0000","last_clean":"2024-03-24T12:35:54.688035+0000","last_became_active":"2024-03-24T12:39:07.748878+0000","last_became_peered":"2024-03-24T12:39:07.748878+0000","last_unstale":"2024-03-24T12:41:08.221187+0000","last_undegraded":"2024-03-24T12:41:08.221187+0000","last_fullsized":"2024-03-24T12:39:07.746317+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:03.555975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.555975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.008134+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":488,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.223018+0000","last_change":"2024-03-24T12:40:16.406432+0000","last_active":"2024-03-24T12:41:08.223018+0000","last_peered":"2024-03-24T12:41:08.223018+0000","last_clean":"2024-03-24T12:41:08.223018+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:41:08.223018+0000","last_undegraded":"2024-03-24T12:41:08.223018+0000","last_fullsized":"2024-03-24T12:41:08.223018+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:16.406313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:40:16.406313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:23.556953+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.228313+0000","last_change":"2024-03-24T12:37:08.569047+0000","last_active":"2024-03-24T12:41:08.228313+0000","last_peered":"2024-03-24T12:41:08.228313+0000","last_clean":"2024-03-24T12:41:08.228313+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:41:08.228313+0000","last_undegraded":"2024-03-24T12:41:08.228313+0000","last_fullsized":"2024-03-24T12:41:08.228313+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:08.568803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:37:08.568803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"207'19","reported_seq":485,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.226097+0000","last_change":"2024-03-24T12:39:57.036943+0000","last_active":"2024-03-24T12:41:08.226097+0000","last_peered":"2024-03-24T12:41:08.226097+0000","last_clean":"2024-03-24T12:41:08.226097+0000","last_became_active":"2024-03-24T12:39:35.107173+0000","last_became_peered":"2024-03-24T12:39:35.107173+0000","last_unstale":"2024-03-24T12:41:08.226097+0000","last_undegraded":"2024-03-24T12:41:08.226097+0000","last_fullsized":"2024-03-24T12:41:08.226097+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:39:57.036691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:39:57.036691+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.917902+0000","scrub_duration":237,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1,0],"avail_no_missing":["6","0","1"],"object_location_counts":[{"shards":"0,1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":491,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.223234+0000","last_change":"2024-03-24T12:40:55.950369+0000","last_active":"2024-03-24T12:41:08.223234+0000","last_peered":"2024-03-24T12:41:08.223234+0000","last_clean":"2024-03-24T12:41:08.223234+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:41:08.223234+0000","last_undegraded":"2024-03-24T12:41:08.223234+0000","last_fullsized":"2024-03-24T12:41:08.223234+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:55.950261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:40:55.950261+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:02.340169+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":81,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.224124+0000","last_change":"2024-03-24T12:40:11.765558+0000","last_active":"2024-03-24T12:41:08.224124+0000","last_peered":"2024-03-24T12:41:08.224124+0000","last_clean":"2024-03-24T12:41:08.224124+0000","last_became_active":"2024-03-24T12:39:34.751127+0000","last_became_peered":"2024-03-24T12:39:34.751127+0000","last_unstale":"2024-03-24T12:41:08.224124+0000","last_undegraded":"2024-03-24T12:41:08.224124+0000","last_fullsized":"2024-03-24T12:41:08.224124+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:11.765447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:40:11.765447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:13.202230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.222571+0000","last_change":"2024-03-24T12:35:58.718236+0000","last_active":"2024-03-24T12:41:08.222571+0000","last_peered":"2024-03-24T12:41:08.222571+0000","last_clean":"2024-03-24T12:35:54.683418+0000","last_became_active":"2024-03-24T12:35:58.718236+0000","last_became_peered":"2024-03-24T12:35:58.718236+0000","last_unstale":"2024-03-24T12:41:08.222571+0000","last_undegraded":"2024-03-24T12:41:08.222571+0000","last_fullsized":"2024-03-24T12:35:58.708687+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.720633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.720633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.359863+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"207'8","reported_seq":527,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.228338+0000","last_change":"2024-03-24T12:40:12.127815+0000","last_active":"2024-03-24T12:41:08.228338+0000","last_peered":"2024-03-24T12:41:08.228338+0000","last_clean":"2024-03-24T12:41:08.228338+0000","last_became_active":"2024-03-24T12:39:34.759619+0000","last_became_peered":"2024-03-24T12:39:34.759619+0000","last_unstale":"2024-03-24T12:41:08.228338+0000","last_undegraded":"2024-03-24T12:41:08.228338+0000","last_fullsized":"2024-03-24T12:41:08.228338+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-24T12:40:12.127602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:12.127602+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:27.114369+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.230505+0000","last_change":"2024-03-24T12:39:34.763955+0000","last_active":"2024-03-24T12:41:08.230505+0000","last_peered":"2024-03-24T12:41:08.230505+0000","last_clean":"2024-03-24T12:41:08.230505+0000","last_became_active":"2024-03-24T12:39:34.763422+0000","last_became_peered":"2024-03-24T12:39:34.763422+0000","last_unstale":"2024-03-24T12:41:08.230505+0000","last_undegraded":"2024-03-24T12:41:08.230505+0000","last_fullsized":"2024-03-24T12:41:08.230505+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:21.837500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:37:21.837500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":507,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.223591+0000","last_change":"2024-03-24T12:40:07.041441+0000","last_active":"2024-03-24T12:41:08.223591+0000","last_peered":"2024-03-24T12:41:08.223591+0000","last_clean":"2024-03-24T12:41:08.223591+0000","last_became_active":"2024-03-24T12:39:34.752397+0000","last_became_peered":"2024-03-24T12:39:34.752397+0000","last_unstale":"2024-03-24T12:41:08.223591+0000","last_undegraded":"2024-03-24T12:41:08.223591+0000","last_fullsized":"2024-03-24T12:41:08.223591+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:07.041342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:40:07.041342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:33.612495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":181,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:08.225064+0000","last_change":"2024-03-24T12:35:58.708757+0000","last_active":"2024-03-24T12:41:08.225064+0000","last_peered":"2024-03-24T12:41:08.225064+0000","last_clean":"2024-03-24T12:26:03.166952+0000","last_became_active":"2024-03-24T12:35:58.708757+0000","last_became_peered":"2024-03-24T12:35:58.708757+0000","last_unstale":"2024-03-24T12:41:08.225064+0000","last_undegraded":"2024-03-24T12:35:58.698998+0000","last_fullsized":"2024-03-24T12:35:58.698185+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:28.059288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.059288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"207'19","reported_seq":536,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.229046+0000","last_change":"2024-03-24T12:40:54.825783+0000","last_active":"2024-03-24T12:41:08.229046+0000","last_peered":"2024-03-24T12:41:08.229046+0000","last_clean":"2024-03-24T12:41:08.229046+0000","last_became_active":"2024-03-24T12:39:34.750031+0000","last_became_peered":"2024-03-24T12:39:34.750031+0000","last_unstale":"2024-03-24T12:41:08.229046+0000","last_undegraded":"2024-03-24T12:41:08.229046+0000","last_fullsized":"2024-03-24T12:41:08.229046+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:40:54.825683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:54.825683+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:21.163089+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":470,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.218435+0000","last_change":"2024-03-24T12:35:58.706939+0000","last_active":"2024-03-24T12:41:08.218435+0000","last_peered":"2024-03-24T12:41:08.218435+0000","last_clean":"2024-03-24T12:35:54.681672+0000","last_became_active":"2024-03-24T12:35:58.706939+0000","last_became_peered":"2024-03-24T12:35:58.706939+0000","last_unstale":"2024-03-24T12:41:08.218435+0000","last_undegraded":"2024-03-24T12:41:08.218435+0000","last_fullsized":"2024-03-24T12:35:58.695715+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:53.895902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.895902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.359817+0000","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.230340+0000","last_change":"2024-03-24T12:41:02.921331+0000","last_active":"2024-03-24T12:41:08.230340+0000","last_peered":"2024-03-24T12:41:08.230340+0000","last_clean":"2024-03-24T12:41:08.230340+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:41:08.230340+0000","last_undegraded":"2024-03-24T12:41:08.230340+0000","last_fullsized":"2024-03-24T12:41:08.230340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:41:02.921251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_clean_scrub_stamp":"2024-03-24T12:41:02.921251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:23.241928+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":503,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.224246+0000","last_change":"2024-03-24T12:40:12.077023+0000","last_active":"2024-03-24T12:41:08.224246+0000","last_peered":"2024-03-24T12:41:08.224246+0000","last_clean":"2024-03-24T12:41:08.224246+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:41:08.224246+0000","last_undegraded":"2024-03-24T12:41:08.224246+0000","last_fullsized":"2024-03-24T12:41:08.224246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:12.076929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:12.076929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:35.722244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"207'15","reported_seq":184,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.224614+0000","last_change":"2024-03-24T12:39:58.840014+0000","last_active":"2024-03-24T12:41:08.224614+0000","last_peered":"2024-03-24T12:41:08.224614+0000","last_clean":"2024-03-24T12:41:08.224614+0000","last_became_active":"2024-03-24T12:39:35.103739+0000","last_became_peered":"2024-03-24T12:39:35.103739+0000","last_unstale":"2024-03-24T12:41:08.224614+0000","last_undegraded":"2024-03-24T12:41:08.224614+0000","last_fullsized":"2024-03-24T12:41:08.224614+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:39:58.839764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:39:58.839764+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:16.355017+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2,0],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":470,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.228605+0000","last_change":"2024-03-24T12:35:58.714875+0000","last_active":"2024-03-24T12:41:08.228605+0000","last_peered":"2024-03-24T12:41:08.228605+0000","last_clean":"2024-03-24T12:35:54.689143+0000","last_became_active":"2024-03-24T12:35:58.714875+0000","last_became_peered":"2024-03-24T12:35:58.714875+0000","last_unstale":"2024-03-24T12:41:08.228605+0000","last_undegraded":"2024-03-24T12:41:08.228605+0000","last_fullsized":"2024-03-24T12:35:58.702457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.898988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.898988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.222572+0000","last_change":"2024-03-24T12:35:58.718562+0000","last_active":"2024-03-24T12:41:08.222572+0000","last_peered":"2024-03-24T12:41:08.222572+0000","last_clean":"2024-03-24T12:35:54.687557+0000","last_became_active":"2024-03-24T12:35:58.718562+0000","last_became_peered":"2024-03-24T12:35:58.718562+0000","last_unstale":"2024-03-24T12:41:08.222572+0000","last_undegraded":"2024-03-24T12:41:08.222572+0000","last_fullsized":"2024-03-24T12:35:58.697301+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.144883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.144883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:10.331081+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":515,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221773+0000","last_change":"2024-03-24T12:39:34.748732+0000","last_active":"2024-03-24T12:41:08.221773+0000","last_peered":"2024-03-24T12:41:08.221773+0000","last_clean":"2024-03-24T12:41:08.221773+0000","last_became_active":"2024-03-24T12:39:34.748131+0000","last_became_peered":"2024-03-24T12:39:34.748131+0000","last_unstale":"2024-03-24T12:41:08.221773+0000","last_undegraded":"2024-03-24T12:41:08.221773+0000","last_fullsized":"2024-03-24T12:41:08.221773+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:36.815404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:36.815404+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"207'24","reported_seq":553,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221448+0000","last_change":"2024-03-24T12:39:54.714731+0000","last_active":"2024-03-24T12:41:08.221448+0000","last_peered":"2024-03-24T12:41:08.221448+0000","last_clean":"2024-03-24T12:41:08.221448+0000","last_became_active":"2024-03-24T12:39:35.101521+0000","last_became_peered":"2024-03-24T12:39:35.101521+0000","last_unstale":"2024-03-24T12:41:08.221448+0000","last_undegraded":"2024-03-24T12:41:08.221448+0000","last_fullsized":"2024-03-24T12:41:08.221448+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-24T12:39:54.714604+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:39:54.714604+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":484,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222447+0000","last_change":"2024-03-24T12:39:55.286760+0000","last_active":"2024-03-24T12:41:08.222447+0000","last_peered":"2024-03-24T12:41:08.222447+0000","last_clean":"2024-03-24T12:41:08.222447+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:41:08.222447+0000","last_undegraded":"2024-03-24T12:41:08.222447+0000","last_fullsized":"2024-03-24T12:41:08.222447+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:39:55.286634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:39:55.286634+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:08.748549+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":227,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:13.388842+0000","last_change":"2024-03-24T12:39:53.260382+0000","last_active":"2024-03-24T12:41:13.388842+0000","last_peered":"2024-03-24T12:41:13.388842+0000","last_clean":"2024-03-24T12:41:13.388842+0000","last_became_active":"2024-03-24T12:36:52.065482+0000","last_became_peered":"2024-03-24T12:36:52.065482+0000","last_unstale":"2024-03-24T12:41:13.388842+0000","last_undegraded":"2024-03-24T12:41:13.388842+0000","last_fullsized":"2024-03-24T12:41:13.388842+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-24T12:39:53.260292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:39:53.260292+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:21.822608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":503,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.223597+0000","last_change":"2024-03-24T12:40:56.893523+0000","last_active":"2024-03-24T12:41:08.223597+0000","last_peered":"2024-03-24T12:41:08.223597+0000","last_clean":"2024-03-24T12:41:08.223597+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:41:08.223597+0000","last_undegraded":"2024-03-24T12:41:08.223597+0000","last_fullsized":"2024-03-24T12:41:08.223597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:56.893413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:56.893413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:19.007919+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"207'20","reported_seq":521,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.230617+0000","last_change":"2024-03-24T12:37:27.793372+0000","last_active":"2024-03-24T12:41:08.230617+0000","last_peered":"2024-03-24T12:41:08.230617+0000","last_clean":"2024-03-24T12:41:08.230617+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:41:08.230617+0000","last_undegraded":"2024-03-24T12:41:08.230617+0000","last_fullsized":"2024-03-24T12:41:08.230617+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:37:27.793250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:27.793250+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":478,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.221996+0000","last_change":"2024-03-24T12:35:58.712757+0000","last_active":"2024-03-24T12:41:08.221996+0000","last_peered":"2024-03-24T12:41:08.221996+0000","last_clean":"2024-03-24T12:35:54.677408+0000","last_became_active":"2024-03-24T12:35:58.712757+0000","last_became_peered":"2024-03-24T12:35:58.712757+0000","last_unstale":"2024-03-24T12:41:08.221996+0000","last_undegraded":"2024-03-24T12:41:08.221996+0000","last_fullsized":"2024-03-24T12:35:58.700636+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:01.913287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:01.913287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.008168+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"93'9","reported_seq":672,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:13.388815+0000","last_change":"2024-03-24T12:37:32.841507+0000","last_active":"2024-03-24T12:41:13.388815+0000","last_peered":"2024-03-24T12:41:13.388815+0000","last_clean":"2024-03-24T12:41:13.388815+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:41:13.388815+0000","last_undegraded":"2024-03-24T12:41:13.388815+0000","last_fullsized":"2024-03-24T12:41:13.388815+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'9","last_scrub_stamp":"2024-03-24T12:37:32.840824+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:37:32.840824+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:21.806217+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":503,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221079+0000","last_change":"2024-03-24T12:40:53.346285+0000","last_active":"2024-03-24T12:41:08.221079+0000","last_peered":"2024-03-24T12:41:08.221079+0000","last_clean":"2024-03-24T12:41:08.221079+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:41:08.221079+0000","last_undegraded":"2024-03-24T12:41:08.221079+0000","last_fullsized":"2024-03-24T12:41:08.221079+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:53.346202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:53.346202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:11.595212+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"207'19","reported_seq":250,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:08.224627+0000","last_change":"2024-03-24T12:35:58.751467+0000","last_active":"2024-03-24T12:41:08.224627+0000","last_peered":"2024-03-24T12:41:08.224627+0000","last_clean":"2024-03-24T12:27:16.240812+0000","last_became_active":"2024-03-24T12:35:58.751467+0000","last_became_peered":"2024-03-24T12:35:58.751467+0000","last_unstale":"2024-03-24T12:41:08.224627+0000","last_undegraded":"2024-03-24T12:35:58.700376+0000","last_fullsized":"2024-03-24T12:35:58.700265+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:35:08.002032+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.002032+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":499,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.226310+0000","last_change":"2024-03-24T12:40:01.551926+0000","last_active":"2024-03-24T12:41:08.226310+0000","last_peered":"2024-03-24T12:41:08.226310+0000","last_clean":"2024-03-24T12:41:08.226310+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:41:08.226310+0000","last_undegraded":"2024-03-24T12:41:08.226310+0000","last_fullsized":"2024-03-24T12:41:08.226310+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:01.551813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:40:01.551813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:11.978491+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":479,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.231138+0000","last_change":"2024-03-24T12:37:48.933827+0000","last_active":"2024-03-24T12:41:08.231138+0000","last_peered":"2024-03-24T12:41:08.231138+0000","last_clean":"2024-03-24T12:41:08.231138+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:41:08.231138+0000","last_undegraded":"2024-03-24T12:41:08.231138+0000","last_fullsized":"2024-03-24T12:41:08.231138+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:48.933558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:37:48.933558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":516,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.223898+0000","last_change":"2024-03-24T12:39:46.372339+0000","last_active":"2024-03-24T12:41:08.223898+0000","last_peered":"2024-03-24T12:41:08.223898+0000","last_clean":"2024-03-24T12:41:08.223898+0000","last_became_active":"2024-03-24T12:39:34.755495+0000","last_became_peered":"2024-03-24T12:39:34.755495+0000","last_unstale":"2024-03-24T12:41:08.223898+0000","last_undegraded":"2024-03-24T12:41:08.223898+0000","last_fullsized":"2024-03-24T12:41:08.223898+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:46.372265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:39:46.372265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:13.468656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"207'15","reported_seq":524,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222815+0000","last_change":"2024-03-24T12:40:14.414330+0000","last_active":"2024-03-24T12:41:08.222815+0000","last_peered":"2024-03-24T12:41:08.222815+0000","last_clean":"2024-03-24T12:41:08.222815+0000","last_became_active":"2024-03-24T12:26:05.357741+0000","last_became_peered":"2024-03-24T12:26:05.357741+0000","last_unstale":"2024-03-24T12:41:08.222815+0000","last_undegraded":"2024-03-24T12:41:08.222815+0000","last_fullsized":"2024-03-24T12:41:08.222815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:40:14.414177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:14.414177+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:19.871005+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":493,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.224154+0000","last_change":"2024-03-24T12:39:55.271843+0000","last_active":"2024-03-24T12:41:08.224154+0000","last_peered":"2024-03-24T12:41:08.224154+0000","last_clean":"2024-03-24T12:41:08.224154+0000","last_became_active":"2024-03-24T12:39:35.100242+0000","last_became_peered":"2024-03-24T12:39:35.100242+0000","last_unstale":"2024-03-24T12:41:08.224154+0000","last_undegraded":"2024-03-24T12:41:08.224154+0000","last_fullsized":"2024-03-24T12:41:08.224154+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:55.271679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:55.271679+0000","last_clean_scrub_stamp":"2024-03-24T12:39:55.271679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":174,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.230325+0000","last_change":"2024-03-24T12:35:58.714488+0000","last_active":"2024-03-24T12:41:08.230325+0000","last_peered":"2024-03-24T12:41:08.230325+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.714488+0000","last_became_peered":"2024-03-24T12:35:58.714488+0000","last_unstale":"2024-03-24T12:41:08.230325+0000","last_undegraded":"2024-03-24T12:41:08.230325+0000","last_fullsized":"2024-03-24T12:35:58.700893+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.995618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.995618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":458,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.225263+0000","last_change":"2024-03-24T12:40:08.835387+0000","last_active":"2024-03-24T12:41:08.225263+0000","last_peered":"2024-03-24T12:41:08.225263+0000","last_clean":"2024-03-24T12:41:08.225263+0000","last_became_active":"2024-03-24T12:39:35.120926+0000","last_became_peered":"2024-03-24T12:39:35.120926+0000","last_unstale":"2024-03-24T12:41:08.225263+0000","last_undegraded":"2024-03-24T12:41:08.225263+0000","last_fullsized":"2024-03-24T12:41:08.225263+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:08.835186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:08.835186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:30.126922+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,3],"avail_no_missing":["6","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"207'12","reported_seq":501,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.231315+0000","last_change":"2024-03-24T12:37:25.779784+0000","last_active":"2024-03-24T12:41:08.231315+0000","last_peered":"2024-03-24T12:41:08.231315+0000","last_clean":"2024-03-24T12:41:08.231315+0000","last_became_active":"2024-03-24T12:26:05.356892+0000","last_became_peered":"2024-03-24T12:26:05.356892+0000","last_unstale":"2024-03-24T12:41:08.231315+0000","last_undegraded":"2024-03-24T12:41:08.231315+0000","last_fullsized":"2024-03-24T12:41:08.231315+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-24T12:37:25.779285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:25.779285+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"286'2350","reported_seq":2372,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:21.415652+0000","last_change":"2024-03-24T12:41:04.289719+0000","last_active":"2024-03-24T12:41:21.415652+0000","last_peered":"2024-03-24T12:41:21.415652+0000","last_clean":"2024-03-24T12:40:50.114364+0000","last_became_active":"2024-03-24T12:40:51.131277+0000","last_became_peered":"2024-03-24T12:40:51.131277+0000","last_unstale":"2024-03-24T12:41:21.415652+0000","last_undegraded":"2024-03-24T12:41:04.288099+0000","last_fullsized":"2024-03-24T12:40:51.129259+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2350,"log_dups_size":0,"ondisk_log_size":2350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:07.900684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154009600,"num_objects":2350,"num_object_clones":0,"num_object_copies":7050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2350,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2350,"num_write_kb":150400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":2350}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":140,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.224372+0000","last_change":"2024-03-24T12:39:53.754810+0000","last_active":"2024-03-24T12:41:08.224372+0000","last_peered":"2024-03-24T12:41:08.224372+0000","last_clean":"2024-03-24T12:41:08.224372+0000","last_became_active":"2024-03-24T12:39:35.103777+0000","last_became_peered":"2024-03-24T12:39:35.103777+0000","last_unstale":"2024-03-24T12:41:08.224372+0000","last_undegraded":"2024-03-24T12:41:08.224372+0000","last_fullsized":"2024-03-24T12:41:08.224372+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:53.754669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:39:53.754669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:10.570711+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,2],"avail_no_missing":["6","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":507,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.225732+0000","last_change":"2024-03-24T12:37:57.274231+0000","last_active":"2024-03-24T12:41:08.225732+0000","last_peered":"2024-03-24T12:41:08.225732+0000","last_clean":"2024-03-24T12:41:08.225732+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:41:08.225732+0000","last_undegraded":"2024-03-24T12:41:08.225732+0000","last_fullsized":"2024-03-24T12:41:08.225732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:37:57.274142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:57.274142+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"207'27","reported_seq":269,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.223418+0000","last_change":"2024-03-24T12:39:42.354983+0000","last_active":"2024-03-24T12:41:08.223418+0000","last_peered":"2024-03-24T12:41:08.223418+0000","last_clean":"2024-03-24T12:41:08.223418+0000","last_became_active":"2024-03-24T12:39:35.103338+0000","last_became_peered":"2024-03-24T12:39:35.103338+0000","last_unstale":"2024-03-24T12:41:08.223418+0000","last_undegraded":"2024-03-24T12:41:08.223418+0000","last_fullsized":"2024-03-24T12:41:08.223418+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-24T12:39:42.354720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:39:42.354720+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:08.243487+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,4],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":7}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"258'10","reported_seq":711,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:18.390478+0000","last_change":"2024-03-24T12:39:34.759270+0000","last_active":"2024-03-24T12:41:18.390478+0000","last_peered":"2024-03-24T12:41:18.390478+0000","last_clean":"2024-03-24T12:41:18.390478+0000","last_became_active":"2024-03-24T12:39:34.753277+0000","last_became_peered":"2024-03-24T12:39:34.753277+0000","last_unstale":"2024-03-24T12:41:18.390478+0000","last_undegraded":"2024-03-24T12:41:18.390478+0000","last_fullsized":"2024-03-24T12:41:18.390478+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:38:01.284422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:38:01.284422+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"286'2691","reported_seq":2714,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:22.619393+0000","last_change":"2024-03-24T12:41:04.285775+0000","last_active":"2024-03-24T12:41:22.619393+0000","last_peered":"2024-03-24T12:41:22.619393+0000","last_clean":"2024-03-24T12:40:50.114364+0000","last_became_active":"2024-03-24T12:40:51.398233+0000","last_became_peered":"2024-03-24T12:40:51.398233+0000","last_unstale":"2024-03-24T12:41:22.619393+0000","last_undegraded":"2024-03-24T12:41:04.285284+0000","last_fullsized":"2024-03-24T12:40:51.133672+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2691,"log_dups_size":0,"ondisk_log_size":2691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:59.224971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176357376,"num_objects":2691,"num_object_clones":0,"num_object_copies":8073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2691,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2691,"num_write_kb":172224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":2691}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":475,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.230432+0000","last_change":"2024-03-24T12:37:51.992994+0000","last_active":"2024-03-24T12:41:08.230432+0000","last_peered":"2024-03-24T12:41:08.230432+0000","last_clean":"2024-03-24T12:41:08.230432+0000","last_became_active":"2024-03-24T12:26:09.350386+0000","last_became_peered":"2024-03-24T12:26:09.350386+0000","last_unstale":"2024-03-24T12:41:08.230432+0000","last_undegraded":"2024-03-24T12:41:08.230432+0000","last_fullsized":"2024-03-24T12:41:08.230432+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:51.992860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:51.992860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":495,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.225256+0000","last_change":"2024-03-24T12:39:48.568995+0000","last_active":"2024-03-24T12:41:08.225256+0000","last_peered":"2024-03-24T12:41:08.225256+0000","last_clean":"2024-03-24T12:41:08.225256+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:41:08.225256+0000","last_undegraded":"2024-03-24T12:41:08.225256+0000","last_fullsized":"2024-03-24T12:41:08.225256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:48.568885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:39:48.568885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":194,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"207'23","reported_seq":545,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221987+0000","last_change":"2024-03-24T12:41:04.012667+0000","last_active":"2024-03-24T12:41:08.221987+0000","last_peered":"2024-03-24T12:41:08.221987+0000","last_clean":"2024-03-24T12:41:08.221987+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:41:08.221987+0000","last_undegraded":"2024-03-24T12:41:08.221987+0000","last_fullsized":"2024-03-24T12:41:08.221987+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-24T12:41:04.012539+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:41:04.012539+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:08.438837+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":504,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.225282+0000","last_change":"2024-03-24T12:40:51.345722+0000","last_active":"2024-03-24T12:41:08.225282+0000","last_peered":"2024-03-24T12:41:08.225282+0000","last_clean":"2024-03-24T12:41:08.225282+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:41:08.225282+0000","last_undegraded":"2024-03-24T12:41:08.225282+0000","last_fullsized":"2024-03-24T12:41:08.225282+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:51.345640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:40:51.345640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:09.559730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"286'2372","reported_seq":2398,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:21.390845+0000","last_change":"2024-03-24T12:40:51.397934+0000","last_active":"2024-03-24T12:41:21.390845+0000","last_peered":"2024-03-24T12:41:21.390845+0000","last_clean":"2024-03-24T12:41:21.390845+0000","last_became_active":"2024-03-24T12:40:51.397617+0000","last_became_peered":"2024-03-24T12:40:51.397617+0000","last_unstale":"2024-03-24T12:41:21.390845+0000","last_undegraded":"2024-03-24T12:41:21.390845+0000","last_fullsized":"2024-03-24T12:41:21.390845+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2372,"log_dups_size":0,"ondisk_log_size":2372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:52.990421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155451392,"num_objects":2372,"num_object_clones":0,"num_object_copies":7116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2372,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2372,"num_write_kb":151808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":492,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.223035+0000","last_change":"2024-03-24T12:40:20.907558+0000","last_active":"2024-03-24T12:41:08.223035+0000","last_peered":"2024-03-24T12:41:08.223035+0000","last_clean":"2024-03-24T12:41:08.223035+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:41:08.223035+0000","last_undegraded":"2024-03-24T12:41:08.223035+0000","last_fullsized":"2024-03-24T12:41:08.223035+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:20.907484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:40:20.907484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:31.090910+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":482,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.225674+0000","last_change":"2024-03-24T12:35:58.711865+0000","last_active":"2024-03-24T12:41:08.225674+0000","last_peered":"2024-03-24T12:41:08.225674+0000","last_clean":"2024-03-24T12:35:54.681353+0000","last_became_active":"2024-03-24T12:35:58.711865+0000","last_became_peered":"2024-03-24T12:35:58.711865+0000","last_unstale":"2024-03-24T12:41:08.225674+0000","last_undegraded":"2024-03-24T12:41:08.225674+0000","last_fullsized":"2024-03-24T12:35:58.701950+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:16.019255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.019255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:10.314715+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":500,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.224458+0000","last_change":"2024-03-24T12:39:45.374985+0000","last_active":"2024-03-24T12:41:08.224458+0000","last_peered":"2024-03-24T12:41:08.224458+0000","last_clean":"2024-03-24T12:41:08.224458+0000","last_became_active":"2024-03-24T12:39:35.119214+0000","last_became_peered":"2024-03-24T12:39:35.119214+0000","last_unstale":"2024-03-24T12:41:08.224458+0000","last_undegraded":"2024-03-24T12:41:08.224458+0000","last_fullsized":"2024-03-24T12:41:08.224458+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:45.374782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:39:45.374782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,3],"avail_no_missing":["7","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"207'35","reported_seq":522,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.225083+0000","last_change":"2024-03-24T12:40:09.821829+0000","last_active":"2024-03-24T12:41:08.225083+0000","last_peered":"2024-03-24T12:41:08.225083+0000","last_clean":"2024-03-24T12:41:08.225083+0000","last_became_active":"2024-03-24T12:39:35.101391+0000","last_became_peered":"2024-03-24T12:39:35.101391+0000","last_unstale":"2024-03-24T12:41:08.225083+0000","last_undegraded":"2024-03-24T12:41:08.225083+0000","last_fullsized":"2024-03-24T12:41:08.225083+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-24T12:40:09.821653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:09.821653+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:18.702710+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"286'2609","reported_seq":2635,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:24.120954+0000","last_change":"2024-03-24T12:40:51.397907+0000","last_active":"2024-03-24T12:41:24.120954+0000","last_peered":"2024-03-24T12:41:24.120954+0000","last_clean":"2024-03-24T12:41:24.120954+0000","last_became_active":"2024-03-24T12:40:51.397589+0000","last_became_peered":"2024-03-24T12:40:51.397589+0000","last_unstale":"2024-03-24T12:41:24.120954+0000","last_undegraded":"2024-03-24T12:41:24.120954+0000","last_fullsized":"2024-03-24T12:41:24.120954+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2609,"log_dups_size":0,"ondisk_log_size":2609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:18.045972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170983424,"num_objects":2609,"num_object_clones":0,"num_object_copies":7827,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2609,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2609,"num_write_kb":166976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":483,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222246+0000","last_change":"2024-03-24T12:39:50.292606+0000","last_active":"2024-03-24T12:41:08.222246+0000","last_peered":"2024-03-24T12:41:08.222246+0000","last_clean":"2024-03-24T12:41:08.222246+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:41:08.222246+0000","last_undegraded":"2024-03-24T12:41:08.222246+0000","last_fullsized":"2024-03-24T12:41:08.222246+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:50.292486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:39:50.292486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:17.458200+0000","scrub_duration":90,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":504,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.226811+0000","last_change":"2024-03-24T12:40:48.382866+0000","last_active":"2024-03-24T12:41:08.226811+0000","last_peered":"2024-03-24T12:41:08.226811+0000","last_clean":"2024-03-24T12:41:08.226811+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:41:08.226811+0000","last_undegraded":"2024-03-24T12:41:08.226811+0000","last_fullsized":"2024-03-24T12:41:08.226811+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:48.382782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:40:48.382782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:01.090962+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":253,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.230719+0000","last_change":"2024-03-24T12:40:03.059161+0000","last_active":"2024-03-24T12:41:08.230719+0000","last_peered":"2024-03-24T12:41:08.230719+0000","last_clean":"2024-03-24T12:41:08.230719+0000","last_became_active":"2024-03-24T12:39:35.119146+0000","last_became_peered":"2024-03-24T12:39:35.119146+0000","last_unstale":"2024-03-24T12:41:08.230719+0000","last_undegraded":"2024-03-24T12:41:08.230719+0000","last_fullsized":"2024-03-24T12:41:08.230719+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:40:03.054668+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:40:03.054668+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:26.013106+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"207'15","reported_seq":532,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221617+0000","last_change":"2024-03-24T12:40:10.443274+0000","last_active":"2024-03-24T12:41:08.221617+0000","last_peered":"2024-03-24T12:41:08.221617+0000","last_clean":"2024-03-24T12:41:08.221617+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:41:08.221617+0000","last_undegraded":"2024-03-24T12:41:08.221617+0000","last_fullsized":"2024-03-24T12:41:08.221617+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:40:10.443002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:10.443002+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:15.844600+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"286'2495","reported_seq":2518,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:22.600382+0000","last_change":"2024-03-24T12:41:04.302350+0000","last_active":"2024-03-24T12:41:22.600382+0000","last_peered":"2024-03-24T12:41:22.600382+0000","last_clean":"2024-03-24T12:40:50.114364+0000","last_became_active":"2024-03-24T12:40:51.136373+0000","last_became_peered":"2024-03-24T12:40:51.136373+0000","last_unstale":"2024-03-24T12:41:22.600382+0000","last_undegraded":"2024-03-24T12:41:04.286161+0000","last_fullsized":"2024-03-24T12:40:51.132279+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2495,"log_dups_size":0,"ondisk_log_size":2495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:55.128608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163512320,"num_objects":2495,"num_object_clones":0,"num_object_copies":7485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2495,"num_write_kb":159680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":2495}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":475,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.230351+0000","last_change":"2024-03-24T12:37:23.691297+0000","last_active":"2024-03-24T12:41:08.230351+0000","last_peered":"2024-03-24T12:41:08.230351+0000","last_clean":"2024-03-24T12:41:08.230351+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:41:08.230351+0000","last_undegraded":"2024-03-24T12:41:08.230351+0000","last_fullsized":"2024-03-24T12:41:08.230351+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:23.691167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:37:23.691167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":221,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.222214+0000","last_change":"2024-03-24T12:39:59.449897+0000","last_active":"2024-03-24T12:41:08.222214+0000","last_peered":"2024-03-24T12:41:08.222214+0000","last_clean":"2024-03-24T12:41:08.222214+0000","last_became_active":"2024-03-24T12:39:35.101653+0000","last_became_peered":"2024-03-24T12:39:35.101653+0000","last_unstale":"2024-03-24T12:41:08.222214+0000","last_undegraded":"2024-03-24T12:41:08.222214+0000","last_fullsized":"2024-03-24T12:41:08.222214+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:39:59.449468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:39:59.449468+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:22.966346+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":516,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221257+0000","last_change":"2024-03-24T12:40:18.803801+0000","last_active":"2024-03-24T12:41:08.221257+0000","last_peered":"2024-03-24T12:41:08.221257+0000","last_clean":"2024-03-24T12:41:08.221257+0000","last_became_active":"2024-03-24T12:39:34.751547+0000","last_became_peered":"2024-03-24T12:39:34.751547+0000","last_unstale":"2024-03-24T12:41:08.221257+0000","last_undegraded":"2024-03-24T12:41:08.221257+0000","last_fullsized":"2024-03-24T12:41:08.221257+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:18.803713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:18.803713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:30.848784+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"248'51","reported_seq":644,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:18.389745+0000","last_change":"2024-03-24T12:39:47.606697+0000","last_active":"2024-03-24T12:41:18.389745+0000","last_peered":"2024-03-24T12:41:18.389745+0000","last_clean":"2024-03-24T12:41:18.389745+0000","last_became_active":"2024-03-24T12:39:07.750351+0000","last_became_peered":"2024-03-24T12:39:07.750351+0000","last_unstale":"2024-03-24T12:41:18.389745+0000","last_undegraded":"2024-03-24T12:41:18.389745+0000","last_fullsized":"2024-03-24T12:41:18.389745+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-03-24T12:39:47.606538+0000","last_deep_scrub":"248'51","last_deep_scrub_stamp":"2024-03-24T12:39:47.606538+0000","last_clean_scrub_stamp":"2024-03-24T12:39:47.606538+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":488,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221819+0000","last_change":"2024-03-24T12:39:43.994532+0000","last_active":"2024-03-24T12:41:08.221819+0000","last_peered":"2024-03-24T12:41:08.221819+0000","last_clean":"2024-03-24T12:41:08.221819+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:41:08.221819+0000","last_undegraded":"2024-03-24T12:41:08.221819+0000","last_fullsized":"2024-03-24T12:41:08.221819+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:43.994243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:39:43.994243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":26,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"286'2504","reported_seq":2527,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:24.123345+0000","last_change":"2024-03-24T12:41:04.289686+0000","last_active":"2024-03-24T12:41:24.123345+0000","last_peered":"2024-03-24T12:41:24.123345+0000","last_clean":"2024-03-24T12:40:50.114364+0000","last_became_active":"2024-03-24T12:40:51.134872+0000","last_became_peered":"2024-03-24T12:40:51.134872+0000","last_unstale":"2024-03-24T12:41:24.123345+0000","last_undegraded":"2024-03-24T12:41:04.283226+0000","last_fullsized":"2024-03-24T12:40:51.131989+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2504,"log_dups_size":0,"ondisk_log_size":2504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:05.512536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164102144,"num_objects":2504,"num_object_clones":0,"num_object_copies":7512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2504,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2504,"num_write_kb":160256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":2504}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":512,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.224824+0000","last_change":"2024-03-24T12:40:15.087246+0000","last_active":"2024-03-24T12:41:08.224824+0000","last_peered":"2024-03-24T12:41:08.224824+0000","last_clean":"2024-03-24T12:41:08.224824+0000","last_became_active":"2024-03-24T12:39:34.755964+0000","last_became_peered":"2024-03-24T12:39:34.755964+0000","last_unstale":"2024-03-24T12:41:08.224824+0000","last_undegraded":"2024-03-24T12:41:08.224824+0000","last_fullsized":"2024-03-24T12:41:08.224824+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:15.087115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:15.087115+0000","last_clean_scrub_stamp":"2024-03-24T12:40:15.087115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:30.648616+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":497,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.225444+0000","last_change":"2024-03-24T12:40:00.549737+0000","last_active":"2024-03-24T12:41:08.225444+0000","last_peered":"2024-03-24T12:41:08.225444+0000","last_clean":"2024-03-24T12:41:08.225444+0000","last_became_active":"2024-03-24T12:39:34.744167+0000","last_became_peered":"2024-03-24T12:39:34.744167+0000","last_unstale":"2024-03-24T12:41:08.225444+0000","last_undegraded":"2024-03-24T12:41:08.225444+0000","last_fullsized":"2024-03-24T12:41:08.225444+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:00.549593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:00.549593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:12.018022+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":527,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.225813+0000","last_change":"2024-03-24T12:40:54.300045+0000","last_active":"2024-03-24T12:41:08.225813+0000","last_peered":"2024-03-24T12:41:08.225813+0000","last_clean":"2024-03-24T12:41:08.225813+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:41:08.225813+0000","last_undegraded":"2024-03-24T12:41:08.225813+0000","last_fullsized":"2024-03-24T12:41:08.225813+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:54.299961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:54.299961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:16.543322+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"207'22","reported_seq":525,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:08.220123+0000","last_change":"2024-03-24T12:35:58.709644+0000","last_active":"2024-03-24T12:41:08.220123+0000","last_peered":"2024-03-24T12:41:08.220123+0000","last_clean":"2024-03-24T12:35:54.687814+0000","last_became_active":"2024-03-24T12:35:58.709644+0000","last_became_peered":"2024-03-24T12:35:58.709644+0000","last_unstale":"2024-03-24T12:41:08.220123+0000","last_undegraded":"2024-03-24T12:35:58.700232+0000","last_fullsized":"2024-03-24T12:35:58.699170+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:33:02.509218+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:31:08.237387+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.509218+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:23.353772+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":483,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.227214+0000","last_change":"2024-03-24T12:35:58.703902+0000","last_active":"2024-03-24T12:41:08.227214+0000","last_peered":"2024-03-24T12:41:08.227214+0000","last_clean":"2024-03-24T12:35:54.680479+0000","last_became_active":"2024-03-24T12:35:58.703902+0000","last_became_peered":"2024-03-24T12:35:58.703902+0000","last_unstale":"2024-03-24T12:41:08.227214+0000","last_undegraded":"2024-03-24T12:41:08.227214+0000","last_fullsized":"2024-03-24T12:35:58.701457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:13.031712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:13.031712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"286'2374","reported_seq":2400,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:22.599961+0000","last_change":"2024-03-24T12:40:51.137225+0000","last_active":"2024-03-24T12:41:22.599961+0000","last_peered":"2024-03-24T12:41:22.599961+0000","last_clean":"2024-03-24T12:41:22.599961+0000","last_became_active":"2024-03-24T12:40:51.136705+0000","last_became_peered":"2024-03-24T12:40:51.136705+0000","last_unstale":"2024-03-24T12:41:22.599961+0000","last_undegraded":"2024-03-24T12:41:22.599961+0000","last_fullsized":"2024-03-24T12:41:22.599961+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2374,"log_dups_size":0,"ondisk_log_size":2374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:12.628181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155582464,"num_objects":2374,"num_object_clones":0,"num_object_copies":7122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2374,"num_write_kb":151936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":173,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.230948+0000","last_change":"2024-03-24T12:35:58.713785+0000","last_active":"2024-03-24T12:41:08.230948+0000","last_peered":"2024-03-24T12:41:08.230948+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.713785+0000","last_became_peered":"2024-03-24T12:35:58.713785+0000","last_unstale":"2024-03-24T12:41:08.230948+0000","last_undegraded":"2024-03-24T12:41:08.230948+0000","last_fullsized":"2024-03-24T12:35:58.700902+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:18.057286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:18.057286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":788,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.223257+0000","last_change":"2024-03-24T12:39:58.263949+0000","last_active":"2024-03-24T12:41:08.223257+0000","last_peered":"2024-03-24T12:41:08.223257+0000","last_clean":"2024-03-24T12:41:08.223257+0000","last_became_active":"2024-03-24T12:39:37.136501+0000","last_became_peered":"2024-03-24T12:39:37.136501+0000","last_unstale":"2024-03-24T12:41:08.223257+0000","last_undegraded":"2024-03-24T12:41:08.223257+0000","last_fullsized":"2024-03-24T12:41:08.223257+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":258,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:39:58.263855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:39:58.263855+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":501,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.229405+0000","last_change":"2024-03-24T12:39:53.978440+0000","last_active":"2024-03-24T12:41:08.229405+0000","last_peered":"2024-03-24T12:41:08.229405+0000","last_clean":"2024-03-24T12:41:08.229405+0000","last_became_active":"2024-03-24T12:39:35.128182+0000","last_became_peered":"2024-03-24T12:39:35.128182+0000","last_unstale":"2024-03-24T12:41:08.229405+0000","last_undegraded":"2024-03-24T12:41:08.229405+0000","last_fullsized":"2024-03-24T12:41:08.229405+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:53.978235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:39:53.978235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"258'30","reported_seq":120,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:13.389039+0000","last_change":"2024-03-24T12:40:16.701393+0000","last_active":"2024-03-24T12:41:13.389039+0000","last_peered":"2024-03-24T12:41:13.389039+0000","last_clean":"2024-03-24T12:41:13.389039+0000","last_became_active":"2024-03-24T12:39:34.753078+0000","last_became_peered":"2024-03-24T12:39:34.753078+0000","last_unstale":"2024-03-24T12:41:13.389039+0000","last_undegraded":"2024-03-24T12:41:13.389039+0000","last_fullsized":"2024-03-24T12:41:13.389039+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'30","last_scrub_stamp":"2024-03-24T12:40:16.701250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:16.701250+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:41.944147+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":475,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.225225+0000","last_change":"2024-03-24T12:35:58.711216+0000","last_active":"2024-03-24T12:41:08.225225+0000","last_peered":"2024-03-24T12:41:08.225225+0000","last_clean":"2024-03-24T12:35:54.688099+0000","last_became_active":"2024-03-24T12:35:58.711216+0000","last_became_peered":"2024-03-24T12:35:58.711216+0000","last_unstale":"2024-03-24T12:41:08.225225+0000","last_undegraded":"2024-03-24T12:41:08.225225+0000","last_fullsized":"2024-03-24T12:35:58.699124+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.802609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.802609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"286'2374","reported_seq":2400,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:21.411320+0000","last_change":"2024-03-24T12:40:51.136635+0000","last_active":"2024-03-24T12:41:21.411320+0000","last_peered":"2024-03-24T12:41:21.411320+0000","last_clean":"2024-03-24T12:41:21.411320+0000","last_became_active":"2024-03-24T12:40:51.136300+0000","last_became_peered":"2024-03-24T12:40:51.136300+0000","last_unstale":"2024-03-24T12:41:21.411320+0000","last_undegraded":"2024-03-24T12:41:21.411320+0000","last_fullsized":"2024-03-24T12:41:21.411320+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2374,"log_dups_size":0,"ondisk_log_size":2374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:12.559903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155582464,"num_objects":2374,"num_object_clones":0,"num_object_copies":7122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2374,"num_write_kb":151936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.220309+0000","last_change":"2024-03-24T12:35:58.717519+0000","last_active":"2024-03-24T12:41:08.220309+0000","last_peered":"2024-03-24T12:41:08.220309+0000","last_clean":"2024-03-24T12:35:54.687071+0000","last_became_active":"2024-03-24T12:35:58.717519+0000","last_became_peered":"2024-03-24T12:35:58.717519+0000","last_unstale":"2024-03-24T12:41:08.220309+0000","last_undegraded":"2024-03-24T12:41:08.220309+0000","last_fullsized":"2024-03-24T12:35:58.696848+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:59.974279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:59.974279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:10.331104+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":504,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222287+0000","last_change":"2024-03-24T12:40:05.457747+0000","last_active":"2024-03-24T12:41:08.222287+0000","last_peered":"2024-03-24T12:41:08.222287+0000","last_clean":"2024-03-24T12:41:08.222287+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:41:08.222287+0000","last_undegraded":"2024-03-24T12:41:08.222287+0000","last_fullsized":"2024-03-24T12:41:08.222287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:05.457632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:13.629968+0000","last_clean_scrub_stamp":"2024-03-24T12:40:05.457632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:11.857042+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1434,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:21.553367+0000","last_change":"2024-03-24T12:41:01.902067+0000","last_active":"2024-03-24T12:41:21.553367+0000","last_peered":"2024-03-24T12:41:21.553367+0000","last_clean":"2024-03-24T12:41:21.553367+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:41:21.553367+0000","last_undegraded":"2024-03-24T12:41:21.553367+0000","last_fullsized":"2024-03-24T12:41:21.553367+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:41:01.901959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:41:01.901959+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:18.560114+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":912,"num_read_kb":907,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"260'30","reported_seq":534,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222498+0000","last_change":"2024-03-24T12:39:51.288449+0000","last_active":"2024-03-24T12:41:08.222498+0000","last_peered":"2024-03-24T12:41:08.222498+0000","last_clean":"2024-03-24T12:41:08.222498+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:41:08.222498+0000","last_undegraded":"2024-03-24T12:41:08.222498+0000","last_fullsized":"2024-03-24T12:41:08.222498+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'30","last_scrub_stamp":"2024-03-24T12:39:51.288276+0000","last_deep_scrub":"260'30","last_deep_scrub_stamp":"2024-03-24T12:39:51.288276+0000","last_clean_scrub_stamp":"2024-03-24T12:39:51.288276+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":174,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.220653+0000","last_change":"2024-03-24T12:35:58.708969+0000","last_active":"2024-03-24T12:41:08.220653+0000","last_peered":"2024-03-24T12:41:08.220653+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.708969+0000","last_became_peered":"2024-03-24T12:35:58.708969+0000","last_unstale":"2024-03-24T12:41:08.220653+0000","last_undegraded":"2024-03-24T12:41:08.220653+0000","last_fullsized":"2024-03-24T12:35:58.699657+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:52.136918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:52.136918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:10.331111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"286'2461","reported_seq":2487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:22.602946+0000","last_change":"2024-03-24T12:40:51.139302+0000","last_active":"2024-03-24T12:41:22.602946+0000","last_peered":"2024-03-24T12:41:22.602946+0000","last_clean":"2024-03-24T12:41:22.602946+0000","last_became_active":"2024-03-24T12:40:51.138950+0000","last_became_peered":"2024-03-24T12:40:51.138950+0000","last_unstale":"2024-03-24T12:41:22.602946+0000","last_undegraded":"2024-03-24T12:41:22.602946+0000","last_fullsized":"2024-03-24T12:41:22.602946+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2461,"log_dups_size":0,"ondisk_log_size":2461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:08.394095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161284096,"num_objects":2461,"num_object_clones":0,"num_object_copies":7383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2461,"num_write_kb":157504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":512,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.219532+0000","last_change":"2024-03-24T12:40:17.290728+0000","last_active":"2024-03-24T12:41:08.219532+0000","last_peered":"2024-03-24T12:41:08.219532+0000","last_clean":"2024-03-24T12:41:08.219532+0000","last_became_active":"2024-03-24T12:39:34.751275+0000","last_became_peered":"2024-03-24T12:39:34.751275+0000","last_unstale":"2024-03-24T12:41:08.219532+0000","last_undegraded":"2024-03-24T12:41:08.219532+0000","last_fullsized":"2024-03-24T12:41:08.219532+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:17.290474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:40:17.290474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:27.874067+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":406,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.225889+0000","last_change":"2024-03-24T12:39:07.747764+0000","last_active":"2024-03-24T12:41:08.225889+0000","last_peered":"2024-03-24T12:41:08.225889+0000","last_clean":"2024-03-24T12:35:54.687290+0000","last_became_active":"2024-03-24T12:39:07.747764+0000","last_became_peered":"2024-03-24T12:39:07.747764+0000","last_unstale":"2024-03-24T12:41:08.225889+0000","last_undegraded":"2024-03-24T12:41:08.225889+0000","last_fullsized":"2024-03-24T12:39:07.743810+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:54.953570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.953570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"207'37","reported_seq":538,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:08.221742+0000","last_change":"2024-03-24T12:35:58.711927+0000","last_active":"2024-03-24T12:41:08.221742+0000","last_peered":"2024-03-24T12:41:08.221742+0000","last_clean":"2024-03-24T12:35:54.682985+0000","last_became_active":"2024-03-24T12:35:58.711927+0000","last_became_peered":"2024-03-24T12:35:58.711927+0000","last_unstale":"2024-03-24T12:41:08.221742+0000","last_undegraded":"2024-03-24T12:35:58.703907+0000","last_fullsized":"2024-03-24T12:35:58.703783+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'27","last_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_deep_scrub":"195'27","last_deep_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.981148+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.359880+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":450,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.225089+0000","last_change":"2024-03-24T12:40:06.758489+0000","last_active":"2024-03-24T12:41:08.225089+0000","last_peered":"2024-03-24T12:41:08.225089+0000","last_clean":"2024-03-24T12:41:08.225089+0000","last_became_active":"2024-03-24T12:39:35.099711+0000","last_became_peered":"2024-03-24T12:39:35.099711+0000","last_unstale":"2024-03-24T12:41:08.225089+0000","last_undegraded":"2024-03-24T12:41:08.225089+0000","last_fullsized":"2024-03-24T12:41:08.225089+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:06.758334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:40:06.758334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:32.772689+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,2],"avail_no_missing":["6","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"286'2533","reported_seq":2559,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:22.633105+0000","last_change":"2024-03-24T12:40:51.138669+0000","last_active":"2024-03-24T12:41:22.633105+0000","last_peered":"2024-03-24T12:41:22.633105+0000","last_clean":"2024-03-24T12:41:22.633105+0000","last_became_active":"2024-03-24T12:40:51.138108+0000","last_became_peered":"2024-03-24T12:40:51.138108+0000","last_unstale":"2024-03-24T12:41:22.633105+0000","last_undegraded":"2024-03-24T12:41:22.633105+0000","last_fullsized":"2024-03-24T12:41:22.633105+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2533,"log_dups_size":0,"ondisk_log_size":2533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:06.894950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166002688,"num_objects":2533,"num_object_clones":0,"num_object_copies":7599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2533,"num_write_kb":162112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":484,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.218584+0000","last_change":"2024-03-24T12:39:56.447208+0000","last_active":"2024-03-24T12:41:08.218584+0000","last_peered":"2024-03-24T12:41:08.218584+0000","last_clean":"2024-03-24T12:41:08.218584+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:41:08.218584+0000","last_undegraded":"2024-03-24T12:41:08.218584+0000","last_fullsized":"2024-03-24T12:41:08.218584+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:56.447087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:39:56.447087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":173,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.230750+0000","last_change":"2024-03-24T12:35:58.711061+0000","last_active":"2024-03-24T12:41:08.230750+0000","last_peered":"2024-03-24T12:41:08.230750+0000","last_clean":"2024-03-24T12:26:02.143860+0000","last_became_active":"2024-03-24T12:35:58.711061+0000","last_became_peered":"2024-03-24T12:35:58.711061+0000","last_unstale":"2024-03-24T12:41:08.230750+0000","last_undegraded":"2024-03-24T12:41:08.230750+0000","last_fullsized":"2024-03-24T12:35:58.704709+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:00.924106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.924106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"207'25","reported_seq":522,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:08.222632+0000","last_change":"2024-03-24T12:35:58.704200+0000","last_active":"2024-03-24T12:41:08.222632+0000","last_peered":"2024-03-24T12:41:08.222632+0000","last_clean":"2024-03-24T12:35:54.687582+0000","last_became_active":"2024-03-24T12:35:58.704200+0000","last_became_peered":"2024-03-24T12:35:58.704200+0000","last_unstale":"2024-03-24T12:41:08.222632+0000","last_undegraded":"2024-03-24T12:35:58.697842+0000","last_fullsized":"2024-03-24T12:35:58.697725+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:33:07.518368+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.518368+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:10.331063+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"258'13","reported_seq":700,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:13.390903+0000","last_change":"2024-03-24T12:41:04.369404+0000","last_active":"2024-03-24T12:41:13.390903+0000","last_peered":"2024-03-24T12:41:13.390903+0000","last_clean":"2024-03-24T12:41:13.390903+0000","last_became_active":"2024-03-24T12:39:35.135410+0000","last_became_peered":"2024-03-24T12:39:35.135410+0000","last_unstale":"2024-03-24T12:41:13.390903+0000","last_undegraded":"2024-03-24T12:41:13.390903+0000","last_fullsized":"2024-03-24T12:41:13.390903+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'13","last_scrub_stamp":"2024-03-24T12:41:04.368025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:41:04.368025+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:05.648908+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,3],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.f","version":"286'2502","reported_seq":2528,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:22.619601+0000","last_change":"2024-03-24T12:40:51.398644+0000","last_active":"2024-03-24T12:41:22.619601+0000","last_peered":"2024-03-24T12:41:22.619601+0000","last_clean":"2024-03-24T12:41:22.619601+0000","last_became_active":"2024-03-24T12:40:51.398299+0000","last_became_peered":"2024-03-24T12:40:51.398299+0000","last_unstale":"2024-03-24T12:41:22.619601+0000","last_undegraded":"2024-03-24T12:41:22.619601+0000","last_fullsized":"2024-03-24T12:41:22.619601+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2502,"log_dups_size":0,"ondisk_log_size":2502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:54.676346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163971072,"num_objects":2502,"num_object_clones":0,"num_object_copies":7506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2502,"num_write_kb":160128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":504,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.223912+0000","last_change":"2024-03-24T12:40:59.957298+0000","last_active":"2024-03-24T12:41:08.223912+0000","last_peered":"2024-03-24T12:41:08.223912+0000","last_clean":"2024-03-24T12:41:08.223912+0000","last_became_active":"2024-03-24T12:39:34.756439+0000","last_became_peered":"2024-03-24T12:39:34.756439+0000","last_unstale":"2024-03-24T12:41:08.223912+0000","last_undegraded":"2024-03-24T12:41:08.223912+0000","last_fullsized":"2024-03-24T12:41:08.223912+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:59.957191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:40:59.957191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:23.238947+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":505,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222650+0000","last_change":"2024-03-24T12:40:19.405751+0000","last_active":"2024-03-24T12:41:08.222650+0000","last_peered":"2024-03-24T12:41:08.222650+0000","last_clean":"2024-03-24T12:41:08.222650+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:41:08.222650+0000","last_undegraded":"2024-03-24T12:41:08.222650+0000","last_fullsized":"2024-03-24T12:41:08.222650+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:40:19.405617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:19.405617+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:30.327261+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"207'35","reported_seq":570,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.221315+0000","last_change":"2024-03-24T12:39:47.746018+0000","last_active":"2024-03-24T12:41:08.221315+0000","last_peered":"2024-03-24T12:41:08.221315+0000","last_clean":"2024-03-24T12:41:08.221315+0000","last_became_active":"2024-03-24T12:39:35.103040+0000","last_became_peered":"2024-03-24T12:39:35.103040+0000","last_unstale":"2024-03-24T12:41:08.221315+0000","last_undegraded":"2024-03-24T12:41:08.221315+0000","last_fullsized":"2024-03-24T12:41:08.221315+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-24T12:39:47.745851+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:39:47.745851+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:10.588669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":680,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:13.388773+0000","last_change":"2024-03-24T12:40:16.189646+0000","last_active":"2024-03-24T12:41:13.388773+0000","last_peered":"2024-03-24T12:41:13.388773+0000","last_clean":"2024-03-24T12:41:13.388773+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:41:13.388773+0000","last_undegraded":"2024-03-24T12:41:13.388773+0000","last_fullsized":"2024-03-24T12:41:13.388773+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:40:16.189517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:40:16.189517+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:42.890789+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.e","version":"286'2469","reported_seq":2495,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:22.507236+0000","last_change":"2024-03-24T12:40:51.139343+0000","last_active":"2024-03-24T12:41:22.507236+0000","last_peered":"2024-03-24T12:41:22.507236+0000","last_clean":"2024-03-24T12:41:22.507236+0000","last_became_active":"2024-03-24T12:40:51.137734+0000","last_became_peered":"2024-03-24T12:40:51.137734+0000","last_unstale":"2024-03-24T12:41:22.507236+0000","last_undegraded":"2024-03-24T12:41:22.507236+0000","last_fullsized":"2024-03-24T12:41:22.507236+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2469,"log_dups_size":0,"ondisk_log_size":2469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:01.087461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161808384,"num_objects":2469,"num_object_clones":0,"num_object_copies":7407,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2469,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2469,"num_write_kb":158016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":492,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221242+0000","last_change":"2024-03-24T12:41:04.979281+0000","last_active":"2024-03-24T12:41:08.221242+0000","last_peered":"2024-03-24T12:41:08.221242+0000","last_clean":"2024-03-24T12:41:08.221242+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:41:08.221242+0000","last_undegraded":"2024-03-24T12:41:08.221242+0000","last_fullsized":"2024-03-24T12:41:08.221242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:41:04.979163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:41:04.979163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:32.082957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":488,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.230825+0000","last_change":"2024-03-24T12:37:30.870003+0000","last_active":"2024-03-24T12:41:08.230825+0000","last_peered":"2024-03-24T12:41:08.230825+0000","last_clean":"2024-03-24T12:41:08.230825+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:41:08.230825+0000","last_undegraded":"2024-03-24T12:41:08.230825+0000","last_fullsized":"2024-03-24T12:41:08.230825+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:30.869794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:30.869794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"207'30","reported_seq":542,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.228439+0000","last_change":"2024-03-24T12:37:54.840702+0000","last_active":"2024-03-24T12:41:08.228439+0000","last_peered":"2024-03-24T12:41:08.228439+0000","last_clean":"2024-03-24T12:41:08.228439+0000","last_became_active":"2024-03-24T12:26:05.358000+0000","last_became_peered":"2024-03-24T12:26:05.358000+0000","last_unstale":"2024-03-24T12:41:08.228439+0000","last_undegraded":"2024-03-24T12:41:08.228439+0000","last_fullsized":"2024-03-24T12:41:08.228439+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'30","last_scrub_stamp":"2024-03-24T12:37:54.840559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:37:54.840559+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":504,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221083+0000","last_change":"2024-03-24T12:40:57.941713+0000","last_active":"2024-03-24T12:41:08.221083+0000","last_peered":"2024-03-24T12:41:08.221083+0000","last_clean":"2024-03-24T12:41:08.221083+0000","last_became_active":"2024-03-24T12:26:07.336624+0000","last_became_peered":"2024-03-24T12:26:07.336624+0000","last_unstale":"2024-03-24T12:41:08.221083+0000","last_undegraded":"2024-03-24T12:41:08.221083+0000","last_fullsized":"2024-03-24T12:41:08.221083+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:57.941616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:40:57.941616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:05.686716+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"286'2486","reported_seq":2512,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:22.494424+0000","last_change":"2024-03-24T12:40:51.135671+0000","last_active":"2024-03-24T12:41:22.494424+0000","last_peered":"2024-03-24T12:41:22.494424+0000","last_clean":"2024-03-24T12:41:22.494424+0000","last_became_active":"2024-03-24T12:40:51.135277+0000","last_became_peered":"2024-03-24T12:40:51.135277+0000","last_unstale":"2024-03-24T12:41:22.494424+0000","last_undegraded":"2024-03-24T12:41:22.494424+0000","last_fullsized":"2024-03-24T12:41:22.494424+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2486,"log_dups_size":0,"ondisk_log_size":2486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:04.171508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162922496,"num_objects":2486,"num_object_clones":0,"num_object_copies":7458,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2486,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2486,"num_write_kb":159104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":173,"reported_epoch":286,"state":"active+undersized","last_fresh":"2024-03-24T12:41:08.219838+0000","last_change":"2024-03-24T12:35:58.711067+0000","last_active":"2024-03-24T12:41:08.219838+0000","last_peered":"2024-03-24T12:41:08.219838+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.711067+0000","last_became_peered":"2024-03-24T12:35:58.711067+0000","last_unstale":"2024-03-24T12:41:08.219838+0000","last_undegraded":"2024-03-24T12:41:08.219838+0000","last_fullsized":"2024-03-24T12:35:58.696416+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:36.210440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.210440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:10.331096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"205'10","reported_seq":676,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:13.388946+0000","last_change":"2024-03-24T12:35:58.712862+0000","last_active":"2024-03-24T12:41:13.388946+0000","last_peered":"2024-03-24T12:41:13.388946+0000","last_clean":"2024-03-24T12:35:54.678342+0000","last_became_active":"2024-03-24T12:35:58.712862+0000","last_became_peered":"2024-03-24T12:35:58.712862+0000","last_unstale":"2024-03-24T12:41:13.388946+0000","last_undegraded":"2024-03-24T12:35:58.700712+0000","last_fullsized":"2024-03-24T12:35:58.700587+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'9","last_scrub_stamp":"2024-03-24T12:34:44.703379+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.703379+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":488,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.228448+0000","last_change":"2024-03-24T12:37:37.893086+0000","last_active":"2024-03-24T12:41:08.228448+0000","last_peered":"2024-03-24T12:41:08.228448+0000","last_clean":"2024-03-24T12:41:08.228448+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:41:08.228448+0000","last_undegraded":"2024-03-24T12:41:08.228448+0000","last_fullsized":"2024-03-24T12:41:08.228448+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:37.892950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:37.892950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"207'34","reported_seq":532,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.227051+0000","last_change":"2024-03-24T12:39:50.751466+0000","last_active":"2024-03-24T12:41:08.227051+0000","last_peered":"2024-03-24T12:41:08.227051+0000","last_clean":"2024-03-24T12:41:08.227051+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:41:08.227051+0000","last_undegraded":"2024-03-24T12:41:08.227051+0000","last_fullsized":"2024-03-24T12:41:08.227051+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-24T12:39:50.751304+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:39:50.751304+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:09.540692+0000","scrub_duration":198,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"286'2454","reported_seq":2480,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:22.603141+0000","last_change":"2024-03-24T12:40:51.138626+0000","last_active":"2024-03-24T12:41:22.603141+0000","last_peered":"2024-03-24T12:41:22.603141+0000","last_clean":"2024-03-24T12:41:22.603141+0000","last_became_active":"2024-03-24T12:40:51.138083+0000","last_became_peered":"2024-03-24T12:40:51.138083+0000","last_unstale":"2024-03-24T12:41:22.603141+0000","last_undegraded":"2024-03-24T12:41:22.603141+0000","last_fullsized":"2024-03-24T12:41:22.603141+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2454,"log_dups_size":0,"ondisk_log_size":2454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:08.366131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160825344,"num_objects":2454,"num_object_clones":0,"num_object_copies":7362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2454,"num_write_kb":157056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":480,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.230116+0000","last_change":"2024-03-24T12:39:55.001557+0000","last_active":"2024-03-24T12:41:08.230116+0000","last_peered":"2024-03-24T12:41:08.230116+0000","last_clean":"2024-03-24T12:41:08.230116+0000","last_became_active":"2024-03-24T12:39:35.129779+0000","last_became_peered":"2024-03-24T12:39:35.129779+0000","last_unstale":"2024-03-24T12:41:08.230116+0000","last_undegraded":"2024-03-24T12:41:08.230116+0000","last_fullsized":"2024-03-24T12:41:08.230116+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:55.000906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:39:55.000906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:16.521086+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":192,"reported_epoch":286,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:41:08.221702+0000","last_change":"2024-03-24T12:39:51.731048+0000","last_active":"2024-03-24T12:41:08.221702+0000","last_peered":"2024-03-24T12:41:08.221702+0000","last_clean":"2024-03-24T12:41:08.221702+0000","last_became_active":"2024-03-24T12:39:35.106802+0000","last_became_peered":"2024-03-24T12:39:35.106802+0000","last_unstale":"2024-03-24T12:41:08.221702+0000","last_undegraded":"2024-03-24T12:41:08.221702+0000","last_fullsized":"2024-03-24T12:41:08.221702+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:51.730856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:39:51.730856+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,2],"avail_no_missing":["4","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":504,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221227+0000","last_change":"2024-03-24T12:40:55.957992+0000","last_active":"2024-03-24T12:41:08.221227+0000","last_peered":"2024-03-24T12:41:08.221227+0000","last_clean":"2024-03-24T12:41:08.221227+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:41:08.221227+0000","last_undegraded":"2024-03-24T12:41:08.221227+0000","last_fullsized":"2024-03-24T12:41:08.221227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:55.957883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:55.957883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:12.507316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"207'19","reported_seq":566,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221749+0000","last_change":"2024-03-24T12:40:19.827132+0000","last_active":"2024-03-24T12:41:08.221749+0000","last_peered":"2024-03-24T12:41:08.221749+0000","last_clean":"2024-03-24T12:41:08.221749+0000","last_became_active":"2024-03-24T12:39:34.752219+0000","last_became_peered":"2024-03-24T12:39:34.752219+0000","last_unstale":"2024-03-24T12:41:08.221749+0000","last_undegraded":"2024-03-24T12:41:08.221749+0000","last_fullsized":"2024-03-24T12:41:08.221749+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:40:19.827036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:19.827036+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:30.819427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"286'2573","reported_seq":2596,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:24.128677+0000","last_change":"2024-03-24T12:41:04.286472+0000","last_active":"2024-03-24T12:41:24.128677+0000","last_peered":"2024-03-24T12:41:24.128677+0000","last_clean":"2024-03-24T12:40:50.114364+0000","last_became_active":"2024-03-24T12:40:51.397675+0000","last_became_peered":"2024-03-24T12:40:51.397675+0000","last_unstale":"2024-03-24T12:41:24.128677+0000","last_undegraded":"2024-03-24T12:41:04.282306+0000","last_fullsized":"2024-03-24T12:40:51.132010+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2573,"log_dups_size":0,"ondisk_log_size":2573,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:56.813691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":168624128,"num_objects":2573,"num_object_clones":0,"num_object_copies":7719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2573,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2573,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2573,"num_write_kb":164672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":2573}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":499,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.227270+0000","last_change":"2024-03-24T12:39:53.627173+0000","last_active":"2024-03-24T12:41:08.227270+0000","last_peered":"2024-03-24T12:41:08.227270+0000","last_clean":"2024-03-24T12:41:08.227270+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:41:08.227270+0000","last_undegraded":"2024-03-24T12:41:08.227270+0000","last_fullsized":"2024-03-24T12:41:08.227270+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:53.627010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:53.627010+0000","last_clean_scrub_stamp":"2024-03-24T12:39:53.627010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:20.713857+0000","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":97,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221922+0000","last_change":"2024-03-24T12:40:00.671902+0000","last_active":"2024-03-24T12:41:08.221922+0000","last_peered":"2024-03-24T12:41:08.221922+0000","last_clean":"2024-03-24T12:41:08.221922+0000","last_became_active":"2024-03-24T12:39:07.753551+0000","last_became_peered":"2024-03-24T12:39:07.753551+0000","last_unstale":"2024-03-24T12:41:08.221922+0000","last_undegraded":"2024-03-24T12:41:08.221922+0000","last_fullsized":"2024-03-24T12:41:08.221922+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:00.671810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:40:00.671810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:30.112553+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":81,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.224273+0000","last_change":"2024-03-24T12:40:14.758714+0000","last_active":"2024-03-24T12:41:08.224273+0000","last_peered":"2024-03-24T12:41:08.224273+0000","last_clean":"2024-03-24T12:41:08.224273+0000","last_became_active":"2024-03-24T12:39:34.745353+0000","last_became_peered":"2024-03-24T12:39:34.745353+0000","last_unstale":"2024-03-24T12:41:08.224273+0000","last_undegraded":"2024-03-24T12:41:08.224273+0000","last_fullsized":"2024-03-24T12:41:08.224273+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:14.758640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:14.758640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:31.654764+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"207'39","reported_seq":556,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221521+0000","last_change":"2024-03-24T12:40:15.867299+0000","last_active":"2024-03-24T12:41:08.221521+0000","last_peered":"2024-03-24T12:41:08.221521+0000","last_clean":"2024-03-24T12:41:08.221521+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:41:08.221521+0000","last_undegraded":"2024-03-24T12:41:08.221521+0000","last_fullsized":"2024-03-24T12:41:08.221521+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-24T12:40:15.867137+0000","last_deep_scrub":"207'39","last_deep_scrub_stamp":"2024-03-24T12:36:22.669686+0000","last_clean_scrub_stamp":"2024-03-24T12:40:15.867137+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:16.572377+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2","version":"286'2196","reported_seq":2222,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:19.586414+0000","last_change":"2024-03-24T12:40:51.131201+0000","last_active":"2024-03-24T12:41:19.586414+0000","last_peered":"2024-03-24T12:41:19.586414+0000","last_clean":"2024-03-24T12:41:19.586414+0000","last_became_active":"2024-03-24T12:40:51.130578+0000","last_became_peered":"2024-03-24T12:40:51.130578+0000","last_unstale":"2024-03-24T12:41:19.586414+0000","last_undegraded":"2024-03-24T12:41:19.586414+0000","last_fullsized":"2024-03-24T12:41:19.586414+0000","mapping_epoch":278,"log_start":"0'0","ondisk_log_start":"0'0","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":2196,"log_dups_size":0,"ondisk_log_size":2196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:00.699555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143917056,"num_objects":2196,"num_object_clones":0,"num_object_copies":6588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2196,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2196,"num_write_kb":140544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":499,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.222299+0000","last_change":"2024-03-24T12:40:52.996240+0000","last_active":"2024-03-24T12:41:08.222299+0000","last_peered":"2024-03-24T12:41:08.222299+0000","last_clean":"2024-03-24T12:41:08.222299+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:41:08.222299+0000","last_undegraded":"2024-03-24T12:41:08.222299+0000","last_fullsized":"2024-03-24T12:41:08.222299+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:52.996140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:40:52.996140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:41:57.915949+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"205'9","reported_seq":676,"reported_epoch":286,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:41:18.390014+0000","last_change":"2024-03-24T12:35:58.752262+0000","last_active":"2024-03-24T12:41:18.390014+0000","last_peered":"2024-03-24T12:41:18.390014+0000","last_clean":"2024-03-24T12:35:54.684037+0000","last_became_active":"2024-03-24T12:35:58.752262+0000","last_became_peered":"2024-03-24T12:35:58.752262+0000","last_unstale":"2024-03-24T12:41:18.390014+0000","last_undegraded":"2024-03-24T12:35:58.701740+0000","last_fullsized":"2024-03-24T12:35:58.701602+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:35:00.998713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.998713+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":487,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.230280+0000","last_change":"2024-03-24T12:37:38.861709+0000","last_active":"2024-03-24T12:41:08.230280+0000","last_peered":"2024-03-24T12:41:08.230280+0000","last_clean":"2024-03-24T12:41:08.230280+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:41:08.230280+0000","last_undegraded":"2024-03-24T12:41:08.230280+0000","last_fullsized":"2024-03-24T12:41:08.230280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:38.861597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:38.861597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"207'20","reported_seq":573,"reported_epoch":286,"state":"active+clean","last_fresh":"2024-03-24T12:41:08.221970+0000","last_change":"2024-03-24T12:41:06.988179+0000","last_active":"2024-03-24T12:41:08.221970+0000","last_peered":"2024-03-24T12:41:08.221970+0000","last_clean":"2024-03-24T12:41:08.221970+0000","last_became_active":"2024-03-24T12:39:34.759881+0000","last_became_peered":"2024-03-24T12:39:34.759881+0000","last_unstale":"2024-03-24T12:41:08.221970+0000","last_undegraded":"2024-03-24T12:41:08.221970+0000","last_fullsized":"2024-03-24T12:41:08.221970+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:41:06.988045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:41:06.988045+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:26.750867+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2584936448,"num_objects":39443,"num_object_clones":0,"num_object_copies":118329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12613,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":39443,"num_write_kb":2524352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6811418624,"data_stored":6811418624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39443,"ondisk_log_size":39443,"up":43,"acting":43,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":912,"num_read_kb":907,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":90,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":23,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1123,"num_read_kb":880,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":924,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":759,"ondisk_log_size":759,"up":88,"acting":92,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":83,"acting":86,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":82,"acting":86,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037948,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1140260,"kb_used_data":613468,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92604892,"statfs":{"total":95995035648,"available":94827409408,"internally_reserved":0,"allocated":628191232,"data_stored":627627725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":539425226},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332290,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1520204,"kb_used_data":1110020,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":92224948,"statfs":{"total":95995035648,"available":94438346752,"internally_reserved":0,"allocated":1136660480,"data_stored":1136120961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":420018631},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1,3],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575068,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":949248,"kb_used_data":404408,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":92795904,"statfs":{"total":95995035648,"available":95023005696,"internally_reserved":0,"allocated":414113792,"data_stored":413522334,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738593,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2107056,"kb_used_data":1498344,"kb_used_omap":1,"kb_used_meta":608702,"kb_avail":91638096,"statfs":{"total":95995035648,"available":93837410304,"internally_reserved":0,"allocated":1534304256,"data_stored":1533728044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":623311301},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,9],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411543,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1497928,"kb_used_data":971136,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":92247224,"statfs":{"total":95995035648,"available":94461157376,"internally_reserved":0,"allocated":994443264,"data_stored":993906757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117199,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1287664,"kb_used_data":706728,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92457488,"statfs":{"total":95995035648,"available":94676467712,"internally_reserved":0,"allocated":723689472,"data_stored":723113449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":594868679},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920972,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1635044,"kb_used_data":1234588,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92110108,"statfs":{"total":95995035648,"available":94320750592,"internally_reserved":0,"allocated":1264218112,"data_stored":1263653313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":410057157},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626566,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957988,"kb_used_data":688028,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92787164,"statfs":{"total":95995035648,"available":95014055936,"internally_reserved":0,"allocated":704540672,"data_stored":704173829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1534984192,"data_stored":1534984192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":419364864,"data_stored":419364864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1026621440,"data_stored":1026621440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":737869824,"data_stored":737869824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1301086208,"data_stored":1301086208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1144455168,"data_stored":1144455168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":647036928,"data_stored":647036928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:41:28.156 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T12:41:28.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pg_num 11 2024-03-24T12:41:28.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:28.229 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:28.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:28.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:28.279 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 24 16 49615 49599 129.137 108.25 0.0056942 0.00773835 2024-03-24T12:41:28.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:28.416 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:28.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:28.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:28 smithi029 ceph-mon[41500]: pgmap v646: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.55k op/s; 13222/125553 objects degraded (10.531%); 26/125553 objects misplaced (0.021%) 2024-03-24T12:41:28.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:28 smithi029 ceph-mon[29919]: pgmap v646: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.55k op/s; 13222/125553 objects degraded (10.531%); 26/125553 objects misplaced (0.021%) 2024-03-24T12:41:28.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:28.533 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:28 smithi190 ceph-mon[33748]: pgmap v646: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.55k op/s; 13222/125553 objects degraded (10.531%); 26/125553 objects misplaced (0.021%) 2024-03-24T12:41:28.533 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:41:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:41:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:28.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:28.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:28.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:28.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:28.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:28.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:28.828 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:28.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:28.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:28.954 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:28.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:29.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:29.087 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:29.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:29.189 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:29.225 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:29.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:29.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:29.279 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 25 16 51006 50990 127.449 86.9375 0.00294887 0.0078339 2024-03-24T12:41:29.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:29.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:29.351 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:29.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:29.452 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:29.486 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:29.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:29.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:29.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:29.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:29.612 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:29.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:29.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:29 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:29.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:29 smithi190 ceph-mon[33748]: pgmap v647: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s; 14258/140130 objects degraded (10.175%); 26/140130 objects misplaced (0.019%) 2024-03-24T12:41:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:29 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:29 smithi029 ceph-mon[29919]: pgmap v647: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s; 14258/140130 objects degraded (10.175%); 26/140130 objects misplaced (0.019%) 2024-03-24T12:41:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:29 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:29 smithi029 ceph-mon[41500]: pgmap v647: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 111 MiB/s wr, 1.77k op/s; 14258/140130 objects degraded (10.175%); 26/140130 objects misplaced (0.019%) 2024-03-24T12:41:29.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:29.744 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:29.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:29.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:29.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:29.887 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:29.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:29.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:30.020 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:30.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:30.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:30.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:30.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:30.170 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:30.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:30.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:30.280 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 26 16 52450 52434 126.018 90.25 0.00975967 0.00793043 2024-03-24T12:41:30.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:30.305 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:30.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:30.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:30.449 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:30.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:30.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:30.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:30.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:30 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 14258/140130 objects degraded (10.175%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:30.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:30 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 14258/140130 objects degraded (10.175%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:30.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:30 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 14258/140130 objects degraded (10.175%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:30.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:30.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:30.706 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:30.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:30.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:30.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:30.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:30.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:30.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:30.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:30.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:31.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:31.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:31.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:31.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:31.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:31.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:31.280 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 27 15 53568 53553 123.941 69.9375 0.00470651 0.00806358 2024-03-24T12:41:31.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:31.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:31.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:31.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:31.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:31.502 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:31.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:31.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:31 smithi190 ceph-mon[33748]: pgmap v648: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.56k op/s; 15303/146484 objects degraded (10.447%); 26/146484 objects misplaced (0.018%) 2024-03-24T12:41:31.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:31.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:31.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:31 smithi029 ceph-mon[41500]: pgmap v648: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.56k op/s; 15303/146484 objects degraded (10.447%); 26/146484 objects misplaced (0.018%) 2024-03-24T12:41:31.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:31 smithi029 ceph-mon[29919]: pgmap v648: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 97 MiB/s wr, 1.56k op/s; 15303/146484 objects degraded (10.447%); 26/146484 objects misplaced (0.018%) 2024-03-24T12:41:31.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:31.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:31.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:31.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:31.894 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:31.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:31.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:32.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:32.058 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:32.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:32.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:32.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:32.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:32.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:32.280 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 28 16 54817 54801 122.301 78 0.00626996 0.00817099 2024-03-24T12:41:32.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:32.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:32.319 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:32.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:32.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:32.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:32.447 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:32.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:32.548 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:32.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:32.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:32.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:32.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:32.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:32.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:32.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:32.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:32.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:32.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:33.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:33.095 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:33.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:33.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:33.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:33.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:33.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:33.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:33.280 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 29 15 55510 55495 119.579 43.375 0.0114117 0.00835765 2024-03-24T12:41:33.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:33.370 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:33.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:33.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:33.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:33.515 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:33.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:33.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:33.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:33.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:33.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:33.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:33.780 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:33.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:33.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:33.798 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:33.810 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:33.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:33.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:33.957 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:33 smithi029 ceph-mon[29919]: pgmap v649: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 16549/161874 objects degraded (10.223%); 26/161874 objects misplaced (0.016%) 2024-03-24T12:41:33.957 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:33.957 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:33 smithi029 ceph-mon[41500]: pgmap v649: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 16549/161874 objects degraded (10.223%); 26/161874 objects misplaced (0.016%) 2024-03-24T12:41:33.957 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:34.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:34.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:34.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:34.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:33 smithi190 ceph-mon[33748]: pgmap v649: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 115 MiB/s wr, 1.84k op/s; 16549/161874 objects degraded (10.223%); 26/161874 objects misplaced (0.016%) 2024-03-24T12:41:34.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:34.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:34.230 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:34.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:34.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:34.280 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 30 16 56946 56930 118.582 89.6875 0.00935837 0.00842791 2024-03-24T12:41:34.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:34.386 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:34.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:34.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:34.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:34.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:34.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:34.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:34.705 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:34.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:34.806 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:34.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:34.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:34.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:34.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:34.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:34.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:34.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:35.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:35.128 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:35.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:35.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:35.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:35.269 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:35.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:35.280 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 31 16 58420 58404 117.729 92.125 0.00606721 0.00848953 2024-03-24T12:41:35.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:35.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:35.404 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:35.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:35.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:35.539 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:35.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:35.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:35.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:35.680 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:35.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:35.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:35.813 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:41:35.825 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:35.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:35.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:35.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:35.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:35.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:36.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:36.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:36.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:35 smithi190 ceph-mon[33748]: pgmap v650: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s; 17310/165309 objects degraded (10.471%); 26/165309 objects misplaced (0.016%) 2024-03-24T12:41:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:35 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3807013335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:41:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:35 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:41:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:35 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 17310/165309 objects degraded (10.471%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:35 smithi029 ceph-mon[29919]: pgmap v650: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s; 17310/165309 objects degraded (10.471%); 26/165309 objects misplaced (0.016%) 2024-03-24T12:41:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:35 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3807013335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:41:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:35 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:41:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:35 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 17310/165309 objects degraded (10.471%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:36.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:35 smithi029 ceph-mon[41500]: pgmap v650: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 89 MiB/s wr, 1.42k op/s; 17310/165309 objects degraded (10.471%); 26/165309 objects misplaced (0.016%) 2024-03-24T12:41:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:35 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3807013335' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:41:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:35 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:41:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:35 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 17310/165309 objects degraded (10.471%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:36.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:36.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:36.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:36.280 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 32 16 59972 59956 117.081 97 0.0103829 0.008536 2024-03-24T12:41:36.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:36.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:36.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:36.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:36.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:36.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:36.520 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:36.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:36.660 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:36.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:36.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:36.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:36.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:36.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:36.933 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:36.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:36.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:37.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:37.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:37.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:36 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:41:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:36 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:41:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:36 smithi190 ceph-mon[33748]: osdmap e287: 8 total, 7 up, 8 in 2024-03-24T12:41:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:36 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:41:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:36 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:41:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:36 smithi029 ceph-mon[29919]: osdmap e287: 8 total, 7 up, 8 in 2024-03-24T12:41:37.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:36 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:41:37.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:36 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:41:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:36 smithi029 ceph-mon[41500]: osdmap e287: 8 total, 7 up, 8 in 2024-03-24T12:41:37.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:37.219 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:37.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:37.280 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 33 16 61402 61386 116.241 89.375 0.00812456 0.00859754 2024-03-24T12:41:37.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:37.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:37.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:37.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:37.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:37.496 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:37.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:37.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:37.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:37.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:37.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:37.779 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:37.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:37.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:37.881 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:37.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:37.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:37.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:37 smithi029 ceph-mon[41500]: pgmap v652: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.48k op/s; 17697/169857 objects degraded (10.419%); 26/169857 objects misplaced (0.015%) 2024-03-24T12:41:37.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:37 smithi029 ceph-mon[29919]: pgmap v652: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.48k op/s; 17697/169857 objects degraded (10.419%); 26/169857 objects misplaced (0.015%) 2024-03-24T12:41:37.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:41:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:41:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:41:38.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:38.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:38.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:38.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:38.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:37 smithi190 ceph-mon[33748]: pgmap v652: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 92 MiB/s wr, 1.48k op/s; 17697/169857 objects degraded (10.419%); 26/169857 objects misplaced (0.015%) 2024-03-24T12:41:38.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:38.196 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:38.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:38.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:38.280 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 34 16 62939 62923 115.647 96.0625 0.0119943 0.00863016 2024-03-24T12:41:38.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:38.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:38.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:38.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:38.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:38.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:38.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:38.615 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:38.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:38.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:38.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:41:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:41:38.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:38.752 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:38.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:38.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:38.877 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:38.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:38.978 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:39.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:39.003 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:39.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:39.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:39.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:39.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:39.240 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:39.273 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:39.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:39.280 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 35 16 64210 64194 114.613 79.4375 0.00935375 0.00872 2024-03-24T12:41:39.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:39.415 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:39.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:39.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:39.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:39.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:39.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:39.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:39.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:39.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:39.834 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:39.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:39.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:39.967 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:39.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:39.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:40.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:40.098 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:40.098 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:39 smithi029 ceph-mon[41500]: pgmap v653: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 18547/181695 objects degraded (10.208%); 26/181695 objects misplaced (0.014%) 2024-03-24T12:41:40.098 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:39 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:41:40.098 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:39 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:41:40.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:40.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:39 smithi029 ceph-mon[29919]: pgmap v653: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 18547/181695 objects degraded (10.208%); 26/181695 objects misplaced (0.014%) 2024-03-24T12:41:40.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:39 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:41:40.099 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:39 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:41:40.113 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:40.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:40.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:39 smithi190 ceph-mon[33748]: pgmap v653: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.39k op/s; 18547/181695 objects degraded (10.208%); 26/181695 objects misplaced (0.014%) 2024-03-24T12:41:40.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:39 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:41:40.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:39 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:41:40.215 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:40.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:40.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:40.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 36 16 65669 65653 113.962 91.1875 0.0278456 0.00876896 2024-03-24T12:41:40.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:40.380 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:40.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:40.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:40.511 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:40.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:40.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:40.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:40.657 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:40.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:40.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:40.797 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:40.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:40.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:40.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:40.942 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:40.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:41.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:41.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:41.085 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:41.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:40 smithi190 ceph-mon[33748]: osdmap e288: 8 total, 7 up, 8 in 2024-03-24T12:41:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:40 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 18547/181695 objects degraded (10.208%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:41.141 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:40 smithi029 ceph-mon[41500]: osdmap e288: 8 total, 7 up, 8 in 2024-03-24T12:41:41.141 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:40 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 18547/181695 objects degraded (10.208%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:41.141 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:40 smithi029 ceph-mon[29919]: osdmap e288: 8 total, 7 up, 8 in 2024-03-24T12:41:41.141 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:40 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 18547/181695 objects degraded (10.208%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:41.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:41.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:41.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:41.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 37 16 66926 66910 113.005 78.5625 0.0126038 0.00884324 2024-03-24T12:41:41.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:41.348 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:41.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:41.387 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:41:41.427 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:41:41.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:41.467 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:41:41.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:41.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:41.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:41.511 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:41:41.551 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:41:41.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:41.596 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:41:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:41.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:41.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:41.635 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:41:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:41:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:41:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:41:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:41:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:41:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:41:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:41:41.636 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:41:41.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:41.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:41.771 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:41.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:41.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:41.917 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:41.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:41.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:42.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:42.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:42.058 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:42.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:41 smithi190 ceph-mon[33748]: pgmap v655: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.05k op/s; 19447/187032 objects degraded (10.398%); 26/187032 objects misplaced (0.014%) 2024-03-24T12:41:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:41 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:41:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:41 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:41:42.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:41 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1714422833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:41:42.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:41 smithi029 ceph-mon[29919]: pgmap v655: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.05k op/s; 19447/187032 objects degraded (10.398%); 26/187032 objects misplaced (0.014%) 2024-03-24T12:41:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:41 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:41:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:41 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:41:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:41 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1714422833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:41:42.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:41 smithi029 ceph-mon[41500]: pgmap v655: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 3.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 65 MiB/s wr, 1.05k op/s; 19447/187032 objects degraded (10.398%); 26/187032 objects misplaced (0.014%) 2024-03-24T12:41:42.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:41 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:41:42.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:41 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:41:42.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:41 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1714422833' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:41:42.201 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:42.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:42.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 38 15 68312 68297 112.312 86.6875 0.017322 0.00889919 2024-03-24T12:41:42.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:42.336 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:42.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:42.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:42.477 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:42.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:42.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:42.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:42.615 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:42.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:42.716 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:42.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:42.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:42.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:42.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:42 smithi029 ceph-mon[29919]: osdmap e289: 8 total, 7 up, 8 in 2024-03-24T12:41:42.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:42 smithi029 ceph-mon[41500]: osdmap e289: 8 total, 7 up, 8 in 2024-03-24T12:41:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:42.919 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:42.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:43.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:43.065 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:43.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:43.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:42 smithi190 ceph-mon[33748]: osdmap e289: 8 total, 7 up, 8 in 2024-03-24T12:41:43.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:43.209 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:43.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:43.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:43.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 39 16 69688 69672 111.636 85.9375 0.0196858 0.00895246 2024-03-24T12:41:43.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:43.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:43.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:43.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:43.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:43.504 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:43.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:43.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:43.629 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:41:43.629 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:41:43.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:43.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:43.771 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:43.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:43.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:43.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:43.912 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:43.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:43.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:44.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:44.104 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:44.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:44.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:44.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:43 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:41:44.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:43 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:41:44.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:43 smithi029 ceph-mon[41500]: pgmap v657: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 20749/202644 objects degraded (10.239%); 26/202644 objects misplaced (0.013%) 2024-03-24T12:41:44.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:43 smithi029 ceph-mon[41500]: osdmap e290: 8 total, 7 up, 8 in 2024-03-24T12:41:44.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:43 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:41:44.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:43 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:41:44.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:43 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:41:44.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:43 smithi029 ceph-mon[29919]: pgmap v657: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 20749/202644 objects degraded (10.239%); 26/202644 objects misplaced (0.013%) 2024-03-24T12:41:44.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:43 smithi029 ceph-mon[29919]: osdmap e290: 8 total, 7 up, 8 in 2024-03-24T12:41:44.117 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:43 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:41:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:43 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:41:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:43 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:41:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:43 smithi190 ceph-mon[33748]: pgmap v657: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.4 KiB/s rd, 109 MiB/s wr, 1.74k op/s; 20749/202644 objects degraded (10.239%); 26/202644 objects misplaced (0.013%) 2024-03-24T12:41:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:43 smithi190 ceph-mon[33748]: osdmap e290: 8 total, 7 up, 8 in 2024-03-24T12:41:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:43 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:41:44.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:44.258 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:44.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:44.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:41:44.286554+0000 min lat: 0.00126577 max lat: 0.383581 avg lat: 0.00902468 2024-03-24T12:41:44.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:41:44.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 40 16 70908 70892 110.75 76.25 0.0103204 0.00902468 2024-03-24T12:41:44.359 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:44.385 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:44.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:44.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:44.526 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:44.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:44.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:44.583 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:41:44.584 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:41:44.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:44.680 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:44.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:44.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:41:44.783 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:44.813 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:44.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:44.877 INFO:teuthology.orchestra.run.smithi029.stderr:set pool 1 pg_num to 11 2024-03-24T12:41:44.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:44.965 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:44.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:44.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:45.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:45.099 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:45.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:44 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/574152392' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:41:45.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:44 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:41:45.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:41:45.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:41:45.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:41:45.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:44 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/574152392' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:41:45.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:44 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:41:45.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:41:45.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:41:45.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:41:45.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:44 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/574152392' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:41:45.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:44 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-03-24T12:41:45.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:41:45.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:41:45.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:41:45.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:45.240 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:45.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:45.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:45.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 41 16 71979 71963 109.682 66.9375 0.00298425 0.00911033 2024-03-24T12:41:45.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:45.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:45.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:45.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:45.524 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:45.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:45.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:45.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:41:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:45.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:45.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:45.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:45.779 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:45.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:45.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:45.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:45.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:45.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:46.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:46.064 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:46.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:45 smithi190 ceph-mon[33748]: pgmap v659: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s; 21579/206241 objects degraded (10.463%); 26/206241 objects misplaced (0.013%) 2024-03-24T12:41:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:45 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T12:41:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:45 smithi190 ceph-mon[33748]: osdmap e291: 8 total, 7 up, 8 in 2024-03-24T12:41:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:45 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 21579/206241 objects degraded (10.463%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:46.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:45 smithi190 ceph-mon[33748]: osdmap e292: 8 total, 7 up, 8 in 2024-03-24T12:41:46.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:46.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:45 smithi029 ceph-mon[41500]: pgmap v659: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s; 21579/206241 objects degraded (10.463%); 26/206241 objects misplaced (0.013%) 2024-03-24T12:41:46.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:45 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T12:41:46.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:45 smithi029 ceph-mon[41500]: osdmap e291: 8 total, 7 up, 8 in 2024-03-24T12:41:46.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:45 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 21579/206241 objects degraded (10.463%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:45 smithi029 ceph-mon[41500]: osdmap e292: 8 total, 7 up, 8 in 2024-03-24T12:41:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:45 smithi029 ceph-mon[29919]: pgmap v659: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 85 MiB/s wr, 1.36k op/s; 21579/206241 objects degraded (10.463%); 26/206241 objects misplaced (0.013%) 2024-03-24T12:41:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:45 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-03-24T12:41:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:45 smithi029 ceph-mon[29919]: osdmap e291: 8 total, 7 up, 8 in 2024-03-24T12:41:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:45 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 21579/206241 objects degraded (10.463%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:46.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:45 smithi029 ceph-mon[29919]: osdmap e292: 8 total, 7 up, 8 in 2024-03-24T12:41:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:46.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:46.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 42 16 73124 73108 108.774 71.5625 0.00369741 0.00918832 2024-03-24T12:41:46.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:46.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:46.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:46.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:46.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:46.463 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:46.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:46.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:46.605 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:46.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:46.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:46.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:46.737 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:46.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:46.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:46.881 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:46.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:46.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:46.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:47.017 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:47.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:47.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:47.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:47.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:47.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:47.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T12:41:47.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T12:41:47.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:47.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:47.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:47.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:47.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 43 16 73887 73871 107.353 47.6875 0.015074 0.00928853 2024-03-24T12:41:47.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-03-24T12:41:47.409 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:47.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:47.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:47.551 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:47.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:47.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:47.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:47.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:47.794 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:47.827 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:47.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:47.892 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:41:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:41:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:41:47.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:47.966 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:47.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:47.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:48.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:48.091 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:48.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:47 smithi029 ceph-mon[41500]: pgmap v662: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 21957/210873 objects degraded (10.412%); 26/210873 objects misplaced (0.012%) 2024-03-24T12:41:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T12:41:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:47 smithi029 ceph-mon[41500]: osdmap e293: 8 total, 7 up, 8 in 2024-03-24T12:41:48.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:47 smithi029 ceph-mon[29919]: pgmap v662: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 21957/210873 objects degraded (10.412%); 26/210873 objects misplaced (0.012%) 2024-03-24T12:41:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T12:41:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:47 smithi029 ceph-mon[29919]: osdmap e293: 8 total, 7 up, 8 in 2024-03-24T12:41:48.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:48.233 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:48.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:48.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:48.252 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:47 smithi190 ceph-mon[33748]: pgmap v662: 148 pgs: 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.0 KiB/s rd, 85 MiB/s wr, 1.36k op/s; 21957/210873 objects degraded (10.412%); 26/210873 objects misplaced (0.012%) 2024-03-24T12:41:48.252 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-03-24T12:41:48.253 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:47 smithi190 ceph-mon[33748]: osdmap e293: 8 total, 7 up, 8 in 2024-03-24T12:41:48.282 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 44 16 75093 75077 106.626 75.375 0.00885283 0.00937298 2024-03-24T12:41:48.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:48.377 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:48.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:48.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:48.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:48.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:48.503 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:48.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:48.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:48.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:41:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:41:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:48.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:48.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:48.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:48.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:48.785 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:48.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:48.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:48.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:48.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:49.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:49.042 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:49.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:49.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:49.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:48 smithi029 ceph-mon[41500]: osdmap e294: 8 total, 7 up, 8 in 2024-03-24T12:41:49.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:49.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T12:41:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:48 smithi029 ceph-mon[29919]: osdmap e294: 8 total, 7 up, 8 in 2024-03-24T12:41:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:49.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T12:41:49.171 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:49.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:49.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:49.273 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:49.281 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 45 16 76216 76200 105.816 70.1875 0.0115756 0.00944576 2024-03-24T12:41:49.316 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:49.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:49.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:48 smithi190 ceph-mon[33748]: osdmap e294: 8 total, 7 up, 8 in 2024-03-24T12:41:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:41:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-03-24T12:41:49.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:49.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:49.443 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:49.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:49.544 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:49.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:49.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:49.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:49.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:49.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:49.816 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:49.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:49.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:49.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:49.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:49.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:49.984 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:49.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:50.085 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:50.109 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:50.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:50.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:50.251 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:50.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:50.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:50.282 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 46 16 77301 77285 104.99 67.8125 0.00702432 0.00951963 2024-03-24T12:41:50.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:50.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:50.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:49 smithi190 ceph-mon[33748]: pgmap v665: 158 pgs: 1 peering, 10 unknown, 17 active+clean+remapped, 95 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 22641/220422 objects degraded (10.272%); 26/220422 objects misplaced (0.012%) 2024-03-24T12:41:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T12:41:50.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:49 smithi190 ceph-mon[33748]: osdmap e295: 8 total, 7 up, 8 in 2024-03-24T12:41:50.396 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:50.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:50.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:49 smithi029 ceph-mon[29919]: pgmap v665: 158 pgs: 1 peering, 10 unknown, 17 active+clean+remapped, 95 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 22641/220422 objects degraded (10.272%); 26/220422 objects misplaced (0.012%) 2024-03-24T12:41:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T12:41:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:49 smithi029 ceph-mon[29919]: osdmap e295: 8 total, 7 up, 8 in 2024-03-24T12:41:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:41:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:49 smithi029 ceph-mon[41500]: pgmap v665: 158 pgs: 1 peering, 10 unknown, 17 active+clean+remapped, 95 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 22641/220422 objects degraded (10.272%); 26/220422 objects misplaced (0.012%) 2024-03-24T12:41:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-03-24T12:41:50.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:49 smithi029 ceph-mon[41500]: osdmap e295: 8 total, 7 up, 8 in 2024-03-24T12:41:50.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:50.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:50.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:50.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:50.700 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:50.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:50.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:50.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:50.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:50.836 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:50.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:50.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:50.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:50.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:50.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:51.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:51.110 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:51.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:51.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:50 smithi029 ceph-mon[29919]: osdmap e296: 8 total, 7 up, 8 in 2024-03-24T12:41:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:50 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 22641/220422 objects degraded (10.272%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:50 smithi029 ceph-mon[29919]: osdmap e297: 8 total, 7 up, 8 in 2024-03-24T12:41:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:50 smithi029 ceph-mon[41500]: osdmap e296: 8 total, 7 up, 8 in 2024-03-24T12:41:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:50 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 22641/220422 objects degraded (10.272%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:50 smithi029 ceph-mon[41500]: osdmap e297: 8 total, 7 up, 8 in 2024-03-24T12:41:51.212 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:51.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:51.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:51.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:51.282 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 47 16 78675 78659 104.583 85.875 0.00441942 0.00955562 2024-03-24T12:41:51.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:51.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:50 smithi190 ceph-mon[33748]: osdmap e296: 8 total, 7 up, 8 in 2024-03-24T12:41:51.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:50 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 22641/220422 objects degraded (10.272%), 13 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:51.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:50 smithi190 ceph-mon[33748]: osdmap e297: 8 total, 7 up, 8 in 2024-03-24T12:41:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:51.384 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:51.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:51.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T12:41:51.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:51.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:51.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:51.706 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:51.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:51.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:51.863 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:51.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:51.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:52.026 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:52.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:52.030 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:41:52.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:52.161 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:52.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:52.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:52 smithi029 ceph-mon[29919]: pgmap v668: 158 pgs: 1 peering, 10 unknown, 17 active+clean+remapped, 95 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 23302/224325 objects degraded (10.388%); 26/224325 objects misplaced (0.012%) 2024-03-24T12:41:52.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:52 smithi029 ceph-mon[29919]: osdmap e298: 8 total, 7 up, 8 in 2024-03-24T12:41:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:52 smithi029 ceph-mon[41500]: pgmap v668: 158 pgs: 1 peering, 10 unknown, 17 active+clean+remapped, 95 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 23302/224325 objects degraded (10.388%); 26/224325 objects misplaced (0.012%) 2024-03-24T12:41:52.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:52 smithi029 ceph-mon[41500]: osdmap e298: 8 total, 7 up, 8 in 2024-03-24T12:41:52.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:52.282 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 48 16 79900 79884 103.999 76.5625 0.00796658 0.00961062 2024-03-24T12:41:52.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:52.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:52.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:52.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:52.430 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:52.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:52.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:52.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:52.575 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:52.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:52 smithi190 ceph-mon[33748]: pgmap v668: 158 pgs: 1 peering, 10 unknown, 17 active+clean+remapped, 95 active+clean, 22 active+undersized, 13 active+undersized+degraded; 4.6 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 70 MiB/s wr, 1.11k op/s; 23302/224325 objects degraded (10.388%); 26/224325 objects misplaced (0.012%) 2024-03-24T12:41:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:52 smithi190 ceph-mon[33748]: osdmap e298: 8 total, 7 up, 8 in 2024-03-24T12:41:52.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:52.714 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:52.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:52.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:52.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:52.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:52.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:52.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:52.997 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:52.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:52.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:53.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:53.137 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:53.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:53.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:53.239 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:53.268 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:53.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:53.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:53.282 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 49 16 81126 81110 103.441 76.625 0.00956444 0.00966259 2024-03-24T12:41:53.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:53.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:53.421 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:53.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:53.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:53.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:53.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:53.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:53.659 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:53.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:53.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:53.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:53.838 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:53.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:53.939 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:53.971 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:53.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:53.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:54.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:54.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:54.115 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:54.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:54.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:54.248 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:54.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:54.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:54.282 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 50 16 82498 82482 103.087 85.75 0.00902724 0.00969577 2024-03-24T12:41:54.324 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:53 smithi029 ceph-mon[41500]: pgmap v671: 158 pgs: 1 remapped+peering, 6 peering, 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 3 active+undersized+remapped, 13 active+undersized+degraded; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s; 24450/237636 objects degraded (10.289%); 27/237636 objects misplaced (0.011%) 2024-03-24T12:41:54.324 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:53 smithi029 ceph-mon[41500]: osdmap e299: 8 total, 7 up, 8 in 2024-03-24T12:41:54.324 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:53 smithi029 ceph-mon[29919]: pgmap v671: 158 pgs: 1 remapped+peering, 6 peering, 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 3 active+undersized+remapped, 13 active+undersized+degraded; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s; 24450/237636 objects degraded (10.289%); 27/237636 objects misplaced (0.011%) 2024-03-24T12:41:54.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:53 smithi029 ceph-mon[29919]: osdmap e299: 8 total, 7 up, 8 in 2024-03-24T12:41:54.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:54.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:53 smithi190 ceph-mon[33748]: pgmap v671: 158 pgs: 1 remapped+peering, 6 peering, 17 active+clean+remapped, 96 active+clean, 22 active+undersized, 3 active+undersized+remapped, 13 active+undersized+degraded; 4.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 90 MiB/s wr, 1.43k op/s; 24450/237636 objects degraded (10.289%); 27/237636 objects misplaced (0.011%) 2024-03-24T12:41:54.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:53 smithi190 ceph-mon[33748]: osdmap e299: 8 total, 7 up, 8 in 2024-03-24T12:41:54.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:54.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:54.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:54.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:54.529 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:54.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:54.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:54.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:54.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:54.675 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:54.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:54.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:54.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:54.801 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:54.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:54.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:54.932 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:54.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:55.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:55.071 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:55.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:55.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:55.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:55.213 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:55.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:55.282 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 51 15 83469 83454 102.257 60.75 0.0109274 0.00977482 2024-03-24T12:41:55.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:55.351 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:55.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:55.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:55.489 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:55.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:55.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:55.591 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:55.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:55.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:55.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:55.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:55.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:55.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:55.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:55.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:55.888 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:55.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:55.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:56.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:56.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:56.125 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:56.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:56.163 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:56.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:56.264 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:56.282 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 52 16 84685 84669 101.75 75.9375 0.0146041 0.00982309 2024-03-24T12:41:56.295 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:56.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:56.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:56.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:55 smithi190 ceph-mon[33748]: pgmap v673: 158 pgs: 1 remapped+peering, 6 peering, 19 active+clean+remapped, 96 active+clean, 22 active+undersized, 1 active+undersized+remapped, 13 active+undersized+degraded; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 11 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 25224/241185 objects degraded (10.458%); 27/241185 objects misplaced (0.011%); 34 KiB/s, 0 objects/s recovering 2024-03-24T12:41:56.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:55 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 25224/241185 objects degraded (10.458%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:56.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:55 smithi190 ceph-mon[33748]: osdmap e300: 8 total, 7 up, 8 in 2024-03-24T12:41:56.397 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:55 smithi029 ceph-mon[29919]: pgmap v673: 158 pgs: 1 remapped+peering, 6 peering, 19 active+clean+remapped, 96 active+clean, 22 active+undersized, 1 active+undersized+remapped, 13 active+undersized+degraded; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 11 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 25224/241185 objects degraded (10.458%); 27/241185 objects misplaced (0.011%); 34 KiB/s, 0 objects/s recovering 2024-03-24T12:41:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:55 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 25224/241185 objects degraded (10.458%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:56.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:55 smithi029 ceph-mon[29919]: osdmap e300: 8 total, 7 up, 8 in 2024-03-24T12:41:56.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:55 smithi029 ceph-mon[41500]: pgmap v673: 158 pgs: 1 remapped+peering, 6 peering, 19 active+clean+remapped, 96 active+clean, 22 active+undersized, 1 active+undersized+remapped, 13 active+undersized+degraded; 4.9 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 11 KiB/s rd, 73 MiB/s wr, 1.18k op/s; 25224/241185 objects degraded (10.458%); 27/241185 objects misplaced (0.011%); 34 KiB/s, 0 objects/s recovering 2024-03-24T12:41:56.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:55 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 25224/241185 objects degraded (10.458%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:41:56.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:55 smithi029 ceph-mon[41500]: osdmap e300: 8 total, 7 up, 8 in 2024-03-24T12:41:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:56.450 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:56.450 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:56.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:56.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:56.593 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:56.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:56.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:56.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:56.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:56.878 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:56.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:56.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:56.960 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:41:56.960 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:41:56.960 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-24T12:41:56.960 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-03-24T12:41:56.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-24T12:41:56.980 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:57.005 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:56 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:41:57.005 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:56 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:41:57.005 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:56 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:41:57.005 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:56 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:41:57.044 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:57.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:57.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:57.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:57.175 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:57.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:57.254 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:41:57.254 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:41:57.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:57.282 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 53 16 85895 85879 101.257 75.625 0.0182356 0.00987142 2024-03-24T12:41:57.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:57.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:57.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:57.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:56 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:41:57.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:56 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:41:57.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:57.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:57.446 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:57.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:57.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:57.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:57.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:57.691 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:57.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:57.751 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:57.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:57.804 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:41:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:41:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:41:57.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:57.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:57.903 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:57.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:58.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:58.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:58.040 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:58.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:58.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:41:58.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:57 smithi029 ceph-mon[41500]: pgmap v675: 158 pgs: 1 remapped+peering, 6 peering, 20 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 9.4 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 25596/245748 objects degraded (10.416%); 27/245748 objects misplaced (0.011%); 28 KiB/s, 0 objects/s recovering 2024-03-24T12:41:58.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:57 smithi029 ceph-mon[41500]: osdmap e301: 8 total, 7 up, 8 in 2024-03-24T12:41:58.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:57 smithi029 ceph-mon[29919]: pgmap v675: 158 pgs: 1 remapped+peering, 6 peering, 20 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 9.4 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 25596/245748 objects degraded (10.416%); 27/245748 objects misplaced (0.011%); 28 KiB/s, 0 objects/s recovering 2024-03-24T12:41:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:57 smithi029 ceph-mon[29919]: osdmap e301: 8 total, 7 up, 8 in 2024-03-24T12:41:58.175 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:58.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:41:58.262 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:57 smithi190 ceph-mon[33748]: pgmap v675: 158 pgs: 1 remapped+peering, 6 peering, 20 active+clean+remapped, 96 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 9.4 KiB/s rd, 77 MiB/s wr, 1.24k op/s; 25596/245748 objects degraded (10.416%); 27/245748 objects misplaced (0.011%); 28 KiB/s, 0 objects/s recovering 2024-03-24T12:41:58.263 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:57 smithi190 ceph-mon[33748]: osdmap e301: 8 total, 7 up, 8 in 2024-03-24T12:41:58.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:58.283 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 54 16 87210 87194 100.902 82.1875 0.0152274 0.00990553 2024-03-24T12:41:58.320 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:58.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:58.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:58.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:58.459 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:58.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:58.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:58.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:58.604 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:58.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:58.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:41:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:41:58.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:58.730 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:58.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:58.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:41:58.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:58.874 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:58.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:41:58.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:59.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:59.016 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:59.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:59.117 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:59.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:59.164 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:59.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:59.265 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:41:59.284 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 55 16 88405 88389 100.425 74.6875 0.0129255 0.00995275 2024-03-24T12:41:59.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:59.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:41:59.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:41:59.443 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:59.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:59.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:41:59.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:41:59.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:41:59.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:59.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:41:59.678 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:41:59.718 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:59.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:59.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:41:59.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:41:59.857 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:41:59.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:41:59.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:41:59.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:00.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:00.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:00.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T12:42:00.104 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:00.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:00.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:00.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:00.263 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:00.263 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:41:59 smithi029 ceph-mon[41500]: pgmap v677: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s; 26359/256086 objects degraded (10.293%); 27/256086 objects misplaced (0.011%) 2024-03-24T12:42:00.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:00.263 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:41:59 smithi029 ceph-mon[29919]: pgmap v677: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s; 26359/256086 objects degraded (10.293%); 27/256086 objects misplaced (0.011%) 2024-03-24T12:42:00.284 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 56 16 89680 89664 100.055 79.6875 0.0188823 0.00998992 2024-03-24T12:42:00.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:00.287 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:00.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:41:59 smithi190 ceph-mon[33748]: pgmap v677: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 64 MiB/s wr, 1.02k op/s; 26359/256086 objects degraded (10.293%); 27/256086 objects misplaced (0.011%) 2024-03-24T12:42:00.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:00.438 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:00.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:00.539 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:00.568 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:00.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:00.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:00.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:00.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:00.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:00.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:00.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:00.860 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:00.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:00.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:01.007 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:01.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:01.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:01.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:01.152 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:01.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:01.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:00 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 26359/256086 objects degraded (10.293%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:01.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:00 smithi029 ceph-mon[29919]: osdmap e302: 8 total, 7 up, 8 in 2024-03-24T12:42:01.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:00 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 26359/256086 objects degraded (10.293%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:01.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:00 smithi029 ceph-mon[41500]: osdmap e302: 8 total, 7 up, 8 in 2024-03-24T12:42:01.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:01.289 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 57 16 90852 90836 99.5781 73.25 0.00637314 0.0100374 2024-03-24T12:42:01.298 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:01.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:00 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 26359/256086 objects degraded (10.293%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:00 smithi190 ceph-mon[33748]: osdmap e302: 8 total, 7 up, 8 in 2024-03-24T12:42:01.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:01.438 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:01.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:01.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:01.540 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:01.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:01.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:01.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:01.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:01.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:01.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:01.718 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:42:01.759 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:42:01.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:01.806 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:42:01.829 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:01.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:01.856 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:42:01.897 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:42:01.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:01.948 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:42:01.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:01.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:01.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:01.993 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:42:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:42:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:42:01.993 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:42:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:42:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:42:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:42:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:42:01.994 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:42:02.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:02.110 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:02.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:02.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:02.173 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:42:02.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:02.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:02.250 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:02.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:02.288 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 58 16 92019 92003 99.1187 72.9375 0.00869548 0.0100842 2024-03-24T12:42:02.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:02.389 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:02.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:02.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:02.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:02 smithi029 ceph-mon[29919]: pgmap v679: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s; 27116/260691 objects degraded (10.402%); 27/260691 objects misplaced (0.010%) 2024-03-24T12:42:02.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:02 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2679439851' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:42:02.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:02 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:42:02.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:02 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/413784871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:42:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:02 smithi029 ceph-mon[41500]: pgmap v679: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s; 27116/260691 objects degraded (10.402%); 27/260691 objects misplaced (0.010%) 2024-03-24T12:42:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:02 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2679439851' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:42:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:02 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:42:02.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:02 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/413784871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:42:02.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:02.533 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:02.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:02.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:02 smithi190 ceph-mon[33748]: pgmap v679: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.3 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.08k op/s; 27116/260691 objects degraded (10.402%); 27/260691 objects misplaced (0.010%) 2024-03-24T12:42:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:02 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2679439851' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:42:02.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:02 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:42:02.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:02 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/413784871' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:42:02.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:02.679 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:02.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:02.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:02.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:02.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:02.823 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:02.923 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:02.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:02.954 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:02.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:03.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:03.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:03.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:03.191 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:03.231 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:03.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:03.291 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 59 15 93300 93285 98.7943 80.125 0.00985016 0.0101173 2024-03-24T12:42:03.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:03.374 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:03.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:03.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:03.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:42:03.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:42:03.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[29919]: osdmap e303: 8 total, 7 up, 8 in 2024-03-24T12:42:03.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:42:03.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:42:03.395 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:03.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:42:03.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:42:03.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[41500]: osdmap e303: 8 total, 7 up, 8 in 2024-03-24T12:42:03.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:42:03.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:42:03.396 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:03.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:03.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:03.514 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:03.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:03.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:03 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:42:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:03 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:42:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:03 smithi190 ceph-mon[33748]: osdmap e303: 8 total, 7 up, 8 in 2024-03-24T12:42:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:03 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:42:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:03 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:42:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:03.644 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:03.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:03.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:03.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:03.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:03.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:03.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:04.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:04.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:04.074 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:04.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:04.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:04.216 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:42:04.216 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:42:04.219 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:04.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:04.290 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:42:04.295092+0000 min lat: 0.00126577 max lat: 0.383581 avg lat: 0.0101508 2024-03-24T12:42:04.290 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:42:04.290 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 60 16 94571 94555 98.4698 79.375 0.0140414 0.0101508 2024-03-24T12:42:04.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:04.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:04.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:04.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[41500]: pgmap v681: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 28279/274218 objects degraded (10.313%); 27/274218 objects misplaced (0.010%) 2024-03-24T12:42:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:42:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:42:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[41500]: 1.1 scrub starts 2024-03-24T12:42:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[41500]: 1.1 scrub ok 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[41500]: 1.3 scrub starts 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[41500]: 1.3 scrub ok 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[29919]: pgmap v681: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 28279/274218 objects degraded (10.313%); 27/274218 objects misplaced (0.010%) 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[29919]: 1.1 scrub starts 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[29919]: 1.1 scrub ok 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:42:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:42:04.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[29919]: 1.3 scrub starts 2024-03-24T12:42:04.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:04 smithi029 ceph-mon[29919]: 1.3 scrub ok 2024-03-24T12:42:04.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:04.491 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:04.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:04.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:04.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:04.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:04.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:04.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:04 smithi190 ceph-mon[33748]: pgmap v681: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 99 MiB/s wr, 1.58k op/s; 28279/274218 objects degraded (10.313%); 27/274218 objects misplaced (0.010%) 2024-03-24T12:42:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:04 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:42:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:04 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:42:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:04 smithi190 ceph-mon[33748]: 1.1 scrub starts 2024-03-24T12:42:04.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:04 smithi190 ceph-mon[33748]: 1.1 scrub ok 2024-03-24T12:42:04.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:04 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:42:04.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:04 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:42:04.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:04 smithi190 ceph-mon[33748]: 1.3 scrub starts 2024-03-24T12:42:04.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:04 smithi190 ceph-mon[33748]: 1.3 scrub ok 2024-03-24T12:42:04.726 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:04.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:04.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:04.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:04.909 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:04.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:05.010 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:05.053 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:05.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:05.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:05.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:42:05.216 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:05.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:05.290 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 61 16 95833 95817 98.1485 78.875 0.0109085 0.0101838 2024-03-24T12:42:05.319 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:05.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:05.377 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:05.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:05.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: 1.a scrub starts 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: 1.a scrub ok 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: 1.4 scrub starts 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: 1.4 scrub ok 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: 3.e scrub starts 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: 3.e scrub ok 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: 1.6 scrub starts 2024-03-24T12:42:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: 1.6 scrub ok 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 28279/274218 objects degraded (10.313%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[29919]: osdmap e304: 8 total, 7 up, 8 in 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: 1.a scrub starts 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: 1.a scrub ok 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: 1.4 scrub starts 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: 1.4 scrub ok 2024-03-24T12:42:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: 3.e scrub starts 2024-03-24T12:42:05.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: 3.e scrub ok 2024-03-24T12:42:05.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: 1.6 scrub starts 2024-03-24T12:42:05.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: 1.6 scrub ok 2024-03-24T12:42:05.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 28279/274218 objects degraded (10.313%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:05.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:05 smithi029 ceph-mon[41500]: osdmap e304: 8 total, 7 up, 8 in 2024-03-24T12:42:05.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:05.527 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:05.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:05.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: 1.a scrub starts 2024-03-24T12:42:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: 1.a scrub ok 2024-03-24T12:42:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: 1.4 scrub starts 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: 1.4 scrub ok 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: 3.e scrub starts 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: 3.e scrub ok 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: 1.6 scrub starts 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: 1.6 scrub ok 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 28279/274218 objects degraded (10.313%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:05.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:05 smithi190 ceph-mon[33748]: osdmap e304: 8 total, 7 up, 8 in 2024-03-24T12:42:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:05.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:05.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:05.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:05.807 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:05.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:05.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:05.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:05.941 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:05.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:05.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:06.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:06.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:06.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:06.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:06.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:42:06.257 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:06.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:06.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:06.291 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 62 16 97098 97082 97.838 79.0625 0.0108629 0.010216 2024-03-24T12:42:06.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: 1.5 scrub starts 2024-03-24T12:42:06.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: 1.5 scrub ok 2024-03-24T12:42:06.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: pgmap v682: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 29013/277656 objects degraded (10.449%); 27/277656 objects misplaced (0.010%) 2024-03-24T12:42:06.305 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: 1.8 scrub starts 2024-03-24T12:42:06.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: 1.8 scrub ok 2024-03-24T12:42:06.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: 4.6 scrub starts 2024-03-24T12:42:06.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: 4.6 scrub ok 2024-03-24T12:42:06.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: 1.2 scrub starts 2024-03-24T12:42:06.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: 1.2 scrub ok 2024-03-24T12:42:06.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: 1.7 scrub starts 2024-03-24T12:42:06.306 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[29919]: 1.7 scrub ok 2024-03-24T12:42:06.306 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: 1.5 scrub starts 2024-03-24T12:42:06.306 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: 1.5 scrub ok 2024-03-24T12:42:06.306 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: pgmap v682: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 29013/277656 objects degraded (10.449%); 27/277656 objects misplaced (0.010%) 2024-03-24T12:42:06.307 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: 1.8 scrub starts 2024-03-24T12:42:06.307 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: 1.8 scrub ok 2024-03-24T12:42:06.307 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: 4.6 scrub starts 2024-03-24T12:42:06.307 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: 4.6 scrub ok 2024-03-24T12:42:06.307 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: 1.2 scrub starts 2024-03-24T12:42:06.307 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: 1.2 scrub ok 2024-03-24T12:42:06.307 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: 1.7 scrub starts 2024-03-24T12:42:06.307 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:06 smithi029 ceph-mon[41500]: 1.7 scrub ok 2024-03-24T12:42:06.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:06.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:06.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:06.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:06.523 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:06.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:06.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:06.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: 1.5 scrub starts 2024-03-24T12:42:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: 1.5 scrub ok 2024-03-24T12:42:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: pgmap v682: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.6 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 85 MiB/s wr, 1.36k op/s; 29013/277656 objects degraded (10.449%); 27/277656 objects misplaced (0.010%) 2024-03-24T12:42:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: 1.8 scrub starts 2024-03-24T12:42:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: 1.8 scrub ok 2024-03-24T12:42:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: 4.6 scrub starts 2024-03-24T12:42:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: 4.6 scrub ok 2024-03-24T12:42:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: 1.2 scrub starts 2024-03-24T12:42:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: 1.2 scrub ok 2024-03-24T12:42:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: 1.7 scrub starts 2024-03-24T12:42:06.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:06 smithi190 ceph-mon[33748]: 1.7 scrub ok 2024-03-24T12:42:06.685 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:06.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:06.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:06.787 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:06.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:06.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:06.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:06.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:06.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:07.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:07.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:07.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:07.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:07.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:07.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:07.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:07.275 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:42:07.275 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:42:07.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:42:07.275 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:2087 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 93828934257920 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:58.114918+0000", 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 369.15119355899998, 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.64108174799999995, 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 2087 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.276 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114918+0000", 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114918+0000", 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114920+0000", 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8369999999999999e-06 2024-03-24T12:42:07.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114926+0000", 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4899999999999997e-06 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114939+0000", 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3174e-05 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.114952+0000", 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2690999999999999e-05 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.115027+0000", 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5133e-05 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.115039+0000", 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1575e-05 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.752611+0000", 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.63757248899999996 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.752807+0000", 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019623899999999999 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.755999+0000", 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0031921200000000001 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.280 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:2093 5.d 5:b4812045:::notify.4:head [watch ping cookie 93828934257920 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:58.327350+0000", 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 368.93876103399998, 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42710226000000001, 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:42:07.281 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 2093 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327350+0000", 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327350+0000", 2024-03-24T12:42:07.282 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327352+0000", 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2640000000000001e-06 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327358+0000", 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2260000000000002e-06 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.283 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327377+0000", 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9695999999999999e-05 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327390+0000", 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2699e-05 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327483+0000", 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2781000000000005e-05 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.284 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.327495+0000", 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2376e-05 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.754015+0000", 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.42651926600000001 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.754101+0000", 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6005000000000001e-05 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.754453+0000", 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00035194700000000002 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:166298 9.6 9:6422e0d4:::benchmark_data_smithi029_134815_object54232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e251)", 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:28.027363+0000", 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 159.23874839699999, 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40582715800000002, 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.286 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 166298 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.027363+0000", 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.027363+0000", 2024-03-24T12:42:07.287 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.027364+0000", 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4699999999999999e-06 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.027370+0000", 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6629999999999998e-06 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.027382+0000", 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2106000000000001e-05 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.027391+0000", 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2629999999999992e-06 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.027456+0000", 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4145000000000003e-05 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.289 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.027569+0000", 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113087 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1", 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.027677+0000", 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000108758 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.428161+0000", 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400483431 2024-03-24T12:42:07.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.432684+0000", 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0045229809999999997 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.433091+0000", 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00040700799999999999 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.433190+0000", 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9246000000000005e-05 2024-03-24T12:42:07.291 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.24989.0:166294 9.0 e251/248 9:02444622:::benchmark_data_smithi029_134815_object54228:head v 251'10280, mlcod=251'10278)", 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:28.029344+0000", 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 159.23676751299999, 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.41083527600000003, 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.292 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.029344+0000", 2024-03-24T12:42:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.029344+0000", 2024-03-24T12:42:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.293 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.029345+0000", 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4499999999999996e-07 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.029352+0000", 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3640000000000001e-06 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.029362+0000", 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0251000000000001e-05 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.029370+0000", 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4309999999999997e-06 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.427762+0000", 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.39839258399999999 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.427828+0000", 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5685999999999994e-05 2024-03-24T12:42:07.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.440137+0000", 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012308754 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.440179+0000", 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2361000000000003e-05 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.296 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.24989.0:166299 9.0 e251/248 9:0570cc3e:::benchmark_data_smithi029_134815_object54233:head v 251'10281, mlcod=251'10278)", 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:28.031696+0000", 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 159.23441518800001, 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.41040410399999999, 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.031696+0000", 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.297 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.031696+0000", 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.031697+0000", 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.093e-06 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.298 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.031704+0000", 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6889999999999998e-06 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.427715+0000", 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.39601135300000001 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.427744+0000", 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.8748999999999999e-05 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.299 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.433116+0000", 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0053724840000000003 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.433139+0000", 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2140000000000001e-05 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.442058+0000", 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0089193669999999992 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.300 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:28.442100+0000", 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2228999999999998e-05 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:29177 10.9 10:98d6d878:::benchmark_data_smithi029_160664_object29176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:16.139132+0000", 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 51.126979386000002, 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20819110699999999, 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.301 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 29177 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139132+0000", 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.302 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139132+0000", 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139133+0000", 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1299999999999999e-07 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139148+0000", 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5858999999999998e-05 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.303 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.304 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139157+0000", 2024-03-24T12:42:07.304 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9269999999999994e-06 2024-03-24T12:42:07.304 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.304 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.304 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.304 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139163+0000", 2024-03-24T12:42:07.304 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0279999999999997e-06 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139208+0000", 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4484000000000003e-05 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139345+0000", 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013668299999999999 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 7", 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139471+0000", 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012629399999999999 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.339815+0000", 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.200344155 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.347184+0000", 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0073685679999999998 2024-03-24T12:42:07.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.347238+0000", 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4688999999999998e-05 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.347323+0000", 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4707000000000001e-05 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.307 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:29178 10.3 10:c39f5778:::benchmark_data_smithi029_160664_object29177:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:16.139319+0000", 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 51.126792076000001, 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20381539600000001, 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 29178 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.308 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139319+0000", 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139319+0000", 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139320+0000", 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3600000000000003e-07 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.309 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139336+0000", 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6481e-05 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139344+0000", 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7589999999999997e-06 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139349+0000", 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2939999999999996e-06 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.310 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139401+0000", 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1589999999999999e-05 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139528+0000", 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012697399999999999 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3", 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.337210+0000", 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.197681792 2024-03-24T12:42:07.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.342821+0000", 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0056111640000000001 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.342984+0000", 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000163521 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.343038+0000", 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3761999999999998e-05 2024-03-24T12:42:07.312 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.343135+0000", 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6422999999999995e-05 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:29176 10.e e286/278 10:7e28bbb1:::benchmark_data_smithi029_160664_object29175:head v 286'1830, mlcod=286'1828)", 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:16.139397+0000", 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 51.126713999000003, 2024-03-24T12:42:07.313 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20275041199999999, 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139397+0000", 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139397+0000", 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.314 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139398+0000", 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3300000000000003e-07 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139464+0000", 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6082999999999997e-05 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139470+0000", 2024-03-24T12:42:07.315 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6819999999999997e-06 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139476+0000", 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6280000000000002e-06 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.139530+0000", 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4289000000000001e-05 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.316 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.337082+0000", 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.19755183500000001 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.342086+0000", 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0050037689999999999 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:16.342148+0000", 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1692999999999994e-05 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.317 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.318 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.318 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:07.318 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:34682 10.9 10:9d6816e8:::benchmark_data_smithi029_160664_object34681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:19.593553+0000", 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.672558690999999, 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.238692823, 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 34682 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.319 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593553+0000", 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593553+0000", 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593554+0000", 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9000000000000005e-07 2024-03-24T12:42:07.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593572+0000", 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8668000000000002e-05 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593581+0000", 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8049999999999996e-06 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593587+0000", 2024-03-24T12:42:07.321 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9399999999999999e-06 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593665+0000", 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7783999999999999e-05 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593774+0000", 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000109201 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 7", 2024-03-24T12:42:07.322 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593867+0000", 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2690999999999997e-05 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595391+0000", 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001523827 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.832091+0000", 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23670010399999999 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.323 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.832150+0000", 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9447000000000002e-05 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.832245+0000", 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5365999999999995e-05 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:34683 10.5 10:a986ec90:::benchmark_data_smithi029_160664_object34682:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:42:07.324 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:19.593863+0000", 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.672248013999997, 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.241747663, 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 34683 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593863+0000", 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593863+0000", 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593864+0000", 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4099999999999997e-07 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.326 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593886+0000", 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1708e-05 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593897+0000", 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1058e-05 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593905+0000", 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2039999999999997e-06 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.327 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.593974+0000", 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8646000000000006e-05 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.594095+0000", 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000121063 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.594232+0000", 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013672399999999999 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.835397+0000", 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24116523200000001 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.835461+0000", 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3836000000000003e-05 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.835509+0000", 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8900000000000003e-05 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.835568+0000", 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8641000000000003e-05 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.835611+0000", 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2809999999999998e-05 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.330 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:34687 10.5 10:aeb3c442:::benchmark_data_smithi029_160664_object34686:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:19.594988+0000", 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.671123567000002, 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24310442700000001, 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 34687 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.331 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.332 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.594988+0000", 2024-03-24T12:42:07.332 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.332 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.332 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.332 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.332 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 63 16 98163 98147 97.3415 66.5625 0.019792 0.0102672 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.594988+0000", 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.594988+0000", 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0299999999999998e-07 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595006+0000", 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7144999999999999e-05 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595014+0000", 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6260000000000001e-06 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595020+0000", 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.959e-06 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.334 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595567+0000", 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00054701399999999996 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.832003+0000", 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23643546100000001 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.832141+0000", 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013809200000000001 2024-03-24T12:42:07.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.836115+0000", 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0039744350000000001 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.836130+0000", 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4603000000000001e-05 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.837956+0000", 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001826591 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.838022+0000", 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5843e-05 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.838092+0000", 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9955000000000004e-05 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:34690 10.5 10:abd501c4:::benchmark_data_smithi029_160664_object34689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:19.595483+0000", 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.670628594999997, 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24272608100000001, 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 34690 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595483+0000", 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595483+0000", 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595484+0000", 2024-03-24T12:42:07.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1399999999999996e-07 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595503+0000", 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9820999999999999e-05 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595511+0000", 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7039999999999994e-06 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.340 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595516+0000", 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1970000000000004e-06 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.833620+0000", 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23810379600000001 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.833755+0000", 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013494700000000001 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.833920+0000", 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016540399999999999 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.836138+0000", 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0022176219999999998 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.836454+0000", 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00031607299999999999 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.838131+0000", 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0016773280000000001 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.838175+0000", 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3961000000000001e-05 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.838209+0000", 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.3414000000000002e-05 2024-03-24T12:42:07.343 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:34680 10.f e286/278 10:f9fae839:::benchmark_data_smithi029_160664_object34679:head v 286'2199, mlcod=286'2197)", 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:19.595504+0000", 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.670607599999997, 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24127831199999999, 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595504+0000", 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595504+0000", 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595505+0000", 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1399999999999995e-07 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.345 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595540+0000", 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.5849999999999997e-05 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595550+0000", 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4460000000000007e-06 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595557+0000", 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7100000000000001e-06 2024-03-24T12:42:07.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.831932+0000", 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.236375057 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.831974+0000", 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2661999999999999e-05 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.836655+0000", 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0046803440000000003 2024-03-24T12:42:07.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.836782+0000", 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000127329 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.15279.0:34682 10.9 e286/278 ondisk, result = 0)", 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:19.595593+0000", 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.670518780000002, 2024-03-24T12:42:07.348 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23666422500000001, 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595593+0000", 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595593+0000", 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.349 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595593+0000", 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2099999999999997e-07 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595597+0000", 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8210000000000003e-06 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595623+0000", 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6067999999999998e-05 2024-03-24T12:42:07.350 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.595630+0000", 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9879999999999998e-06 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.832070+0000", 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23643990000000001 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.832084+0000", 2024-03-24T12:42:07.351 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4470999999999999e-05 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.832257+0000", 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000172456 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(client.15279.0:34683 10.5 e286/278 ondisk, result = 0)", 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:19.596015+0000", 2024-03-24T12:42:07.352 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.670095795000002, 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23945687199999999, 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.596015+0000", 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.596015+0000", 2024-03-24T12:42:07.353 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.596016+0000", 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4300000000000003e-07 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.596020+0000", 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9339999999999999e-06 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.354 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.832480+0000", 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23645991099999999 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.832491+0000", 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0762e-05 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.835445+0000", 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002954071 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.835453+0000", 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9019999999999999e-06 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.835472+0000", 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9749e-05 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:34685 10.f e286/278 10:f12a3b4b:::benchmark_data_smithi029_160664_object34684:head v 286'2200, mlcod=286'2197)", 2024-03-24T12:42:07.356 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:19.597017+0000", 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 47.669094465999997, 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.24553852900000001, 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.597017+0000", 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.597017+0000", 2024-03-24T12:42:07.357 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.597017+0000", 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.06e-07 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.597103+0000", 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5370000000000004e-05 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.358 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.831945+0000", 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.234842262 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.831959+0000", 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4219999999999999e-05 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.833376+0000", 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001416956 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.833408+0000", 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2072000000000001e-05 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.841843+0000", 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084346979999999992 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:19.842555+0000", 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00071234499999999997 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.360 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:53412 10.5 10:aefada39:::benchmark_data_smithi029_160664_object53411:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:31.011236+0000", 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 36.254874794000003, 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.21024533000000001, 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:42:07.361 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 53412 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.011236+0000", 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.011236+0000", 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.362 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.011238+0000", 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5319999999999999e-06 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.011262+0000", 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3649999999999999e-05 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.011273+0000", 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1651000000000001e-05 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.011282+0000", 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3550000000000005e-06 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.011318+0000", 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6644999999999999e-05 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.011458+0000", 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000139398 2024-03-24T12:42:07.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.011599+0000", 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014132500000000001 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.019224+0000", 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0076248039999999998 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.019398+0000", 2024-03-24T12:42:07.365 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000174344 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.221369+0000", 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20197091 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.221430+0000", 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1334999999999995e-05 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.221482+0000", 2024-03-24T12:42:07.366 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1381000000000002e-05 2024-03-24T12:42:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:53416 10.5 10:acabd612:::benchmark_data_smithi029_160664_object53415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:42:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:31.012914+0000", 2024-03-24T12:42:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 36.253196908, 2024-03-24T12:42:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.213988494, 2024-03-24T12:42:07.367 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 53416 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.012914+0000", 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.368 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.012914+0000", 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.012915+0000", 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.031e-06 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.012945+0000", 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9513e-05 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.012955+0000", 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0346000000000001e-05 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.012964+0000", 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2600000000000005e-06 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.013940+0000", 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00097694399999999999 2024-03-24T12:42:07.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.014063+0000", 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012245100000000001 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.014198+0000", 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000135234 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.019271+0000", 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0050728489999999999 2024-03-24T12:42:07.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.019404+0000", 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000132853 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.225988+0000", 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20658422800000001 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.226851+0000", 2024-03-24T12:42:07.372 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00086306700000000002 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.226903+0000", 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1718000000000002e-05 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:53418 10.5 10:a63d8534:::benchmark_data_smithi029_160664_object53417:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:31.013290+0000", 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 36.252821034999997, 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.21893769799999999, 2024-03-24T12:42:07.373 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.374 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.374 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.374 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:42:07.374 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:42:07.374 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 53418 2024-03-24T12:42:07.374 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.374 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.374 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.374 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:07.375 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.013290+0000", 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.013290+0000", 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.013291+0000", 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.54e-07 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.013312+0000", 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0696999999999999e-05 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.013322+0000", 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0411000000000001e-05 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.013330+0000", 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8800000000000008e-06 2024-03-24T12:42:07.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.016652+0000", 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0033222019999999998 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.016781+0000", 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012895199999999999 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.016922+0000", 2024-03-24T12:42:07.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014037199999999999 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.019409+0000", 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0024874070000000001 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.019455+0000", 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6186999999999997e-05 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.232082+0000", 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.21262721600000001 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.232168+0000", 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5334000000000007e-05 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.232228+0000", 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0186000000000003e-05 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.380 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.381 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:53435 10.5 10:a8ba0447:::benchmark_data_smithi029_160664_object53434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:42:07.382 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:41:31.021433+0000", 2024-03-24T12:42:07.382 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 36.244678401000002, 2024-03-24T12:42:07.382 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.210947626, 2024-03-24T12:42:07.382 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:42:07.382 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:42:07.382 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:42:07.382 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:42:07.382 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:42:07.382 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 53435 2024-03-24T12:42:07.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.021433+0000", 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.021433+0000", 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.021434+0000", 2024-03-24T12:42:07.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.469e-06 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.021458+0000", 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3714e-05 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.021471+0000", 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3317e-05 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:42:07.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.021480+0000", 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.14e-06 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.022307+0000", 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00082719800000000002 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.022655+0000", 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00034852899999999998 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,3", 2024-03-24T12:42:07.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.022808+0000", 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015300200000000001 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.024641+0000", 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0018323289999999999 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.025313+0000", 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00067225399999999995 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.232284+0000", 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.20697079099999999 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.232341+0000", 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7661999999999998e-05 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:41:31.232381+0000", 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9221000000000003e-05 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.387 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.388 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.388 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:42:07.388 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:42:07.388 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:42:07.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:07.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:07.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:07.434 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:42:07.434 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:42:07.434 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:42:07.434 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:42:07.435 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:42:07.435 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[29919]: pgmap v684: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 29392/282300 objects degraded (10.412%); 26/282300 objects misplaced (0.009%) 2024-03-24T12:42:07.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:42:07.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:42:07.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:42:07.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:42:07.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:42:07.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-mon[41500]: pgmap v684: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 29392/282300 objects degraded (10.412%); 26/282300 objects misplaced (0.009%) 2024-03-24T12:42:07.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:07.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:07.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:07 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:42:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:07 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:42:07.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:07 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:42:07.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:07 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:42:07.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:07 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:42:07.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:07 smithi190 ceph-mon[33748]: pgmap v684: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 68 MiB/s wr, 1.09k op/s; 29392/282300 objects degraded (10.412%); 26/282300 objects misplaced (0.009%) 2024-03-24T12:42:07.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:07.698 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:07.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:07.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:07.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:07.836 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:07.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:07.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:07.897 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:42:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:42:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:42:07.938 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:07.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:07.976 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:08.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:08.114 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:08.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:08.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:08.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-24T12:42:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:08.261 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:08.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:08.292 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 64 16 99357 99341 96.9865 74.625 0.00901445 0.0103061 2024-03-24T12:42:08.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:08.400 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:08.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:08.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:08.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:42:08.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[29919]: 3.14 scrub starts 2024-03-24T12:42:08.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[29919]: 3.14 scrub ok 2024-03-24T12:42:08.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/34870063' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:42:08.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:42:08.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[29919]: 1.9 scrub starts 2024-03-24T12:42:08.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[29919]: 1.9 scrub ok 2024-03-24T12:42:08.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:42:08.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[41500]: 3.14 scrub starts 2024-03-24T12:42:08.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[41500]: 3.14 scrub ok 2024-03-24T12:42:08.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/34870063' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:42:08.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:42:08.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[41500]: 1.9 scrub starts 2024-03-24T12:42:08.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:08 smithi029 ceph-mon[41500]: 1.9 scrub ok 2024-03-24T12:42:08.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:08.542 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:08.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:08.543 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:08.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:42:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:42:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:08 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:42:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:08 smithi190 ceph-mon[33748]: 3.14 scrub starts 2024-03-24T12:42:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:08 smithi190 ceph-mon[33748]: 3.14 scrub ok 2024-03-24T12:42:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:08 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/34870063' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:42:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:08 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:42:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:08 smithi190 ceph-mon[33748]: 1.9 scrub starts 2024-03-24T12:42:08.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:08 smithi190 ceph-mon[33748]: 1.9 scrub ok 2024-03-24T12:42:08.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:08.694 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:08.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:08.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:08.831 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:08.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:08.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:08.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:08.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:08.979 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:08.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:09.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:09.123 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:09.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:09.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:09.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:09.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:09.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:09.292 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 65 16 100649 100633 96.7367 80.75 0.0126219 0.0103326 2024-03-24T12:42:09.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:09.402 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:09.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:09.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:09.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:42:09.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[41500]: osdmap e305: 8 total, 7 up, 8 in 2024-03-24T12:42:09.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:09.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:42:09.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:42:09.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[41500]: 3.d scrub starts 2024-03-24T12:42:09.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[41500]: 3.d scrub ok 2024-03-24T12:42:09.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[41500]: pgmap v686: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 90 MiB/s wr, 1.42k op/s; 30180/292521 objects degraded (10.317%); 27/292521 objects misplaced (0.009%) 2024-03-24T12:42:09.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:42:09.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[29919]: osdmap e305: 8 total, 7 up, 8 in 2024-03-24T12:42:09.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:09.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:42:09.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:42:09.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[29919]: 3.d scrub starts 2024-03-24T12:42:09.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[29919]: 3.d scrub ok 2024-03-24T12:42:09.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:09 smithi029 ceph-mon[29919]: pgmap v686: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 90 MiB/s wr, 1.42k op/s; 30180/292521 objects degraded (10.317%); 27/292521 objects misplaced (0.009%) 2024-03-24T12:42:09.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:09.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:09.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:09.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:09 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:42:09.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:09 smithi190 ceph-mon[33748]: osdmap e305: 8 total, 7 up, 8 in 2024-03-24T12:42:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:09 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:42:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:09 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:42:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:09 smithi190 ceph-mon[33748]: 3.d scrub starts 2024-03-24T12:42:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:09 smithi190 ceph-mon[33748]: 3.d scrub ok 2024-03-24T12:42:09.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:09 smithi190 ceph-mon[33748]: pgmap v686: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 5.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 90 MiB/s wr, 1.42k op/s; 30180/292521 objects degraded (10.317%); 27/292521 objects misplaced (0.009%) 2024-03-24T12:42:09.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:09.693 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:09.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:09.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:09.837 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:09.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:09.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:09.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:09.978 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:09.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:10.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:10.120 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:10.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:10.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:10.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:10.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:10.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:10.292 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 66 16 101901 101885 96.4559 78.25 0.00517208 0.0103626 2024-03-24T12:42:10.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:10.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:10.386 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:10.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:10.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T12:42:10.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[41500]: 6.18 scrub starts 2024-03-24T12:42:10.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[41500]: 6.18 scrub ok 2024-03-24T12:42:10.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[41500]: 3.18 scrub starts 2024-03-24T12:42:10.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[41500]: 3.18 scrub ok 2024-03-24T12:42:10.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 30180/292521 objects degraded (10.317%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:10.449 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[41500]: osdmap e306: 8 total, 7 up, 8 in 2024-03-24T12:42:10.449 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[29919]: 6.18 scrub starts 2024-03-24T12:42:10.449 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[29919]: 6.18 scrub ok 2024-03-24T12:42:10.450 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[29919]: 3.18 scrub starts 2024-03-24T12:42:10.450 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[29919]: 3.18 scrub ok 2024-03-24T12:42:10.450 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 30180/292521 objects degraded (10.317%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:10.450 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:10 smithi029 ceph-mon[29919]: osdmap e306: 8 total, 7 up, 8 in 2024-03-24T12:42:10.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:10.532 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:10.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:10.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:10 smithi190 ceph-mon[33748]: 6.18 scrub starts 2024-03-24T12:42:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:10 smithi190 ceph-mon[33748]: 6.18 scrub ok 2024-03-24T12:42:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:10 smithi190 ceph-mon[33748]: 3.18 scrub starts 2024-03-24T12:42:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:10 smithi190 ceph-mon[33748]: 3.18 scrub ok 2024-03-24T12:42:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:10 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 30180/292521 objects degraded (10.317%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:10.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:10 smithi190 ceph-mon[33748]: osdmap e306: 8 total, 7 up, 8 in 2024-03-24T12:42:10.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:10.680 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:10.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:10.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:10.782 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:10.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:10.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:10.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:10.952 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:10.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:11.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:11.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:11.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:11.191 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:11.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:11.223 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:11.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:11.293 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 67 16 103088 103072 96.1229 74.1875 0.0105421 0.0103987 2024-03-24T12:42:11.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:11.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:11.377 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:11.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:11.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:42:11.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[41500]: 6.15 scrub starts 2024-03-24T12:42:11.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[41500]: 6.15 scrub ok 2024-03-24T12:42:11.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[41500]: pgmap v688: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.05k op/s; 30959/297111 objects degraded (10.420%); 28/297111 objects misplaced (0.009%) 2024-03-24T12:42:11.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:42:11.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:42:11.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:42:11.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[29919]: 6.15 scrub starts 2024-03-24T12:42:11.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[29919]: 6.15 scrub ok 2024-03-24T12:42:11.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[29919]: pgmap v688: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.05k op/s; 30959/297111 objects degraded (10.420%); 28/297111 objects misplaced (0.009%) 2024-03-24T12:42:11.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:42:11.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:11 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:42:11.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:11.518 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:11.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:11.619 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:11.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:11 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:42:11.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:11 smithi190 ceph-mon[33748]: 6.15 scrub starts 2024-03-24T12:42:11.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:11 smithi190 ceph-mon[33748]: 6.15 scrub ok 2024-03-24T12:42:11.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:11 smithi190 ceph-mon[33748]: pgmap v688: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 67 MiB/s wr, 1.05k op/s; 30959/297111 objects degraded (10.420%); 28/297111 objects misplaced (0.009%) 2024-03-24T12:42:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:11 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:42:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:11 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:42:11.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:11.654 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:11.654 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:11.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:11.793 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:11.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:11.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:11.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:11.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:12.032 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:12.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:12.064 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:12.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:12.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:12.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:12.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:12.293 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 68 16 104341 104325 95.861 78.3125 0.00754297 0.0104269 2024-03-24T12:42:12.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:12.332 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:12.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:12.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:12.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 4.4 scrub ok 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: osdmap e307: 8 total, 7 up, 8 in 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 4.8 scrub starts 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 4.8 scrub ok 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 4.1a scrub starts 2024-03-24T12:42:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[41500]: 4.1a scrub ok 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 4.4 scrub ok 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: osdmap e307: 8 total, 7 up, 8 in 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 4.8 scrub starts 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 4.8 scrub ok 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 4.1a scrub starts 2024-03-24T12:42:12.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:12 smithi029 ceph-mon[29919]: 4.1a scrub ok 2024-03-24T12:42:12.434 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:12.471 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:12.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:12.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:12.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:12.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:12.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:12.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:42:12.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:42:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:42:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:42:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 4.4 scrub ok 2024-03-24T12:42:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: osdmap e307: 8 total, 7 up, 8 in 2024-03-24T12:42:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:42:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:42:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 4.8 scrub starts 2024-03-24T12:42:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 4.8 scrub ok 2024-03-24T12:42:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 4.1a scrub starts 2024-03-24T12:42:12.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:12 smithi190 ceph-mon[33748]: 4.1a scrub ok 2024-03-24T12:42:12.710 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:12.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:12.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:12.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:12.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:12.888 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:12.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:12.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:12.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:13.032 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:13.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:13.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:13.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:13.159 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:13.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:13.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:13.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:13.285 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:13.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:13.293 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 69 16 105519 105503 95.5386 73.625 0.00343527 0.0104547 2024-03-24T12:42:13.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:13.425 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:13.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:13.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:13.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:13.554 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:13.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:13.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:13 smithi190 ceph-mon[33748]: 6.a scrub starts 2024-03-24T12:42:13.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:13 smithi190 ceph-mon[33748]: 6.a scrub ok 2024-03-24T12:42:13.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:13 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:42:13.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:13 smithi190 ceph-mon[33748]: 3.1 scrub starts 2024-03-24T12:42:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:13 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:42:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:13 smithi190 ceph-mon[33748]: 3.1 scrub ok 2024-03-24T12:42:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:13 smithi190 ceph-mon[33748]: 3.1d scrub starts 2024-03-24T12:42:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:13 smithi190 ceph-mon[33748]: 3.1d scrub ok 2024-03-24T12:42:13.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:13 smithi190 ceph-mon[33748]: pgmap v690: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 100 MiB/s wr, 1.57k op/s; 32108/311091 objects degraded (10.321%); 28/311091 objects misplaced (0.009%) 2024-03-24T12:42:13.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[29919]: 6.a scrub starts 2024-03-24T12:42:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[29919]: 6.a scrub ok 2024-03-24T12:42:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:42:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[29919]: 3.1 scrub starts 2024-03-24T12:42:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:42:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[29919]: 3.1 scrub ok 2024-03-24T12:42:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[29919]: 3.1d scrub starts 2024-03-24T12:42:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[29919]: 3.1d scrub ok 2024-03-24T12:42:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[29919]: pgmap v690: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 100 MiB/s wr, 1.57k op/s; 32108/311091 objects degraded (10.321%); 28/311091 objects misplaced (0.009%) 2024-03-24T12:42:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[41500]: 6.a scrub starts 2024-03-24T12:42:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[41500]: 6.a scrub ok 2024-03-24T12:42:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:42:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[41500]: 3.1 scrub starts 2024-03-24T12:42:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:42:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[41500]: 3.1 scrub ok 2024-03-24T12:42:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[41500]: 3.1d scrub starts 2024-03-24T12:42:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[41500]: 3.1d scrub ok 2024-03-24T12:42:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:13 smithi029 ceph-mon[41500]: pgmap v690: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 100 MiB/s wr, 1.57k op/s; 32108/311091 objects degraded (10.321%); 28/311091 objects misplaced (0.009%) 2024-03-24T12:42:13.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:13.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:13.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:13.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:13.811 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:13.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:13.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:13.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:13.948 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:13.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:14.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:14.084 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:14.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:14.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:14.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:14.229 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:14.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:14.293 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 70 16 106573 106557 95.1148 65.875 0.00328426 0.0105026 2024-03-24T12:42:14.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:14.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:14.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:14.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:14.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:14.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:14.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:14.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:14.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:14.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:14.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:14.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:14.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:14.752 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:14.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:14.799 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:42:14.799 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:42:14.799 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.8285273185299069 2024-03-24T12:42:14.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd reweight 7 0.8285273185299069 2024-03-24T12:42:14.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:14.881 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:14.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:14.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:15.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:15.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:15.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:15.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:15.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:15.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:15.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:15.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:15.280 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:15.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:15.293 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 71 16 107360 107344 94.468 49.1875 0.00478456 0.0105808 2024-03-24T12:42:15.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:15.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:15.414 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:15.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:15.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:15.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:15.554 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:15.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:15.655 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:15.692 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:15.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:15.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:15.826 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:15.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:15.928 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:15.964 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:15.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:16.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:16.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:16.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:16.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:15 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:42:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:15 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:42:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:15 smithi190 ceph-mon[33748]: 6.1e scrub starts 2024-03-24T12:42:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:15 smithi190 ceph-mon[33748]: 6.1e scrub ok 2024-03-24T12:42:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:15 smithi190 ceph-mon[33748]: pgmap v691: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 32866/314541 objects degraded (10.449%); 28/314541 objects misplaced (0.009%) 2024-03-24T12:42:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:15 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 32866/314541 objects degraded (10.449%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:15 smithi190 ceph-mon[33748]: osdmap e308: 8 total, 7 up, 8 in 2024-03-24T12:42:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:15 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:42:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:15 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:42:16.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:15 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:42:16.142 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:42:16.142 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:42:16.143 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[41500]: 6.1e scrub starts 2024-03-24T12:42:16.143 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[41500]: 6.1e scrub ok 2024-03-24T12:42:16.143 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[41500]: pgmap v691: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 32866/314541 objects degraded (10.449%); 28/314541 objects misplaced (0.009%) 2024-03-24T12:42:16.143 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 32866/314541 objects degraded (10.449%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:16.143 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[41500]: osdmap e308: 8 total, 7 up, 8 in 2024-03-24T12:42:16.143 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:42:16.143 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:42:16.143 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:42:16.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:42:16.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:42:16.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[29919]: 6.1e scrub starts 2024-03-24T12:42:16.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[29919]: 6.1e scrub ok 2024-03-24T12:42:16.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[29919]: pgmap v691: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.4 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 70 MiB/s wr, 1.12k op/s; 32866/314541 objects degraded (10.449%); 28/314541 objects misplaced (0.009%) 2024-03-24T12:42:16.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 32866/314541 objects degraded (10.449%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:16.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[29919]: osdmap e308: 8 total, 7 up, 8 in 2024-03-24T12:42:16.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:42:16.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:42:16.144 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:15 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:42:16.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:16.250 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:16.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:16.293 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 72 15 108423 108408 94.0796 66.5 0.0159873 0.0106249 2024-03-24T12:42:16.353 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:16.385 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:16.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:16.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:16.519 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:16.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:16.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:16.657 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:16.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:16.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:16.759 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:16.783 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:16.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:16.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:16.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:16.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:16.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:17.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:17.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:17.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:17.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:17.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:16 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:42:17.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:16 smithi190 ceph-mon[33748]: 6.1b scrub starts 2024-03-24T12:42:17.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:16 smithi190 ceph-mon[33748]: 6.1b scrub ok 2024-03-24T12:42:17.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:16 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:42:17.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:16 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:42:17.146 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:17.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:16 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:42:17.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:16 smithi029 ceph-mon[41500]: 6.1b scrub starts 2024-03-24T12:42:17.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:16 smithi029 ceph-mon[41500]: 6.1b scrub ok 2024-03-24T12:42:17.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:16 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:42:17.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:16 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:42:17.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:16 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:42:17.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:16 smithi029 ceph-mon[29919]: 6.1b scrub starts 2024-03-24T12:42:17.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:16 smithi029 ceph-mon[29919]: 6.1b scrub ok 2024-03-24T12:42:17.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:16 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:42:17.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:16 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:42:17.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:17.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:17.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:17.278 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:17.293 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 73 16 109361 109345 93.5931 58.5625 0.0125326 0.0106801 2024-03-24T12:42:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:17.313 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:17.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:17.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:17.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:17.455 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:17.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:17.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:17.590 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:17.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:17.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:17.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:17.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:17.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:17.813 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:42:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:42:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:42:17.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:17.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:17.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:17.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:17.951 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:17.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:17.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:18.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:18.123 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:18.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:18.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:17 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:42:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:17 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:42:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:17 smithi190 ceph-mon[33748]: pgmap v693: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 775 B/s rd, 68 MiB/s wr, 1.08k op/s; 33176/318600 objects degraded (10.413%); 28/318600 objects misplaced (0.009%) 2024-03-24T12:42:18.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:17 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:42:18.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:17 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:42:18.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:17 smithi029 ceph-mon[41500]: pgmap v693: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 775 B/s rd, 68 MiB/s wr, 1.08k op/s; 33176/318600 objects degraded (10.413%); 28/318600 objects misplaced (0.009%) 2024-03-24T12:42:18.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:17 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:42:18.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:17 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:42:18.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:17 smithi029 ceph-mon[29919]: pgmap v693: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 775 B/s rd, 68 MiB/s wr, 1.08k op/s; 33176/318600 objects degraded (10.413%); 28/318600 objects misplaced (0.009%) 2024-03-24T12:42:18.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:18.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:18.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:18.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:18.293 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 74 16 110570 110554 93.3494 75.5625 0.0130013 0.0107078 2024-03-24T12:42:18.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:18.394 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:18.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:18.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:18.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:18.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:18.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:42:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:42:18.636 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:18.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:18.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:18.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:18.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:18.805 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:18.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:18.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:18.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:18.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:19.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:19.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:19.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:18 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:42:19.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:18 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:42:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:18 smithi190 ceph-mon[33748]: 5.15 scrub starts 2024-03-24T12:42:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:18 smithi190 ceph-mon[33748]: 5.15 scrub ok 2024-03-24T12:42:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:18 smithi190 ceph-mon[33748]: 4.17 scrub starts 2024-03-24T12:42:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:18 smithi190 ceph-mon[33748]: 4.17 scrub ok 2024-03-24T12:42:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:42:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:42:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[41500]: 5.15 scrub starts 2024-03-24T12:42:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[41500]: 5.15 scrub ok 2024-03-24T12:42:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[41500]: 4.17 scrub starts 2024-03-24T12:42:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[41500]: 4.17 scrub ok 2024-03-24T12:42:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:19.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:42:19.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:42:19.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[29919]: 5.15 scrub starts 2024-03-24T12:42:19.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[29919]: 5.15 scrub ok 2024-03-24T12:42:19.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[29919]: 4.17 scrub starts 2024-03-24T12:42:19.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[29919]: 4.17 scrub ok 2024-03-24T12:42:19.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:19.191 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:19.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:19.293 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 75 16 111543 111527 92.9156 60.8125 0.00832776 0.0107572 2024-03-24T12:42:19.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:19.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:19.356 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:19.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:19.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:19.499 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:19.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:19.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:19.646 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:19.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:19.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:19.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:19.790 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:19.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:19.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:19.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:19.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:20.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:20.071 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:20.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:20.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:20.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:19 smithi190 ceph-mon[33748]: 6.6 scrub starts 2024-03-24T12:42:20.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:19 smithi190 ceph-mon[33748]: 6.6 scrub ok 2024-03-24T12:42:20.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:19 smithi190 ceph-mon[33748]: pgmap v694: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 33791/326910 objects degraded (10.336%); 28/326910 objects misplaced (0.009%) 2024-03-24T12:42:20.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:19 smithi190 ceph-mon[33748]: 4.12 scrub starts 2024-03-24T12:42:20.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:19 smithi190 ceph-mon[33748]: 4.12 scrub ok 2024-03-24T12:42:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[41500]: 6.6 scrub starts 2024-03-24T12:42:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[41500]: 6.6 scrub ok 2024-03-24T12:42:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[41500]: pgmap v694: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 33791/326910 objects degraded (10.336%); 28/326910 objects misplaced (0.009%) 2024-03-24T12:42:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[41500]: 4.12 scrub starts 2024-03-24T12:42:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[41500]: 4.12 scrub ok 2024-03-24T12:42:20.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:20.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[29919]: 6.6 scrub starts 2024-03-24T12:42:20.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[29919]: 6.6 scrub ok 2024-03-24T12:42:20.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[29919]: pgmap v694: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.6 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.24k op/s; 33791/326910 objects degraded (10.336%); 28/326910 objects misplaced (0.009%) 2024-03-24T12:42:20.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[29919]: 4.12 scrub starts 2024-03-24T12:42:20.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:19 smithi029 ceph-mon[29919]: 4.12 scrub ok 2024-03-24T12:42:20.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:20.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:20.199 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:20.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:20.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 76 16 112716 112700 92.6576 73.3125 0.00723554 0.0107875 2024-03-24T12:42:20.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:20.349 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:20.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:20.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:20.494 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:20.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:20.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:20.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:20.631 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:20.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:20.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:20.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:20.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:20.767 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:20.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:20.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:20.894 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:20.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:20.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:21.036 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:21.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:20 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:42:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:20 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:42:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:20 smithi190 ceph-mon[33748]: 4.b scrub starts 2024-03-24T12:42:21.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:20 smithi190 ceph-mon[33748]: 4.b scrub ok 2024-03-24T12:42:21.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:20 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 33791/326910 objects degraded (10.336%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:21.137 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:21.157 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:20 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:42:21.157 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:20 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:42:21.157 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:20 smithi029 ceph-mon[41500]: 4.b scrub starts 2024-03-24T12:42:21.157 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:20 smithi029 ceph-mon[41500]: 4.b scrub ok 2024-03-24T12:42:21.157 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:20 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 33791/326910 objects degraded (10.336%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:21.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:20 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:42:21.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:20 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:42:21.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:20 smithi029 ceph-mon[29919]: 4.b scrub starts 2024-03-24T12:42:21.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:20 smithi029 ceph-mon[29919]: 4.b scrub ok 2024-03-24T12:42:21.158 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:20 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 33791/326910 objects degraded (10.336%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:21.168 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:21.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:21.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:21.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:21.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 77 16 113788 113772 92.3244 67 0.00359799 0.0108208 2024-03-24T12:42:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:21.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:21.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:21.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:21.421 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:21.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:21.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:21.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:21.555 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:21.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:21.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:21.685 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:21.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:21.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:21.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:21.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:21.826 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:21.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:21.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:21.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:21.952 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:21.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:22.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:22.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:22.093 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:22.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:21 smithi190 ceph-mon[33748]: 3.11 scrub starts 2024-03-24T12:42:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:21 smithi190 ceph-mon[33748]: 3.11 scrub ok 2024-03-24T12:42:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:21 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:42:22.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:21 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:42:22.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:21 smithi190 ceph-mon[33748]: pgmap v695: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 34465/330864 objects degraded (10.417%); 28/330864 objects misplaced (0.008%) 2024-03-24T12:42:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:21 smithi029 ceph-mon[41500]: 3.11 scrub starts 2024-03-24T12:42:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[41500]: 3.11 scrub ok 2024-03-24T12:42:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:42:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:42:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[41500]: pgmap v695: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 34465/330864 objects degraded (10.417%); 28/330864 objects misplaced (0.008%) 2024-03-24T12:42:22.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:21 smithi029 ceph-mon[29919]: 3.11 scrub starts 2024-03-24T12:42:22.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[29919]: 3.11 scrub ok 2024-03-24T12:42:22.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:42:22.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:42:22.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[29919]: pgmap v695: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 34465/330864 objects degraded (10.417%); 28/330864 objects misplaced (0.008%) 2024-03-24T12:42:22.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:22.235 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:22.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:22.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:22.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 78 15 114515 114500 91.7241 45.5 0.009142 0.0108965 2024-03-24T12:42:22.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:22.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:22.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:22.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:22.503 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:22.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:22.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:22.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:22.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:22.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:22.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:22.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:22.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:22.898 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:22.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:22.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:23.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:23.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:23.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:23.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:23.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:23.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:22 smithi190 ceph-mon[33748]: 6.13 scrub starts 2024-03-24T12:42:23.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:22 smithi190 ceph-mon[33748]: 10.2 scrub starts 2024-03-24T12:42:23.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:22 smithi190 ceph-mon[33748]: 6.13 scrub ok 2024-03-24T12:42:23.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:22 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:42:23.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:22 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:42:23.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:23.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[41500]: 6.13 scrub starts 2024-03-24T12:42:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[41500]: 10.2 scrub starts 2024-03-24T12:42:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[41500]: 6.13 scrub ok 2024-03-24T12:42:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:42:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:42:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[29919]: 6.13 scrub starts 2024-03-24T12:42:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[29919]: 10.2 scrub starts 2024-03-24T12:42:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[29919]: 6.13 scrub ok 2024-03-24T12:42:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:42:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:22 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:42:23.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:23.284 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:23.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:23.293 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 79 16 115163 115147 91.075 40.4375 0.0552488 0.0109737 2024-03-24T12:42:23.386 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:23.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:23.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:23.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:23.545 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:23.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:23.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:23.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:23.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:23.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:23.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:23.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:23.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:23.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:23.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:23.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:24.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:24.076 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:24.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:24.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:24.131 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:23 smithi029 ceph-mon[41500]: pgmap v696: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 35362/341757 objects degraded (10.347%); 28/341757 objects misplaced (0.008%) 2024-03-24T12:42:24.132 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:23 smithi029 ceph-mon[41500]: 5.1c scrub starts 2024-03-24T12:42:24.132 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:23 smithi029 ceph-mon[41500]: 5.1c scrub ok 2024-03-24T12:42:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:23 smithi029 ceph-mon[29919]: pgmap v696: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 35362/341757 objects degraded (10.347%); 28/341757 objects misplaced (0.008%) 2024-03-24T12:42:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:23 smithi029 ceph-mon[29919]: 5.1c scrub starts 2024-03-24T12:42:24.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:23 smithi029 ceph-mon[29919]: 5.1c scrub ok 2024-03-24T12:42:24.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:23 smithi190 ceph-mon[33748]: pgmap v696: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 6.9 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s; 35362/341757 objects degraded (10.347%); 28/341757 objects misplaced (0.008%) 2024-03-24T12:42:24.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:23 smithi190 ceph-mon[33748]: 5.1c scrub starts 2024-03-24T12:42:24.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:23 smithi190 ceph-mon[33748]: 5.1c scrub ok 2024-03-24T12:42:24.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:24.223 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:24.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:24.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:24.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:42:24.299343+0000 min lat: 0.00126577 max lat: 0.422159 avg lat: 0.0110184 2024-03-24T12:42:24.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:42:24.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 80 15 116161 116146 90.717 62.4375 0.0104317 0.0110184 2024-03-24T12:42:24.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:24.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:24.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:24.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:24.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:24.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:24.475 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:24.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:24.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:24.614 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:24.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:24.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:24.748 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:24.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:24.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:24.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:24.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:24.887 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:24.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[29919]: 4.14 scrub starts 2024-03-24T12:42:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[29919]: 4.14 scrub ok 2024-03-24T12:42:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[29919]: 6.9 scrub starts 2024-03-24T12:42:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[29919]: 4.1 scrub starts 2024-03-24T12:42:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[29919]: 4.1 scrub ok 2024-03-24T12:42:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[29919]: 6.9 scrub ok 2024-03-24T12:42:24.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[41500]: 4.14 scrub starts 2024-03-24T12:42:24.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[41500]: 4.14 scrub ok 2024-03-24T12:42:24.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[41500]: 6.9 scrub starts 2024-03-24T12:42:24.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[41500]: 4.1 scrub starts 2024-03-24T12:42:24.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[41500]: 4.1 scrub ok 2024-03-24T12:42:24.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:24 smithi029 ceph-mon[41500]: 6.9 scrub ok 2024-03-24T12:42:24.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:25.027 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:25.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:25.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:25.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:25.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:24 smithi190 ceph-mon[33748]: 4.14 scrub starts 2024-03-24T12:42:25.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:24 smithi190 ceph-mon[33748]: 4.14 scrub ok 2024-03-24T12:42:25.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:24 smithi190 ceph-mon[33748]: 6.9 scrub starts 2024-03-24T12:42:25.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:24 smithi190 ceph-mon[33748]: 4.1 scrub starts 2024-03-24T12:42:25.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:24 smithi190 ceph-mon[33748]: 4.1 scrub ok 2024-03-24T12:42:25.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:24 smithi190 ceph-mon[33748]: 6.9 scrub ok 2024-03-24T12:42:25.169 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:25.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:25.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:25.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:25.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 81 16 117038 117022 90.2729 54.75 0.00638808 0.0110725 2024-03-24T12:42:25.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:25.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:25.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:25.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:25.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:25.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:25.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:25.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:25.570 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:25.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:25.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:25.672 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:25.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:25.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:25.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:25.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:25.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:25.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:25.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:25.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:25.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:25.976 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:25.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:26.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:26.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:26.106 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:26.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:26.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:25 smithi190 ceph-mon[33748]: 10.2 scrub ok 2024-03-24T12:42:26.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:25 smithi190 ceph-mon[33748]: 5.1a scrub starts 2024-03-24T12:42:26.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:25 smithi190 ceph-mon[33748]: 5.1a scrub ok 2024-03-24T12:42:26.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:25 smithi190 ceph-mon[33748]: pgmap v697: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 999 op/s; 35967/344496 objects degraded (10.440%); 28/344496 objects misplaced (0.008%) 2024-03-24T12:42:26.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:25 smithi190 ceph-mon[33748]: 4.d scrub starts 2024-03-24T12:42:26.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:25 smithi190 ceph-mon[33748]: 4.d scrub ok 2024-03-24T12:42:26.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:25 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 35967/344496 objects degraded (10.440%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[29919]: 10.2 scrub ok 2024-03-24T12:42:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[29919]: 5.1a scrub starts 2024-03-24T12:42:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[29919]: 5.1a scrub ok 2024-03-24T12:42:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[29919]: pgmap v697: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 999 op/s; 35967/344496 objects degraded (10.440%); 28/344496 objects misplaced (0.008%) 2024-03-24T12:42:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[29919]: 4.d scrub starts 2024-03-24T12:42:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[29919]: 4.d scrub ok 2024-03-24T12:42:26.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 35967/344496 objects degraded (10.440%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:26.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[41500]: 10.2 scrub ok 2024-03-24T12:42:26.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[41500]: 5.1a scrub starts 2024-03-24T12:42:26.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[41500]: 5.1a scrub ok 2024-03-24T12:42:26.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[41500]: pgmap v697: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 999 op/s; 35967/344496 objects degraded (10.440%); 28/344496 objects misplaced (0.008%) 2024-03-24T12:42:26.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[41500]: 4.d scrub starts 2024-03-24T12:42:26.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[41500]: 4.d scrub ok 2024-03-24T12:42:26.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:25 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 35967/344496 objects degraded (10.440%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:26.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:26.250 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:26.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:26.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:26.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 82 16 117994 117978 89.9007 59.75 0.0103294 0.0111188 2024-03-24T12:42:26.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:26.381 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:26.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:26.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:26.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:26.508 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:26.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:26.609 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:26.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:26.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:26.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:26.783 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:26.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:26.790 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:42:26.835 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:42:26.866 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:42:26.867 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:42:26.867 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:42:26.867 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:42:26.867 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[41500]: 5.18 scrub starts 2024-03-24T12:42:26.867 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[41500]: 5.18 scrub ok 2024-03-24T12:42:26.868 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[41500]: 3.10 scrub starts 2024-03-24T12:42:26.868 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[41500]: 3.10 scrub ok 2024-03-24T12:42:26.868 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:42:26.868 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:42:26.869 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:42:26.869 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:42:26.869 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[29919]: 5.18 scrub starts 2024-03-24T12:42:26.869 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[29919]: 5.18 scrub ok 2024-03-24T12:42:26.869 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[29919]: 3.10 scrub starts 2024-03-24T12:42:26.869 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[29919]: 3.10 scrub ok 2024-03-24T12:42:26.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[29919]: osdmap e309: 8 total, 7 up, 8 in 2024-03-24T12:42:26.870 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3517181298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:42:26.882 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:42:26.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:26.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:26.924 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:42:26.948 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:26.964 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:42:27.007 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:42:27.025 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:27.045 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:42:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:42:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:42:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:42:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:42:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:42:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:42:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:42:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:42:27.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:27.076 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:27.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:27.119 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[41500]: osdmap e309: 8 total, 7 up, 8 in 2024-03-24T12:42:27.119 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:26 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3517181298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:42:27.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:26 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:42:27.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:26 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:42:27.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:26 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:42:27.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:26 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:42:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:26 smithi190 ceph-mon[33748]: 5.18 scrub starts 2024-03-24T12:42:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:26 smithi190 ceph-mon[33748]: 5.18 scrub ok 2024-03-24T12:42:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:26 smithi190 ceph-mon[33748]: 3.10 scrub starts 2024-03-24T12:42:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:26 smithi190 ceph-mon[33748]: 3.10 scrub ok 2024-03-24T12:42:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:26 smithi190 ceph-mon[33748]: osdmap e309: 8 total, 7 up, 8 in 2024-03-24T12:42:27.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:26 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3517181298' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:42:27.177 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:27.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:27.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:27.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:27.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 83 16 118946 118930 89.5344 59.5 0.0124892 0.0111645 2024-03-24T12:42:27.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:27.355 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:27.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:27.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:27.457 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:27.502 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:27.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:27.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:27.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:27.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:27.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:27.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:27.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:27.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:27.882 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:42:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:42:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:42:27.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:27.925 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:27.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:27.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:28.027 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:28.058 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:28.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:28.059 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:28.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:27 smithi190 ceph-mon[33748]: pgmap v699: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 971 op/s; 36254/347724 objects degraded (10.426%); 28/347724 objects misplaced (0.008%) 2024-03-24T12:42:28.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:27 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:42:28.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:27 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:42:28.160 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:28.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:27 smithi029 ceph-mon[41500]: pgmap v699: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 971 op/s; 36254/347724 objects degraded (10.426%); 28/347724 objects misplaced (0.008%) 2024-03-24T12:42:28.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:27 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:42:28.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:27 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:42:28.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:27 smithi029 ceph-mon[29919]: pgmap v699: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 61 MiB/s wr, 971 op/s; 36254/347724 objects degraded (10.426%); 28/347724 objects misplaced (0.008%) 2024-03-24T12:42:28.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:27 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:42:28.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:27 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:42:28.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:28.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:28.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:28.277 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:42:28.278 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:42:28.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:28.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 84 16 120225 120209 89.4201 79.9375 0.00829807 0.0111785 2024-03-24T12:42:28.332 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:28.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:28.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:28.434 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:28.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:28.478 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:28.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:28.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:28.614 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:28.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:28.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:28.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:42:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:42:28.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:28.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:28.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:28.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:28.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:28.891 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:28.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:28.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:28.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:29.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:29.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:29.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:29.167 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:29.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:28 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:42:29.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:28 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:42:29.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:28 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:42:29.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:28 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:42:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:29.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:29.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:29.295 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 85 16 121459 121443 89.2754 77.125 0.0144345 0.0111969 2024-03-24T12:42:29.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:29.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:29.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:29.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:28 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:42:29.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:28 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:42:29.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:29.449 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:29.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:29.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:29.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:29.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:29.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:29.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:29.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:29.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:29.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:29.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:29.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:29.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:29.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:29.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:29.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:30.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:30.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:30.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:30.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:29 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:30.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:29 smithi029 ceph-mon[41500]: pgmap v700: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 958 op/s; 36852/355647 objects degraded (10.362%); 28/355647 objects misplaced (0.008%) 2024-03-24T12:42:30.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:42:30.172 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:42:30.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:29 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:30.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:29 smithi029 ceph-mon[29919]: pgmap v700: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 958 op/s; 36852/355647 objects degraded (10.362%); 28/355647 objects misplaced (0.008%) 2024-03-24T12:42:30.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:42:30.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:42:30.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:30.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:30.249 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:30.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:30.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:42:30.294 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 86 16 122440 122424 88.9502 61.3125 0.0183193 0.0112377 2024-03-24T12:42:30.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:30.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:30.381 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:30.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:29 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:29 smithi190 ceph-mon[33748]: pgmap v700: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 958 op/s; 36852/355647 objects degraded (10.362%); 28/355647 objects misplaced (0.008%) 2024-03-24T12:42:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:29 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:42:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:29 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:42:30.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:30.527 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:30.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:30.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:30.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:30.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:30.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:30.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:30.778 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:30.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:30.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:30.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:30.938 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:30.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:31.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:31.084 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:31.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:42:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:42:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 36852/355647 objects degraded (10.362%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[29919]: osdmap e310: 8 total, 7 up, 8 in 2024-03-24T12:42:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2659233957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:42:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:42:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:42:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 36852/355647 objects degraded (10.362%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[41500]: osdmap e310: 8 total, 7 up, 8 in 2024-03-24T12:42:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:30 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2659233957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:42:31.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:31.186 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:42:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:31.240 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:31.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:31.295 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 87 16 123393 123377 88.6123 59.5625 0.0141819 0.0112801 2024-03-24T12:42:31.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:31.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:31.381 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:31.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:31.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:30 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:42:31.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:30 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:42:31.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:30 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 36852/355647 objects degraded (10.362%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:31.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:30 smithi190 ceph-mon[33748]: osdmap e310: 8 total, 7 up, 8 in 2024-03-24T12:42:31.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:30 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2659233957' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:42:31.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:31.507 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:31.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:31.609 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:31.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:31.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:31.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:31.784 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:31.784 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:31.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:31.910 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:42:31.910 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:42:31.910 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[29919]: pgmap v702: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 748 op/s; 37515/359706 objects degraded (10.429%); 28/359706 objects misplaced (0.008%) 2024-03-24T12:42:31.910 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[29919]: 4.1d scrub starts 2024-03-24T12:42:31.910 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[29919]: 4.1d scrub ok 2024-03-24T12:42:31.928 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:31.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:32.030 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:32.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:32.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:32.166 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:32.172 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:42:32.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2659233957' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:42:32.173 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[29919]: osdmap e311: 8 total, 7 up, 8 in 2024-03-24T12:42:32.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:42:32.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:42:32.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[41500]: pgmap v702: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 748 op/s; 37515/359706 objects degraded (10.429%); 28/359706 objects misplaced (0.008%) 2024-03-24T12:42:32.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[41500]: 4.1d scrub starts 2024-03-24T12:42:32.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[41500]: 4.1d scrub ok 2024-03-24T12:42:32.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:42:32.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2659233957' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:42:32.174 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:31 smithi029 ceph-mon[41500]: osdmap e311: 8 total, 7 up, 8 in 2024-03-24T12:42:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:32.215 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:32.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:32.295 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 88 16 124485 124469 88.3809 68.25 0.00683375 0.0113102 2024-03-24T12:42:32.317 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:32.360 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:32.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:32.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:31 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:42:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:31 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:42:32.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:31 smithi190 ceph-mon[33748]: pgmap v702: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 748 op/s; 37515/359706 objects degraded (10.429%); 28/359706 objects misplaced (0.008%) 2024-03-24T12:42:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:31 smithi190 ceph-mon[33748]: 4.1d scrub starts 2024-03-24T12:42:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:31 smithi190 ceph-mon[33748]: 4.1d scrub ok 2024-03-24T12:42:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:31 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:42:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:31 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2659233957' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:42:32.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:31 smithi190 ceph-mon[33748]: osdmap e311: 8 total, 7 up, 8 in 2024-03-24T12:42:32.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:32.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:32.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:32.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:32.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:32.644 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:32.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:32.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:32.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:32.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:32.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:32.950 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:32.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:33.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:33.100 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:33.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[41500]: 6.12 scrub starts 2024-03-24T12:42:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[41500]: 6.12 scrub ok 2024-03-24T12:42:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[41500]: 5.2 scrub starts 2024-03-24T12:42:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[41500]: 5.2 scrub ok 2024-03-24T12:42:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:42:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:42:33.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[29919]: 6.12 scrub starts 2024-03-24T12:42:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[29919]: 6.12 scrub ok 2024-03-24T12:42:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[29919]: 5.2 scrub starts 2024-03-24T12:42:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[29919]: 5.2 scrub ok 2024-03-24T12:42:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:42:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:32 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:42:33.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:33.232 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:33.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:33.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:33.295 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 89 16 125730 125714 88.2621 77.8125 0.0215653 0.0113254 2024-03-24T12:42:33.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:33.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:33.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:33.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:32 smithi190 ceph-mon[33748]: 6.12 scrub starts 2024-03-24T12:42:33.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:32 smithi190 ceph-mon[33748]: 6.12 scrub ok 2024-03-24T12:42:33.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:32 smithi190 ceph-mon[33748]: 5.2 scrub starts 2024-03-24T12:42:33.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:32 smithi190 ceph-mon[33748]: 5.2 scrub ok 2024-03-24T12:42:33.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:32 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:42:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:32 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:42:33.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:33.515 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:33.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:33.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:33.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:33.659 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:33.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:33.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:33.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:33.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:33.799 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:33.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:33.947 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:33.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:33.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:34.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:34.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:34.092 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:34.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:34 smithi029 ceph-mon[41500]: pgmap v704: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 38548/372198 objects degraded (10.357%); 28/372198 objects misplaced (0.008%) 2024-03-24T12:42:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:34 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:42:34.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:34 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:42:34.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:42:34.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:34.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:34 smithi029 ceph-mon[29919]: pgmap v704: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 38548/372198 objects degraded (10.357%); 28/372198 objects misplaced (0.008%) 2024-03-24T12:42:34.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:34 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:42:34.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:34 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:42:34.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:42:34.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:34.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:34.230 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:34.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:34.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:34.295 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Total time run: 90.0056 2024-03-24T12:42:34.295 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Total writes made: 126971 2024-03-24T12:42:34.295 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Write size: 65536 2024-03-24T12:42:34.295 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Object size: 65536 2024-03-24T12:42:34.295 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Bandwidth (MB/sec): 88.1688 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev Bandwidth: 34.5152 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max bandwidth (MB/sec): 228.562 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min bandwidth (MB/sec): 40.4375 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Average IOPS: 1410 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev IOPS: 552.244 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max IOPS: 3657 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min IOPS: 647 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Average Latency(s): 0.011338 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev Latency(s): 0.0154843 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max latency(s): 0.422159 2024-03-24T12:42:34.296 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min latency(s): 0.00126577 2024-03-24T12:42:34.300 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T12:42:34.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:34.364 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:34.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:34 smithi190 ceph-mon[33748]: pgmap v704: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.4 KiB/s rd, 81 MiB/s wr, 1.30k op/s; 38548/372198 objects degraded (10.357%); 28/372198 objects misplaced (0.008%) 2024-03-24T12:42:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:34 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:42:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:34 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:42:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:42:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:34.465 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:34.509 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:34.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:34.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:34.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:34.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:34.749 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:34.779 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:34.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:34.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:34.881 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:34.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:34.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:34.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:35.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:35.050 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:42:35.050 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:42:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:35.062 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:35.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:35.079 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[29919]: 5.17 scrub starts 2024-03-24T12:42:35.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:35.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:35.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:35.309 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:35.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:35.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:35 smithi190 ceph-mon[33748]: 5.17 scrub starts 2024-03-24T12:42:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:35 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:42:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:35 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:42:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:35 smithi190 ceph-mon[33748]: 5.17 scrub ok 2024-03-24T12:42:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:35 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:42:35.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:35 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:42:35.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[41500]: 5.17 scrub starts 2024-03-24T12:42:35.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:42:35.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:42:35.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[41500]: 5.17 scrub ok 2024-03-24T12:42:35.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:42:35.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:42:35.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:42:35.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:42:35.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[29919]: 5.17 scrub ok 2024-03-24T12:42:35.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:42:35.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:35 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:42:35.460 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:35.501 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:35.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:35.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:35.631 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:35.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:35.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:35.733 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:35.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:35.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:35.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:35.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:35.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:35.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:35.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:42:36.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:36.052 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:36.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T12:42:36.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:36.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.7 to 0.828527 (d41a) 2024-03-24T12:42:36.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:36.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:36.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:36.343 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[41500]: pgmap v705: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 39254/375366 objects degraded (10.458%); 28/375366 objects misplaced (0.007%) 2024-03-24T12:42:36.343 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[41500]: 5.1e scrub starts 2024-03-24T12:42:36.343 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[41500]: 5.1e scrub ok 2024-03-24T12:42:36.344 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 39254/375366 objects degraded (10.458%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:36.344 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[41500]: osdmap e312: 8 total, 7 up, 8 in 2024-03-24T12:42:36.344 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2408338341' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.8285273185299069}]: dispatch 2024-03-24T12:42:36.344 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.8285273185299069}]: dispatch 2024-03-24T12:42:36.344 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[41500]: 5.a scrub starts 2024-03-24T12:42:36.344 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[41500]: 5.a scrub ok 2024-03-24T12:42:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[29919]: pgmap v705: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 39254/375366 objects degraded (10.458%); 28/375366 objects misplaced (0.007%) 2024-03-24T12:42:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[29919]: 5.1e scrub starts 2024-03-24T12:42:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[29919]: 5.1e scrub ok 2024-03-24T12:42:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 39254/375366 objects degraded (10.458%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:36.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[29919]: osdmap e312: 8 total, 7 up, 8 in 2024-03-24T12:42:36.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2408338341' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.8285273185299069}]: dispatch 2024-03-24T12:42:36.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.8285273185299069}]: dispatch 2024-03-24T12:42:36.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[29919]: 5.a scrub starts 2024-03-24T12:42:36.350 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:36 smithi029 ceph-mon[29919]: 5.a scrub ok 2024-03-24T12:42:36.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:36 smithi190 ceph-mon[33748]: pgmap v705: 158 pgs: 21 active+clean+remapped, 102 active+clean, 22 active+undersized, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.15k op/s; 39254/375366 objects degraded (10.458%); 28/375366 objects misplaced (0.007%) 2024-03-24T12:42:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:36 smithi190 ceph-mon[33748]: 5.1e scrub starts 2024-03-24T12:42:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:36 smithi190 ceph-mon[33748]: 5.1e scrub ok 2024-03-24T12:42:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:36 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 39254/375366 objects degraded (10.458%), 13 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:36 smithi190 ceph-mon[33748]: osdmap e312: 8 total, 7 up, 8 in 2024-03-24T12:42:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:36 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2408338341' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.8285273185299069}]: dispatch 2024-03-24T12:42:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:36 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.8285273185299069}]: dispatch 2024-03-24T12:42:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:36 smithi190 ceph-mon[33748]: 5.a scrub starts 2024-03-24T12:42:36.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:36 smithi190 ceph-mon[33748]: 5.a scrub ok 2024-03-24T12:42:36.395 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:36.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:36.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:36.526 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:36.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:36.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:36.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:36.680 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:36.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:36.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:36.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:36.830 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:36.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:36.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:36.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:36.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:37.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:37.105 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:37.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:37.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:37.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:37.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:37.251 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:37.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:37.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:37.380 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:37.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:37.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:37.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:37.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:37.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:37.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:37 smithi029 ceph-mon[29919]: 3.1a scrub starts 2024-03-24T12:42:37.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:37 smithi029 ceph-mon[29919]: 3.1a scrub ok 2024-03-24T12:42:37.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:37 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.8285273185299069}]': finished 2024-03-24T12:42:37.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:37 smithi029 ceph-mon[29919]: osdmap e313: 8 total, 7 up, 8 in 2024-03-24T12:42:37.606 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:37 smithi029 ceph-mon[41500]: 3.1a scrub starts 2024-03-24T12:42:37.606 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:37 smithi029 ceph-mon[41500]: 3.1a scrub ok 2024-03-24T12:42:37.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:37 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.8285273185299069}]': finished 2024-03-24T12:42:37.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:37 smithi029 ceph-mon[41500]: osdmap e313: 8 total, 7 up, 8 in 2024-03-24T12:42:37.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:37 smithi190 ceph-mon[33748]: 3.1a scrub starts 2024-03-24T12:42:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:37 smithi190 ceph-mon[33748]: 3.1a scrub ok 2024-03-24T12:42:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:37 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.8285273185299069}]': finished 2024-03-24T12:42:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:37 smithi190 ceph-mon[33748]: osdmap e313: 8 total, 7 up, 8 in 2024-03-24T12:42:37.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:37.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:37.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:37.762 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:37.793 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:37.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:37.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:37.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:37.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:42:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:42:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:42:37.926 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:37.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:38.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:38.061 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:38.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:38.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:38.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:38.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:38.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:38.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:38.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:38.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:38.321 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:38.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:38.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:38.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:38.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:38.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:38.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:38.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:38.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:42:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:42:38.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:38 smithi190 ceph-mon[33748]: pgmap v708: 158 pgs: 3 peering, 1 unknown, 21 active+clean+remapped, 99 active+clean, 21 active+undersized, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 5.3 KiB/s rd, 64 MiB/s wr, 1.12k op/s; 39242/376293 objects degraded (10.429%); 28/376293 objects misplaced (0.007%); 26 KiB/s, 0 objects/s recovering 2024-03-24T12:42:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:38 smithi190 ceph-mon[33748]: 10.f deep-scrub starts 2024-03-24T12:42:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:38 smithi190 ceph-mon[33748]: osdmap e314: 8 total, 7 up, 8 in 2024-03-24T12:42:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:38 smithi190 ceph-mon[33748]: 4.0 scrub starts 2024-03-24T12:42:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:38 smithi190 ceph-mon[33748]: 4.0 scrub ok 2024-03-24T12:42:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:38 smithi190 ceph-mon[33748]: osdmap e315: 8 total, 7 up, 8 in 2024-03-24T12:42:38.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[29919]: pgmap v708: 158 pgs: 3 peering, 1 unknown, 21 active+clean+remapped, 99 active+clean, 21 active+undersized, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 5.3 KiB/s rd, 64 MiB/s wr, 1.12k op/s; 39242/376293 objects degraded (10.429%); 28/376293 objects misplaced (0.007%); 26 KiB/s, 0 objects/s recovering 2024-03-24T12:42:38.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[29919]: 10.f deep-scrub starts 2024-03-24T12:42:38.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[29919]: osdmap e314: 8 total, 7 up, 8 in 2024-03-24T12:42:38.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[29919]: 4.0 scrub starts 2024-03-24T12:42:38.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[29919]: 4.0 scrub ok 2024-03-24T12:42:38.636 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[29919]: osdmap e315: 8 total, 7 up, 8 in 2024-03-24T12:42:38.637 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[41500]: pgmap v708: 158 pgs: 3 peering, 1 unknown, 21 active+clean+remapped, 99 active+clean, 21 active+undersized, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 5.3 KiB/s rd, 64 MiB/s wr, 1.12k op/s; 39242/376293 objects degraded (10.429%); 28/376293 objects misplaced (0.007%); 26 KiB/s, 0 objects/s recovering 2024-03-24T12:42:38.637 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[41500]: 10.f deep-scrub starts 2024-03-24T12:42:38.637 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[41500]: osdmap e314: 8 total, 7 up, 8 in 2024-03-24T12:42:38.637 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[41500]: 4.0 scrub starts 2024-03-24T12:42:38.637 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[41500]: 4.0 scrub ok 2024-03-24T12:42:38.637 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:38 smithi029 ceph-mon[41500]: osdmap e315: 8 total, 7 up, 8 in 2024-03-24T12:42:38.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:38.742 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:38.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:38.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:38.844 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:38.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:38.884 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:38.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:38.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:39.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:39.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:39.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:39.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:39.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:39.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:39.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:39.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:39.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:39.292 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:39.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:39.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:39.416 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:39.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:39.518 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:39.546 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:39.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:39 smithi190 ceph-mon[33748]: 10.0 scrub starts 2024-03-24T12:42:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:39 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:42:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:39 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:42:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:39.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:39 smithi190 ceph-mon[33748]: pgmap v711: 158 pgs: 1 remapped+peering, 3 peering, 1 unknown, 20 active+clean+remapped, 98 active+clean, 21 active+undersized, 1 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 8.0 KiB/s rd, 32 MiB/s wr, 1.20k op/s; 39023/373377 objects degraded (10.451%); 21/373377 objects misplaced (0.006%); 39 KiB/s, 1 objects/s recovering 2024-03-24T12:42:39.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:39 smithi029 ceph-mon[41500]: 10.0 scrub starts 2024-03-24T12:42:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:39 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:42:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:39 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:42:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:39 smithi029 ceph-mon[41500]: pgmap v711: 158 pgs: 1 remapped+peering, 3 peering, 1 unknown, 20 active+clean+remapped, 98 active+clean, 21 active+undersized, 1 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 8.0 KiB/s rd, 32 MiB/s wr, 1.20k op/s; 39023/373377 objects degraded (10.451%); 21/373377 objects misplaced (0.006%); 39 KiB/s, 1 objects/s recovering 2024-03-24T12:42:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:39 smithi029 ceph-mon[29919]: 10.0 scrub starts 2024-03-24T12:42:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:39 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:42:39.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:39 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:42:39.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:39.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:39 smithi029 ceph-mon[29919]: pgmap v711: 158 pgs: 1 remapped+peering, 3 peering, 1 unknown, 20 active+clean+remapped, 98 active+clean, 21 active+undersized, 1 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 8.0 KiB/s rd, 32 MiB/s wr, 1.20k op/s; 39023/373377 objects degraded (10.451%); 21/373377 objects misplaced (0.006%); 39 KiB/s, 1 objects/s recovering 2024-03-24T12:42:39.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:39.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:39.776 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:39.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:39.816 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:39.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:39.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:39.947 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:39.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:40.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:40.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:40.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:40.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:40.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:40.222 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:40.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:40.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:40.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:40.331 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:40.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:40.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:40.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:40.444 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:40 smithi029 ceph-mon[29919]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T12:42:40.444 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:40 smithi029 ceph-mon[29919]: osdmap e316: 8 total, 7 up, 8 in 2024-03-24T12:42:40.444 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:40 smithi029 ceph-mon[29919]: 10.c deep-scrub starts 2024-03-24T12:42:40.444 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:40 smithi029 ceph-mon[29919]: 10.0 scrub ok 2024-03-24T12:42:40.444 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:40 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 39023/373377 objects degraded (10.451%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:40.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:40 smithi029 ceph-mon[41500]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T12:42:40.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:40 smithi029 ceph-mon[41500]: osdmap e316: 8 total, 7 up, 8 in 2024-03-24T12:42:40.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:40 smithi029 ceph-mon[41500]: 10.c deep-scrub starts 2024-03-24T12:42:40.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:40 smithi029 ceph-mon[41500]: 10.0 scrub ok 2024-03-24T12:42:40.445 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:40 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 39023/373377 objects degraded (10.451%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:40.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:40.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:40.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:40.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:40.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:40.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:40.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:40.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:40 smithi190 ceph-mon[33748]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T12:42:40.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:40 smithi190 ceph-mon[33748]: osdmap e316: 8 total, 7 up, 8 in 2024-03-24T12:42:40.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:40 smithi190 ceph-mon[33748]: 10.c deep-scrub starts 2024-03-24T12:42:40.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:40 smithi190 ceph-mon[33748]: 10.0 scrub ok 2024-03-24T12:42:40.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:40 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 39023/373377 objects degraded (10.451%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:40.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:40.772 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:40.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:40.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:40.923 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:40.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:40.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:40.959 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:41.025 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:41.050 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:41.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:41.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:41.194 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:41.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:41.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:41.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:41.332 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:41.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:41.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:41.434 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:41.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:41.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:41.613 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:41.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:41.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:41 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:42:41.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:41 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:42:41.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:41 smithi190 ceph-mon[33748]: 10.d deep-scrub starts 2024-03-24T12:42:41.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:41 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:42:41.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:41 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:42:41.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:41 smithi190 ceph-mon[33748]: osdmap e317: 8 total, 7 up, 8 in 2024-03-24T12:42:41.635 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:41 smithi190 ceph-mon[33748]: pgmap v714: 158 pgs: 1 remapped+peering, 3 peering, 1 unknown, 20 active+clean+remapped, 98 active+clean, 21 active+undersized, 1 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 1.17k op/s; 38313/369162 objects degraded (10.378%); 21/369162 objects misplaced (0.006%); 27 B/s, 1 objects/s recovering 2024-03-24T12:42:41.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:42:41.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:42:41.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[41500]: 10.d deep-scrub starts 2024-03-24T12:42:41.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:42:41.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:42:41.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[41500]: osdmap e317: 8 total, 7 up, 8 in 2024-03-24T12:42:41.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[41500]: pgmap v714: 158 pgs: 1 remapped+peering, 3 peering, 1 unknown, 20 active+clean+remapped, 98 active+clean, 21 active+undersized, 1 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 1.17k op/s; 38313/369162 objects degraded (10.378%); 21/369162 objects misplaced (0.006%); 27 B/s, 1 objects/s recovering 2024-03-24T12:42:41.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:42:41.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:42:41.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[29919]: 10.d deep-scrub starts 2024-03-24T12:42:41.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:42:41.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:42:41.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[29919]: osdmap e317: 8 total, 7 up, 8 in 2024-03-24T12:42:41.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:41 smithi029 ceph-mon[29919]: pgmap v714: 158 pgs: 1 remapped+peering, 3 peering, 1 unknown, 20 active+clean+remapped, 98 active+clean, 21 active+undersized, 1 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 1.17k op/s; 38313/369162 objects degraded (10.378%); 21/369162 objects misplaced (0.006%); 27 B/s, 1 objects/s recovering 2024-03-24T12:42:41.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:41.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:41.753 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:41.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:41.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:41.894 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:41.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:41.996 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:42.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:42.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:42.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:42.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:42.168 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:42.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:42.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:42.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:42.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:42.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:42.417 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:42.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:42 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:42:42.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:42 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:42:42.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:42 smithi029 ceph-mon[41500]: 10.b scrub starts 2024-03-24T12:42:42.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:42 smithi029 ceph-mon[41500]: 1.2 scrub starts 2024-03-24T12:42:42.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:42 smithi029 ceph-mon[41500]: 1.2 scrub ok 2024-03-24T12:42:42.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:42 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:42:42.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:42 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:42:42.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:42 smithi029 ceph-mon[29919]: 10.b scrub starts 2024-03-24T12:42:42.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:42 smithi029 ceph-mon[29919]: 1.2 scrub starts 2024-03-24T12:42:42.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:42 smithi029 ceph-mon[29919]: 1.2 scrub ok 2024-03-24T12:42:42.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:42.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:42.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:42.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:42.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:42.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:42.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:42 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:42:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:42 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:42:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:42 smithi190 ceph-mon[33748]: 10.b scrub starts 2024-03-24T12:42:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:42 smithi190 ceph-mon[33748]: 1.2 scrub starts 2024-03-24T12:42:42.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:42 smithi190 ceph-mon[33748]: 1.2 scrub ok 2024-03-24T12:42:42.679 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:42.711 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:42.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:42.813 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:42.854 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:42.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:42.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:42.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:43.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:43.125 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:43.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:43.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:43.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:43.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:43.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:43.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:43.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:43.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:43.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:43.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:43.540 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:43.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:43 smithi190 ceph-mon[33748]: 5.1b scrub starts 2024-03-24T12:42:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:43 smithi190 ceph-mon[33748]: 5.1b scrub ok 2024-03-24T12:42:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:43 smithi190 ceph-mon[33748]: pgmap v715: 158 pgs: 1 remapped+peering, 21 active+clean+remapped, 100 active+clean, 21 active+undersized, 2 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 13 MiB/s wr, 1.11k op/s; 38050/364434 objects degraded (10.441%); 21/364434 objects misplaced (0.006%); 0 B/s, 2 objects/s recovering 2024-03-24T12:42:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:43 smithi190 ceph-mon[33748]: 3.8 scrub starts 2024-03-24T12:42:43.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:43 smithi190 ceph-mon[33748]: 3.8 scrub ok 2024-03-24T12:42:43.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:43 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:42:43.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:43 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:42:43.639 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[41500]: 5.1b scrub starts 2024-03-24T12:42:43.639 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[41500]: 5.1b scrub ok 2024-03-24T12:42:43.639 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[41500]: pgmap v715: 158 pgs: 1 remapped+peering, 21 active+clean+remapped, 100 active+clean, 21 active+undersized, 2 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 13 MiB/s wr, 1.11k op/s; 38050/364434 objects degraded (10.441%); 21/364434 objects misplaced (0.006%); 0 B/s, 2 objects/s recovering 2024-03-24T12:42:43.639 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[41500]: 3.8 scrub starts 2024-03-24T12:42:43.639 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[41500]: 3.8 scrub ok 2024-03-24T12:42:43.639 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:42:43.639 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:42:43.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[29919]: 5.1b scrub starts 2024-03-24T12:42:43.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[29919]: 5.1b scrub ok 2024-03-24T12:42:43.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[29919]: pgmap v715: 158 pgs: 1 remapped+peering, 21 active+clean+remapped, 100 active+clean, 21 active+undersized, 2 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.4 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 13 MiB/s wr, 1.11k op/s; 38050/364434 objects degraded (10.441%); 21/364434 objects misplaced (0.006%); 0 B/s, 2 objects/s recovering 2024-03-24T12:42:43.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[29919]: 3.8 scrub starts 2024-03-24T12:42:43.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[29919]: 3.8 scrub ok 2024-03-24T12:42:43.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:42:43.640 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:43 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:42:43.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:43.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:43.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:43.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:43.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:43.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:43.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:43.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:43.959 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:43.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:44.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:44.098 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:44.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:44.200 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:44.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:44.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:44.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:44.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:44.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:44.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:44.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:44.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:44.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:44.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:44.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:44 smithi029 ceph-mon[29919]: 10.b scrub ok 2024-03-24T12:42:44.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:44 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:42:44.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:44 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:42:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:44 smithi029 ceph-mon[41500]: 10.b scrub ok 2024-03-24T12:42:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:44 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:42:44.551 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:44 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:42:44.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:44.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:44 smithi190 ceph-mon[33748]: 10.b scrub ok 2024-03-24T12:42:44.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:44 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:42:44.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:44 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:42:44.658 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:44.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:44.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:44.662 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:42:44.663 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:42:44.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:44.807 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:44.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:44.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:44.948 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:44.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:44.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:45.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:45.104 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:45.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:45.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:45.266 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:45.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:45.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:45.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:45.406 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:45.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:45.508 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:45.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:45.543 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:45.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:45.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:42:45.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:42:45.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: 10.4 deep-scrub starts 2024-03-24T12:42:45.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: osdmap e318: 8 total, 7 up, 8 in 2024-03-24T12:42:45.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:42:45.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:42:45.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4209335733' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:42:45.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: pgmap v717: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 98 active+clean, 21 active+undersized, 3 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 37094/357972 objects degraded (10.362%); 28/357972 objects misplaced (0.008%); 0 B/s, 1 objects/s recovering 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: 6.e deep-scrub starts 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: 6.e deep-scrub ok 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 38050/364434 objects degraded (10.441%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: 10.4 deep-scrub starts 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: osdmap e318: 8 total, 7 up, 8 in 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4209335733' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:42:45.552 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: pgmap v717: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 98 active+clean, 21 active+undersized, 3 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 37094/357972 objects degraded (10.362%); 28/357972 objects misplaced (0.008%); 0 B/s, 1 objects/s recovering 2024-03-24T12:42:45.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: 6.e deep-scrub starts 2024-03-24T12:42:45.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:42:45.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: 6.e deep-scrub ok 2024-03-24T12:42:45.553 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:45 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 38050/364434 objects degraded (10.441%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:45.628 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: 10.4 deep-scrub starts 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: osdmap e318: 8 total, 7 up, 8 in 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4209335733' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: pgmap v717: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 98 active+clean, 21 active+undersized, 3 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 853 op/s; 37094/357972 objects degraded (10.362%); 28/357972 objects misplaced (0.008%); 0 B/s, 1 objects/s recovering 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: 6.e deep-scrub starts 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: 6.e deep-scrub ok 2024-03-24T12:42:45.629 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 38050/364434 objects degraded (10.441%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:45.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:45.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:45.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:45.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:45.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:45.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:45.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:45.882 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:42:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=sqlstore.transactions t=2024-03-24T12:42:45.633573769Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-24T12:42:45.916 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:42:45.917 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:42:45.917 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-03-24T12:42:45.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:42:45.930 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:45.958 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:45.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:45.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:46.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:46.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:46.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:46.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[29919]: 5.3 deep-scrub starts 2024-03-24T12:42:46.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[29919]: 5.3 deep-scrub ok 2024-03-24T12:42:46.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[29919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T12:42:46.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[29919]: 10.e scrub starts 2024-03-24T12:42:46.227 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:42:46.228 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:42:46.228 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:42:46.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:46.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:46.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:46.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:46.403 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:46.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:46.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:46.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:46.546 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:46.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:46.559 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:42:46.559 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:42:46.559 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:42:46.559 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:42:46.560 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:42:46.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[41500]: 5.3 deep-scrub starts 2024-03-24T12:42:46.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[41500]: 5.3 deep-scrub ok 2024-03-24T12:42:46.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[41500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T12:42:46.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[41500]: 10.e scrub starts 2024-03-24T12:42:46.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:42:46.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:42:46.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:42:46.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:46 smithi190 ceph-mon[33748]: 5.3 deep-scrub starts 2024-03-24T12:42:46.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:46 smithi190 ceph-mon[33748]: 5.3 deep-scrub ok 2024-03-24T12:42:46.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:46 smithi190 ceph-mon[33748]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T12:42:46.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:46 smithi190 ceph-mon[33748]: 10.e scrub starts 2024-03-24T12:42:46.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:42:46.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:42:46.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:42:46.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:46.680 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:46.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:46.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:46.812 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:46.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:46.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:46.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:46.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:46.951 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:46.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:47.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:47.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:47.081 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:47.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:47.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:47.222 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:47.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:47.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:47.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:47.367 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:47.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:47.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:47.506 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:47.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:47.606 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:47 smithi029 ceph-mon[41500]: pgmap v718: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 97 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 36903/355641 objects degraded (10.376%); 28/355641 objects misplaced (0.008%); 0 B/s, 1 objects/s recovering 2024-03-24T12:42:47.606 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:47 smithi029 ceph-mon[41500]: 10.5 scrub starts 2024-03-24T12:42:47.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:47 smithi029 ceph-mon[29919]: pgmap v718: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 97 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 36903/355641 objects degraded (10.376%); 28/355641 objects misplaced (0.008%); 0 B/s, 1 objects/s recovering 2024-03-24T12:42:47.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:47 smithi029 ceph-mon[29919]: 10.5 scrub starts 2024-03-24T12:42:47.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:47 smithi190 ceph-mon[33748]: pgmap v718: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 97 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 777 op/s; 36903/355641 objects degraded (10.376%); 28/355641 objects misplaced (0.008%); 0 B/s, 1 objects/s recovering 2024-03-24T12:42:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:47 smithi190 ceph-mon[33748]: 10.5 scrub starts 2024-03-24T12:42:47.644 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:47.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:47.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:47.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:47.781 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:47.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:47.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:47.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:47.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:42:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:42:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:42:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:47.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:47.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:48.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:48.062 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:48.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:48.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:48.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:48.188 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:48.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:48.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:48.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:48.326 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:48.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:48.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:48.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:48.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:48.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:48.583 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:48.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:48.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:42:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:42:48.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:48 smithi190 ceph-mon[33748]: 10.e scrub ok 2024-03-24T12:42:48.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:48 smithi190 ceph-mon[33748]: 5.14 scrub starts 2024-03-24T12:42:48.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:48 smithi190 ceph-mon[33748]: 5.14 scrub ok 2024-03-24T12:42:48.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:48 smithi029 ceph-mon[29919]: 10.e scrub ok 2024-03-24T12:42:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:48 smithi029 ceph-mon[29919]: 5.14 scrub starts 2024-03-24T12:42:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:48 smithi029 ceph-mon[29919]: 5.14 scrub ok 2024-03-24T12:42:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:48 smithi029 ceph-mon[41500]: 10.e scrub ok 2024-03-24T12:42:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:48 smithi029 ceph-mon[41500]: 5.14 scrub starts 2024-03-24T12:42:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:48 smithi029 ceph-mon[41500]: 5.14 scrub ok 2024-03-24T12:42:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:42:48.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:48.708 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:48.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:48.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:48.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:48.849 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:48.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:48.951 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:48.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:48.995 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:48.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:49.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:49.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:49.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:49.122 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:49.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:49.251 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:49.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:49.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:49.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:49.388 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:49.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:49.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:49.529 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:49.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:49.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:49.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:49 smithi190 ceph-mon[33748]: pgmap v719: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 97 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 36451/350376 objects degraded (10.403%); 28/350376 objects misplaced (0.008%); 0 B/s, 0 objects/s recovering 2024-03-24T12:42:49.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:49.655 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:49.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:49.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:49.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:49 smithi029 ceph-mon[41500]: pgmap v719: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 97 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 36451/350376 objects degraded (10.403%); 28/350376 objects misplaced (0.008%); 0 B/s, 0 objects/s recovering 2024-03-24T12:42:49.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:42:49.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:49 smithi029 ceph-mon[29919]: pgmap v719: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 97 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 0 B/s wr, 727 op/s; 36451/350376 objects degraded (10.403%); 28/350376 objects misplaced (0.008%); 0 B/s, 0 objects/s recovering 2024-03-24T12:42:49.756 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:49.787 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:49.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:49.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:49.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:50.026 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:50.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:50.063 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:50.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:50.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:50.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:50.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:50.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:50.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:50.321 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:50.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:50.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:50.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:50.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:50.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:50.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:50.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:50.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:50 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:42:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:50 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:42:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:50 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 36451/350376 objects degraded (10.403%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:50.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:50 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:42:50.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:50 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:42:50.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:50 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 36451/350376 objects degraded (10.403%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:50.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:50 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:42:50.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:50 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:42:50.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:50 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 36451/350376 objects degraded (10.403%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:50.683 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:50.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:50.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:50.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:50.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:50.855 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:50.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:50.957 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:50.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:50.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:51.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:51.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:51.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:51.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:51.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:51.251 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:51.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:51.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:51.354 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:51.382 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:51.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:51.484 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:51.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:51.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:51.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:51 smithi190 ceph-mon[33748]: osdmap e319: 8 total, 7 up, 8 in 2024-03-24T12:42:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:51 smithi190 ceph-mon[33748]: pgmap v721: 158 pgs: 2 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s; 36160/348609 objects degraded (10.373%); 28/348609 objects misplaced (0.008%) 2024-03-24T12:42:51.649 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:51.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:51.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:51.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:51 smithi029 ceph-mon[41500]: osdmap e319: 8 total, 7 up, 8 in 2024-03-24T12:42:51.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:51 smithi029 ceph-mon[41500]: pgmap v721: 158 pgs: 2 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s; 36160/348609 objects degraded (10.373%); 28/348609 objects misplaced (0.008%) 2024-03-24T12:42:51.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:51 smithi029 ceph-mon[29919]: osdmap e319: 8 total, 7 up, 8 in 2024-03-24T12:42:51.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:51 smithi029 ceph-mon[29919]: pgmap v721: 158 pgs: 2 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 659 op/s; 36160/348609 objects degraded (10.373%); 28/348609 objects misplaced (0.008%) 2024-03-24T12:42:51.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:51.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:51.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:51.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:51.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:51.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:52.022 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:52.062 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:52.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:52.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:52.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:52.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:52.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:52.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:52.316 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:52.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:52.417 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:52.456 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:52.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:52.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:52.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:52.589 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:52.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:52.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:52.728 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:52.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:52.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:52.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:52.854 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:52.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:52.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:52.956 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:52.986 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:52.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:52.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:53.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:53.130 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:53.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:53.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:53.271 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:53.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:53.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:53.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:53.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T12:42:53.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:53.408 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:53.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:53.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:53.551 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:53.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:53.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:53.731 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:53.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:53.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:53.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:53.871 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:53.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:53.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:53 smithi029 ceph-mon[29919]: pgmap v722: 158 pgs: 4 active+clean+scrubbing, 22 active+clean+remapped, 94 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 542 op/s; 36016/346029 objects degraded (10.408%); 28/346029 objects misplaced (0.008%) 2024-03-24T12:42:53.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:53 smithi029 ceph-mon[41500]: pgmap v722: 158 pgs: 4 active+clean+scrubbing, 22 active+clean+remapped, 94 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 542 op/s; 36016/346029 objects degraded (10.408%); 28/346029 objects misplaced (0.008%) 2024-03-24T12:42:53.972 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:54.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:54.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:54.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:54.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:54.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:53 smithi190 ceph-mon[33748]: pgmap v722: 158 pgs: 4 active+clean+scrubbing, 22 active+clean+remapped, 94 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 542 op/s; 36016/346029 objects degraded (10.408%); 28/346029 objects misplaced (0.008%) 2024-03-24T12:42:54.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:53 smithi190 ceph-mon[33748]: 10.5 scrub ok 2024-03-24T12:42:54.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:53 smithi190 ceph-mon[33748]: 10.1 scrub starts 2024-03-24T12:42:54.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:54.152 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:54.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:53 smithi029 ceph-mon[29919]: 10.5 scrub ok 2024-03-24T12:42:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:53 smithi029 ceph-mon[29919]: 10.1 scrub starts 2024-03-24T12:42:54.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:53 smithi029 ceph-mon[41500]: 10.5 scrub ok 2024-03-24T12:42:54.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:53 smithi029 ceph-mon[41500]: 10.1 scrub starts 2024-03-24T12:42:54.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:42:54.258 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:42:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:54.288 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:54.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:54.389 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:54.427 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:54.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:54.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:54.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:54.544 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:54.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:54.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:54.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:54.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:54.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:54.698 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:54.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:54.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:54.859 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:54.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:54.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:55.025 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:55.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:55.070 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:54 smithi029 ceph-mon[29919]: 3.12 scrub starts 2024-03-24T12:42:55.070 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:54 smithi029 ceph-mon[29919]: 3.12 scrub ok 2024-03-24T12:42:55.071 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:54 smithi029 ceph-mon[29919]: 10.8 deep-scrub starts 2024-03-24T12:42:55.071 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:54 smithi029 ceph-mon[41500]: 3.12 scrub starts 2024-03-24T12:42:55.071 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:54 smithi029 ceph-mon[41500]: 3.12 scrub ok 2024-03-24T12:42:55.071 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:54 smithi029 ceph-mon[41500]: 10.8 deep-scrub starts 2024-03-24T12:42:55.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:54 smithi190 ceph-mon[33748]: 3.12 scrub starts 2024-03-24T12:42:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:54 smithi190 ceph-mon[33748]: 3.12 scrub ok 2024-03-24T12:42:55.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:54 smithi190 ceph-mon[33748]: 10.8 deep-scrub starts 2024-03-24T12:42:55.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:55.171 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:55.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:55.175 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:42:55.273 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:55.306 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:55.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:55.408 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:55.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:55.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:55.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:55.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:55.578 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:55.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:55.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:55.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:55.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:55.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:55.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:55.852 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:55.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:55.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:55.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:55.983 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:55.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:56.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:56.123 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:56.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:56.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:55 smithi190 ceph-mon[33748]: 10.1 scrub ok 2024-03-24T12:42:56.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:55 smithi190 ceph-mon[33748]: pgmap v723: 158 pgs: 3 active+clean+scrubbing, 22 active+clean+remapped, 95 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 520 op/s; 35476/342372 objects degraded (10.362%); 28/342372 objects misplaced (0.008%) 2024-03-24T12:42:56.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:55 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3714283070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:42:56.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:55 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:42:56.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:55 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 35476/342372 objects degraded (10.362%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:55 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:42:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:55 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:42:56.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:55 smithi190 ceph-mon[33748]: osdmap e320: 8 total, 7 up, 8 in 2024-03-24T12:42:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[29919]: 10.1 scrub ok 2024-03-24T12:42:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[29919]: pgmap v723: 158 pgs: 3 active+clean+scrubbing, 22 active+clean+remapped, 95 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 520 op/s; 35476/342372 objects degraded (10.362%); 28/342372 objects misplaced (0.008%) 2024-03-24T12:42:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3714283070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:42:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:42:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 35476/342372 objects degraded (10.362%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[29919]: osdmap e320: 8 total, 7 up, 8 in 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[41500]: 10.1 scrub ok 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[41500]: pgmap v723: 158 pgs: 3 active+clean+scrubbing, 22 active+clean+remapped, 95 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 7.0 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 520 op/s; 35476/342372 objects degraded (10.362%); 28/342372 objects misplaced (0.008%) 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3714283070' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 35476/342372 objects degraded (10.362%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:42:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:55 smithi029 ceph-mon[41500]: osdmap e320: 8 total, 7 up, 8 in 2024-03-24T12:42:56.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:56.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:42:56.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:56.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:56.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:56.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:56.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:56.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:56.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:56.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:56.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:56.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:56.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:56.674 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:56.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:56.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:42:56.815 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:56.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:56.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:42:56.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:56.965 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:56.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:56.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:57.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:57.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:57.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:57.140 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:57.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:57.258 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:57.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:57.359 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:57.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:57.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:57.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:57.567 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:57.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:57.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:57.669 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:57.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:57.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:57.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:57.779 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:42:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:42:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:42:57.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:57.863 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:57.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:57.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:58.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:58.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:58.103 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:57 smithi029 ceph-mon[41500]: pgmap v725: 158 pgs: 3 active+clean+scrubbing, 22 active+clean+remapped, 95 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 354 op/s; 35476/341886 objects degraded (10.377%); 28/341886 objects misplaced (0.008%) 2024-03-24T12:42:58.104 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:57 smithi029 ceph-mon[29919]: pgmap v725: 158 pgs: 3 active+clean+scrubbing, 22 active+clean+remapped, 95 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 354 op/s; 35476/341886 objects degraded (10.377%); 28/341886 objects misplaced (0.008%) 2024-03-24T12:42:58.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:58.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:57 smithi190 ceph-mon[33748]: pgmap v725: 158 pgs: 3 active+clean+scrubbing, 22 active+clean+remapped, 95 active+clean, 21 active+undersized, 4 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.9 GiB data, 24 GiB used, 692 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 354 op/s; 35476/341886 objects degraded (10.377%); 28/341886 objects misplaced (0.008%) 2024-03-24T12:42:58.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:58.149 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:58.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:58.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:58.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:58.288 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:58.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:58.343 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:42:58.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:42:58.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:58.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:58.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:42:58.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:58.561 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:58.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:58.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:42:58.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:42:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:42:58.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:58.697 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:58.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:58.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:42:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:58.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:58.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:42:58.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:58.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:58.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:58.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:59.073 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:42:59.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:59.088 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:42:59.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:59.131 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:59.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:59.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:42:59.280 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:59.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:59.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:42:59.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:59.418 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:59.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:42:59.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:59.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:59.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:59.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:42:59.702 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:59.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:59.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:42:59.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:42:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:59.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:42:59.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:42:59.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:42:59.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:42:59.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:42:59.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:00.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:00.115 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:00.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:00.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:00.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:00.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:59 smithi190 ceph-mon[33748]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:43:00.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:42:59 smithi190 ceph-mon[33748]: pgmap v726: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 480 op/s; 35024/336369 objects degraded (10.412%); 28/336369 objects misplaced (0.008%) 2024-03-24T12:43:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:59 smithi029 ceph-mon[29919]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:43:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:42:59 smithi029 ceph-mon[29919]: pgmap v726: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 480 op/s; 35024/336369 objects degraded (10.412%); 28/336369 objects misplaced (0.008%) 2024-03-24T12:43:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:59 smithi029 ceph-mon[41500]: from='client.15333 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:43:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:42:59 smithi029 ceph-mon[41500]: pgmap v726: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 480 op/s; 35024/336369 objects degraded (10.412%); 28/336369 objects misplaced (0.008%) 2024-03-24T12:43:00.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:00.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:00.252 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:00.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:00.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:00.392 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:00.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:00.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:00.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:00.528 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:00.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:00.629 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:00.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:00.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:00.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:00.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:00.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:00.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:00.869 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":725,"stamp":"2024-03-24T12:42:56.774916+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7456168310,"num_objects":113962,"num_object_clones":0,"num_object_copies":341886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35476,"num_objects_misplaced":28,"num_objects_unfound":0,"num_objects_dirty":113962,"num_whiteouts":0,"num_read":2142,"num_read_kb":1894,"num_write":140902,"num_write_kb":8126190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":2038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82905,"ondisk_log_size":82905,"up":418,"acting":440,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":497,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":24813640,"kb_used_data":20331264,"kb_used_omap":12,"kb_used_meta":4482227,"kb_avail":725147576,"statfs":{"total":767960285184,"available":742551117824,"internally_reserved":0,"allocated":20819214336,"data_stored":20813813557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4589801009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,11,21,8,10,6,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-185466880,"num_objects":-2830,"num_object_clones":0,"num_object_copies":-8490,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-975,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2830,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2830,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003691"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":544,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.260719+0000","last_change":"2024-03-24T12:35:58.701806+0000","last_active":"2024-03-24T12:42:50.260719+0000","last_peered":"2024-03-24T12:42:50.260719+0000","last_clean":"2024-03-24T12:35:54.676897+0000","last_became_active":"2024-03-24T12:35:58.701806+0000","last_became_peered":"2024-03-24T12:35:58.701806+0000","last_unstale":"2024-03-24T12:42:50.260719+0000","last_undegraded":"2024-03-24T12:42:50.260719+0000","last_fullsized":"2024-03-24T12:35:58.695795+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:08.859869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.859869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":464,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.264846+0000","last_change":"2024-03-24T12:39:07.750122+0000","last_active":"2024-03-24T12:42:50.264846+0000","last_peered":"2024-03-24T12:42:50.264846+0000","last_clean":"2024-03-24T12:35:54.687134+0000","last_became_active":"2024-03-24T12:39:07.750122+0000","last_became_peered":"2024-03-24T12:39:07.750122+0000","last_unstale":"2024-03-24T12:42:50.264846+0000","last_undegraded":"2024-03-24T12:42:50.264846+0000","last_fullsized":"2024-03-24T12:39:07.746192+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:48.744937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:48.744937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":550,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.468853+0000","last_change":"2024-03-24T12:35:58.712452+0000","last_active":"2024-03-24T12:42:50.468853+0000","last_peered":"2024-03-24T12:42:50.468853+0000","last_clean":"2024-03-24T12:35:54.683285+0000","last_became_active":"2024-03-24T12:35:58.712452+0000","last_became_peered":"2024-03-24T12:35:58.712452+0000","last_unstale":"2024-03-24T12:42:50.468853+0000","last_undegraded":"2024-03-24T12:42:50.468853+0000","last_fullsized":"2024-03-24T12:35:58.707789+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:14.966328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:14.966328+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":["0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"207'15","reported_seq":630,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.260469+0000","last_change":"2024-03-24T12:42:33.772791+0000","last_active":"2024-03-24T12:42:50.260469+0000","last_peered":"2024-03-24T12:42:50.260469+0000","last_clean":"2024-03-24T12:42:50.260469+0000","last_became_active":"2024-03-24T12:39:34.752258+0000","last_became_peered":"2024-03-24T12:39:34.752258+0000","last_unstale":"2024-03-24T12:42:50.260469+0000","last_undegraded":"2024-03-24T12:42:50.260469+0000","last_fullsized":"2024-03-24T12:42:50.260469+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:42:33.772520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:33.772520+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:52.241582+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":561,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.353665+0000","last_change":"2024-03-24T12:42:15.775318+0000","last_active":"2024-03-24T12:42:50.353665+0000","last_peered":"2024-03-24T12:42:50.353665+0000","last_clean":"2024-03-24T12:42:50.353665+0000","last_became_active":"2024-03-24T12:39:34.758425+0000","last_became_peered":"2024-03-24T12:39:34.758425+0000","last_unstale":"2024-03-24T12:42:50.353665+0000","last_undegraded":"2024-03-24T12:42:50.353665+0000","last_fullsized":"2024-03-24T12:42:50.353665+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:15.775222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:42:15.775222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:19.455764+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":581,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.260566+0000","last_change":"2024-03-24T12:42:25.785915+0000","last_active":"2024-03-24T12:42:50.260566+0000","last_peered":"2024-03-24T12:42:50.260566+0000","last_clean":"2024-03-24T12:42:50.260566+0000","last_became_active":"2024-03-24T12:39:34.744030+0000","last_became_peered":"2024-03-24T12:39:34.744030+0000","last_unstale":"2024-03-24T12:42:50.260566+0000","last_undegraded":"2024-03-24T12:42:50.260566+0000","last_fullsized":"2024-03-24T12:42:50.260566+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:25.785810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:17.828354+0000","last_clean_scrub_stamp":"2024-03-24T12:42:25.785810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:52.345979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":575,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.352284+0000","last_change":"2024-03-24T12:40:20.094901+0000","last_active":"2024-03-24T12:42:50.352284+0000","last_peered":"2024-03-24T12:42:50.352284+0000","last_clean":"2024-03-24T12:42:50.352284+0000","last_became_active":"2024-03-24T12:39:34.761198+0000","last_became_peered":"2024-03-24T12:39:34.761198+0000","last_unstale":"2024-03-24T12:42:50.352284+0000","last_undegraded":"2024-03-24T12:42:50.352284+0000","last_fullsized":"2024-03-24T12:42:50.352284+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:20.094789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:20.094789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"207'34","reported_seq":607,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349838+0000","last_change":"2024-03-24T12:40:57.892747+0000","last_active":"2024-03-24T12:42:50.349838+0000","last_peered":"2024-03-24T12:42:50.349838+0000","last_clean":"2024-03-24T12:42:50.349838+0000","last_became_active":"2024-03-24T12:26:05.320564+0000","last_became_peered":"2024-03-24T12:26:05.320564+0000","last_unstale":"2024-03-24T12:42:50.349838+0000","last_undegraded":"2024-03-24T12:42:50.349838+0000","last_fullsized":"2024-03-24T12:42:50.349838+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-24T12:40:57.892649+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2024-03-24T12:27:25.396514+0000","last_clean_scrub_stamp":"2024-03-24T12:40:57.892649+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":565,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.351789+0000","last_change":"2024-03-24T12:42:09.586054+0000","last_active":"2024-03-24T12:42:50.351789+0000","last_peered":"2024-03-24T12:42:50.351789+0000","last_clean":"2024-03-24T12:42:50.351789+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:42:50.351789+0000","last_undegraded":"2024-03-24T12:42:50.351789+0000","last_fullsized":"2024-03-24T12:42:50.351789+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:09.585933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:42:09.585933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:17.659621+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":253,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.467103+0000","last_change":"2024-03-24T12:42:42.059162+0000","last_active":"2024-03-24T12:42:50.467103+0000","last_peered":"2024-03-24T12:42:50.467103+0000","last_clean":"2024-03-24T12:42:50.467103+0000","last_became_active":"2024-03-24T12:42:38.210182+0000","last_became_peered":"2024-03-24T12:42:38.210182+0000","last_unstale":"2024-03-24T12:42:50.467103+0000","last_undegraded":"2024-03-24T12:42:50.467103+0000","last_fullsized":"2024-03-24T12:42:50.467103+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:42.058922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:42.058922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:49.150568+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,7],"avail_no_missing":["0","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":265,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.262586+0000","last_change":"2024-03-24T12:42:12.733978+0000","last_active":"2024-03-24T12:42:50.262586+0000","last_peered":"2024-03-24T12:42:50.262586+0000","last_clean":"2024-03-24T12:42:50.262586+0000","last_became_active":"2024-03-24T12:39:35.101206+0000","last_became_peered":"2024-03-24T12:39:35.101206+0000","last_unstale":"2024-03-24T12:42:50.262586+0000","last_undegraded":"2024-03-24T12:42:50.262586+0000","last_fullsized":"2024-03-24T12:42:50.262586+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:12.733723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:12.733723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:32.949408+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,1],"avail_no_missing":["4","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1a","version":"207'15","reported_seq":590,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.261047+0000","last_change":"2024-03-24T12:42:11.750569+0000","last_active":"2024-03-24T12:42:50.261047+0000","last_peered":"2024-03-24T12:42:50.261047+0000","last_clean":"2024-03-24T12:42:50.261047+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:42:50.261047+0000","last_undegraded":"2024-03-24T12:42:50.261047+0000","last_fullsized":"2024-03-24T12:42:50.261047+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:42:11.750435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:11.750435+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:29.759520+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":239,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.356226+0000","last_change":"2024-03-24T12:35:58.718922+0000","last_active":"2024-03-24T12:42:50.356226+0000","last_peered":"2024-03-24T12:42:50.356226+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.718922+0000","last_became_peered":"2024-03-24T12:35:58.718922+0000","last_unstale":"2024-03-24T12:42:50.356226+0000","last_undegraded":"2024-03-24T12:42:50.356226+0000","last_fullsized":"2024-03-24T12:35:58.702792+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.155345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.155345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:51.617227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":569,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264110+0000","last_change":"2024-03-24T12:42:24.564287+0000","last_active":"2024-03-24T12:42:50.264110+0000","last_peered":"2024-03-24T12:42:50.264110+0000","last_clean":"2024-03-24T12:42:50.264110+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:42:50.264110+0000","last_undegraded":"2024-03-24T12:42:50.264110+0000","last_fullsized":"2024-03-24T12:42:50.264110+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:24.564198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:42:24.564198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:42.542214+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":239,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.260648+0000","last_change":"2024-03-24T12:35:58.708742+0000","last_active":"2024-03-24T12:42:50.260648+0000","last_peered":"2024-03-24T12:42:50.260648+0000","last_clean":"2024-03-24T12:26:02.143860+0000","last_became_active":"2024-03-24T12:35:58.708742+0000","last_became_peered":"2024-03-24T12:35:58.708742+0000","last_unstale":"2024-03-24T12:42:50.260648+0000","last_undegraded":"2024-03-24T12:42:50.260648+0000","last_fullsized":"2024-03-24T12:35:58.699613+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:23.095706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:33:23.095706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"207'7","reported_seq":586,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.261580+0000","last_change":"2024-03-24T12:42:26.834814+0000","last_active":"2024-03-24T12:42:50.261580+0000","last_peered":"2024-03-24T12:42:50.261580+0000","last_clean":"2024-03-24T12:42:50.261580+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:42:50.261580+0000","last_undegraded":"2024-03-24T12:42:50.261580+0000","last_fullsized":"2024-03-24T12:42:50.261580+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-24T12:42:26.834710+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:26.834710+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:56.704017+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":572,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.266985+0000","last_change":"2024-03-24T12:42:14.740313+0000","last_active":"2024-03-24T12:42:50.266985+0000","last_peered":"2024-03-24T12:42:50.266985+0000","last_clean":"2024-03-24T12:42:50.266985+0000","last_became_active":"2024-03-24T12:39:35.101660+0000","last_became_peered":"2024-03-24T12:39:35.101660+0000","last_unstale":"2024-03-24T12:42:50.266985+0000","last_undegraded":"2024-03-24T12:42:50.266985+0000","last_fullsized":"2024-03-24T12:42:50.266985+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:14.740133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:42:14.740133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:16.630185+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,3],"avail_no_missing":["4","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":585,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.465780+0000","last_change":"2024-03-24T12:42:20.570443+0000","last_active":"2024-03-24T12:42:50.465780+0000","last_peered":"2024-03-24T12:42:50.465780+0000","last_clean":"2024-03-24T12:42:50.465780+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:42:50.465780+0000","last_undegraded":"2024-03-24T12:42:50.465780+0000","last_fullsized":"2024-03-24T12:42:50.465780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:20.569742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:15.270454+0000","last_clean_scrub_stamp":"2024-03-24T12:42:20.569742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:21.676394+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":569,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.467890+0000","last_change":"2024-03-24T12:40:58.334350+0000","last_active":"2024-03-24T12:42:50.467890+0000","last_peered":"2024-03-24T12:42:50.467890+0000","last_clean":"2024-03-24T12:42:50.467890+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:42:50.467890+0000","last_undegraded":"2024-03-24T12:42:50.467890+0000","last_fullsized":"2024-03-24T12:42:50.467890+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:58.334266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:40:58.334266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"207'27","reported_seq":616,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349226+0000","last_change":"2024-03-24T12:42:05.946260+0000","last_active":"2024-03-24T12:42:50.349226+0000","last_peered":"2024-03-24T12:42:50.349226+0000","last_clean":"2024-03-24T12:42:50.349226+0000","last_became_active":"2024-03-24T12:26:05.322440+0000","last_became_peered":"2024-03-24T12:26:05.322440+0000","last_unstale":"2024-03-24T12:42:50.349226+0000","last_undegraded":"2024-03-24T12:42:50.349226+0000","last_fullsized":"2024-03-24T12:42:50.349226+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-24T12:42:05.945985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:05.945985+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:08.208229+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":556,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.357583+0000","last_change":"2024-03-24T12:39:59.167526+0000","last_active":"2024-03-24T12:42:50.357583+0000","last_peered":"2024-03-24T12:42:50.357583+0000","last_clean":"2024-03-24T12:42:50.357583+0000","last_became_active":"2024-03-24T12:39:35.129797+0000","last_became_peered":"2024-03-24T12:39:35.129797+0000","last_unstale":"2024-03-24T12:42:50.357583+0000","last_undegraded":"2024-03-24T12:42:50.357583+0000","last_fullsized":"2024-03-24T12:42:50.357583+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:59.167308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:39:59.167308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":196,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":569,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.261150+0000","last_change":"2024-03-24T12:42:22.806404+0000","last_active":"2024-03-24T12:42:50.261150+0000","last_peered":"2024-03-24T12:42:50.261150+0000","last_clean":"2024-03-24T12:42:50.261150+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:42:50.261150+0000","last_undegraded":"2024-03-24T12:42:50.261150+0000","last_fullsized":"2024-03-24T12:42:50.261150+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:22.806267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:42:22.806267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:48.727061+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":577,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.261192+0000","last_change":"2024-03-24T12:42:35.818909+0000","last_active":"2024-03-24T12:42:50.261192+0000","last_peered":"2024-03-24T12:42:50.261192+0000","last_clean":"2024-03-24T12:42:50.261192+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:42:50.261192+0000","last_undegraded":"2024-03-24T12:42:50.261192+0000","last_fullsized":"2024-03-24T12:42:50.261192+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:35.818787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:35.818787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:00.645273+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"207'18","reported_seq":589,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.353177+0000","last_change":"2024-03-24T12:42:30.783198+0000","last_active":"2024-03-24T12:42:50.353177+0000","last_peered":"2024-03-24T12:42:50.353177+0000","last_clean":"2024-03-24T12:42:50.353177+0000","last_became_active":"2024-03-24T12:26:05.321002+0000","last_became_peered":"2024-03-24T12:26:05.321002+0000","last_unstale":"2024-03-24T12:42:50.353177+0000","last_undegraded":"2024-03-24T12:42:50.353177+0000","last_fullsized":"2024-03-24T12:42:50.353177+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-24T12:42:30.783104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:30.783104+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:00.488035+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":537,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:50.303767+0000","last_change":"2024-03-24T12:35:58.711430+0000","last_active":"2024-03-24T12:42:50.303767+0000","last_peered":"2024-03-24T12:42:50.303767+0000","last_clean":"2024-03-24T12:35:54.688013+0000","last_became_active":"2024-03-24T12:35:58.711430+0000","last_became_peered":"2024-03-24T12:35:58.711430+0000","last_unstale":"2024-03-24T12:42:50.303767+0000","last_undegraded":"2024-03-24T12:35:58.697902+0000","last_fullsized":"2024-03-24T12:35:58.697784+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:34:42.820487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:42.820487+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"315'11","reported_seq":209,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.271486+0000","last_change":"2024-03-24T12:42:37.293967+0000","last_active":"2024-03-24T12:42:50.271486+0000","last_peered":"2024-03-24T12:42:50.271486+0000","last_clean":"2024-03-24T12:42:50.271486+0000","last_became_active":"2024-03-24T12:42:37.207937+0000","last_became_peered":"2024-03-24T12:42:37.207937+0000","last_unstale":"2024-03-24T12:42:50.271486+0000","last_undegraded":"2024-03-24T12:42:50.271486+0000","last_fullsized":"2024-03-24T12:42:50.271486+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-24T12:42:33.862557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:33.862557+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:49.713276+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"67'1","reported_seq":586,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.356108+0000","last_change":"2024-03-24T12:42:11.540145+0000","last_active":"2024-03-24T12:42:50.356108+0000","last_peered":"2024-03-24T12:42:50.356108+0000","last_clean":"2024-03-24T12:42:50.356108+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:42:50.356108+0000","last_undegraded":"2024-03-24T12:42:50.356108+0000","last_fullsized":"2024-03-24T12:42:50.356108+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:42:11.540016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:11.540016+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:30.080998+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"207'20","reported_seq":623,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.467640+0000","last_change":"2024-03-24T12:42:37.223302+0000","last_active":"2024-03-24T12:42:50.467640+0000","last_peered":"2024-03-24T12:42:50.467640+0000","last_clean":"2024-03-24T12:42:50.467640+0000","last_became_active":"2024-03-24T12:42:37.185902+0000","last_became_peered":"2024-03-24T12:42:37.185902+0000","last_unstale":"2024-03-24T12:42:50.467640+0000","last_undegraded":"2024-03-24T12:42:50.467640+0000","last_fullsized":"2024-03-24T12:42:50.467640+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:42:15.590187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:15.590187+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:39.288298+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":459,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.262196+0000","last_change":"2024-03-24T12:39:07.748878+0000","last_active":"2024-03-24T12:42:50.262196+0000","last_peered":"2024-03-24T12:42:50.262196+0000","last_clean":"2024-03-24T12:35:54.688035+0000","last_became_active":"2024-03-24T12:39:07.748878+0000","last_became_peered":"2024-03-24T12:39:07.748878+0000","last_unstale":"2024-03-24T12:42:50.262196+0000","last_undegraded":"2024-03-24T12:42:50.262196+0000","last_fullsized":"2024-03-24T12:39:07.746317+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:03.555975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:03.555975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":563,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.467992+0000","last_change":"2024-03-24T12:42:34.936269+0000","last_active":"2024-03-24T12:42:50.467992+0000","last_peered":"2024-03-24T12:42:50.467992+0000","last_clean":"2024-03-24T12:42:50.467992+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:42:50.467992+0000","last_undegraded":"2024-03-24T12:42:50.467992+0000","last_fullsized":"2024-03-24T12:42:50.467992+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:34.936156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:34.936156+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:36.636757+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":561,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349220+0000","last_change":"2024-03-24T12:42:09.642246+0000","last_active":"2024-03-24T12:42:50.349220+0000","last_peered":"2024-03-24T12:42:50.349220+0000","last_clean":"2024-03-24T12:42:50.349220+0000","last_became_active":"2024-03-24T12:26:03.165366+0000","last_became_peered":"2024-03-24T12:26:03.165366+0000","last_unstale":"2024-03-24T12:42:50.349220+0000","last_undegraded":"2024-03-24T12:42:50.349220+0000","last_fullsized":"2024-03-24T12:42:50.349220+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:09.642102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:42:09.642102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:22.605254+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"207'19","reported_seq":560,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.271169+0000","last_change":"2024-03-24T12:42:10.830989+0000","last_active":"2024-03-24T12:42:50.271169+0000","last_peered":"2024-03-24T12:42:50.271169+0000","last_clean":"2024-03-24T12:42:50.271169+0000","last_became_active":"2024-03-24T12:39:35.107173+0000","last_became_peered":"2024-03-24T12:39:35.107173+0000","last_unstale":"2024-03-24T12:42:50.271169+0000","last_undegraded":"2024-03-24T12:42:50.271169+0000","last_fullsized":"2024-03-24T12:42:50.271169+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:42:10.830713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:10.830713+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:26.417060+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1,0],"avail_no_missing":["6","0","1"],"object_location_counts":[{"shards":"0,1,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":566,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264083+0000","last_change":"2024-03-24T12:42:31.536965+0000","last_active":"2024-03-24T12:42:50.264083+0000","last_peered":"2024-03-24T12:42:50.264083+0000","last_clean":"2024-03-24T12:42:50.264083+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:42:50.264083+0000","last_undegraded":"2024-03-24T12:42:50.264083+0000","last_fullsized":"2024-03-24T12:42:50.264083+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:31.536830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:42:31.536830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:35.340706+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":156,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.265304+0000","last_change":"2024-03-24T12:42:15.761824+0000","last_active":"2024-03-24T12:42:50.265304+0000","last_peered":"2024-03-24T12:42:50.265304+0000","last_clean":"2024-03-24T12:42:50.265304+0000","last_became_active":"2024-03-24T12:39:34.751127+0000","last_became_peered":"2024-03-24T12:39:34.751127+0000","last_unstale":"2024-03-24T12:42:50.265304+0000","last_undegraded":"2024-03-24T12:42:50.265304+0000","last_fullsized":"2024-03-24T12:42:50.265304+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:15.761745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:15.761745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:40.554503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":548,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.470206+0000","last_change":"2024-03-24T12:35:58.718236+0000","last_active":"2024-03-24T12:42:50.470206+0000","last_peered":"2024-03-24T12:42:50.470206+0000","last_clean":"2024-03-24T12:35:54.683418+0000","last_became_active":"2024-03-24T12:35:58.718236+0000","last_became_peered":"2024-03-24T12:35:58.718236+0000","last_unstale":"2024-03-24T12:42:50.470206+0000","last_undegraded":"2024-03-24T12:42:50.470206+0000","last_fullsized":"2024-03-24T12:35:58.708687+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:04.720633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:04.720633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":["0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"207'8","reported_seq":594,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349376+0000","last_change":"2024-03-24T12:40:12.127815+0000","last_active":"2024-03-24T12:42:50.349376+0000","last_peered":"2024-03-24T12:42:50.349376+0000","last_clean":"2024-03-24T12:42:50.349376+0000","last_became_active":"2024-03-24T12:39:34.759619+0000","last_became_peered":"2024-03-24T12:39:34.759619+0000","last_unstale":"2024-03-24T12:42:50.349376+0000","last_undegraded":"2024-03-24T12:42:50.349376+0000","last_fullsized":"2024-03-24T12:42:50.349376+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-24T12:40:12.127602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:12.127602+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":562,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.353755+0000","last_change":"2024-03-24T12:42:21.736402+0000","last_active":"2024-03-24T12:42:50.353755+0000","last_peered":"2024-03-24T12:42:50.353755+0000","last_clean":"2024-03-24T12:42:50.353755+0000","last_became_active":"2024-03-24T12:39:34.763422+0000","last_became_peered":"2024-03-24T12:39:34.763422+0000","last_unstale":"2024-03-24T12:42:50.353755+0000","last_undegraded":"2024-03-24T12:42:50.353755+0000","last_fullsized":"2024-03-24T12:42:50.353755+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:21.736266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:42:21.736266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:31.269852+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":582,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264848+0000","last_change":"2024-03-24T12:42:29.526666+0000","last_active":"2024-03-24T12:42:50.264848+0000","last_peered":"2024-03-24T12:42:50.264848+0000","last_clean":"2024-03-24T12:42:50.264848+0000","last_became_active":"2024-03-24T12:39:34.752397+0000","last_became_peered":"2024-03-24T12:39:34.752397+0000","last_unstale":"2024-03-24T12:42:50.264848+0000","last_undegraded":"2024-03-24T12:42:50.264848+0000","last_fullsized":"2024-03-24T12:42:50.264848+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:29.526581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:29.526581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:39.125275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":247,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:50.264694+0000","last_change":"2024-03-24T12:35:58.708757+0000","last_active":"2024-03-24T12:42:50.264694+0000","last_peered":"2024-03-24T12:42:50.264694+0000","last_clean":"2024-03-24T12:26:03.166952+0000","last_became_active":"2024-03-24T12:35:58.708757+0000","last_became_peered":"2024-03-24T12:35:58.708757+0000","last_unstale":"2024-03-24T12:42:50.264694+0000","last_undegraded":"2024-03-24T12:35:58.698998+0000","last_fullsized":"2024-03-24T12:35:58.698185+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:33:28.059288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:33:28.059288+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"207'19","reported_seq":603,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349748+0000","last_change":"2024-03-24T12:40:54.825783+0000","last_active":"2024-03-24T12:42:50.349748+0000","last_peered":"2024-03-24T12:42:50.349748+0000","last_clean":"2024-03-24T12:42:50.349748+0000","last_became_active":"2024-03-24T12:39:34.750031+0000","last_became_peered":"2024-03-24T12:39:34.750031+0000","last_unstale":"2024-03-24T12:42:50.349748+0000","last_undegraded":"2024-03-24T12:42:50.349748+0000","last_fullsized":"2024-03-24T12:42:50.349748+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:40:54.825683+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:40:54.825683+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":537,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.350214+0000","last_change":"2024-03-24T12:35:58.706939+0000","last_active":"2024-03-24T12:42:50.350214+0000","last_peered":"2024-03-24T12:42:50.350214+0000","last_clean":"2024-03-24T12:35:54.681672+0000","last_became_active":"2024-03-24T12:35:58.706939+0000","last_became_peered":"2024-03-24T12:35:58.706939+0000","last_unstale":"2024-03-24T12:42:50.350214+0000","last_undegraded":"2024-03-24T12:42:50.350214+0000","last_fullsized":"2024-03-24T12:35:58.695715+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:53.895902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:34:53.895902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":167,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":["0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":554,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.353554+0000","last_change":"2024-03-24T12:41:02.921331+0000","last_active":"2024-03-24T12:42:50.353554+0000","last_peered":"2024-03-24T12:42:50.353554+0000","last_clean":"2024-03-24T12:42:50.353554+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:42:50.353554+0000","last_undegraded":"2024-03-24T12:42:50.353554+0000","last_fullsized":"2024-03-24T12:42:50.353554+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:41:02.921251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:31.383330+0000","last_clean_scrub_stamp":"2024-03-24T12:41:02.921251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":577,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264517+0000","last_change":"2024-03-24T12:42:30.538310+0000","last_active":"2024-03-24T12:42:50.264517+0000","last_peered":"2024-03-24T12:42:50.264517+0000","last_clean":"2024-03-24T12:42:50.264517+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:42:50.264517+0000","last_undegraded":"2024-03-24T12:42:50.264517+0000","last_fullsized":"2024-03-24T12:42:50.264517+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:30.538193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:30.538193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:52.436948+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"207'15","reported_seq":259,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.266562+0000","last_change":"2024-03-24T12:42:19.778987+0000","last_active":"2024-03-24T12:42:50.266562+0000","last_peered":"2024-03-24T12:42:50.266562+0000","last_clean":"2024-03-24T12:42:50.266562+0000","last_became_active":"2024-03-24T12:39:35.103739+0000","last_became_peered":"2024-03-24T12:39:35.103739+0000","last_unstale":"2024-03-24T12:42:50.266562+0000","last_undegraded":"2024-03-24T12:42:50.266562+0000","last_fullsized":"2024-03-24T12:42:50.266562+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:42:19.778429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:19.778429+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:26.607957+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2,0],"avail_no_missing":["6","0","2"],"object_location_counts":[{"shards":"0,2,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":537,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.349324+0000","last_change":"2024-03-24T12:35:58.714875+0000","last_active":"2024-03-24T12:42:50.349324+0000","last_peered":"2024-03-24T12:42:50.349324+0000","last_clean":"2024-03-24T12:35:54.689143+0000","last_became_active":"2024-03-24T12:35:58.714875+0000","last_became_peered":"2024-03-24T12:35:58.714875+0000","last_unstale":"2024-03-24T12:42:50.349324+0000","last_undegraded":"2024-03-24T12:42:50.349324+0000","last_fullsized":"2024-03-24T12:35:58.702457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.898988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.898988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":["3","0"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":549,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.356572+0000","last_change":"2024-03-24T12:35:58.718562+0000","last_active":"2024-03-24T12:42:50.356572+0000","last_peered":"2024-03-24T12:42:50.356572+0000","last_clean":"2024-03-24T12:35:54.687557+0000","last_became_active":"2024-03-24T12:35:58.718562+0000","last_became_peered":"2024-03-24T12:35:58.718562+0000","last_unstale":"2024-03-24T12:42:50.356572+0000","last_undegraded":"2024-03-24T12:42:50.356572+0000","last_fullsized":"2024-03-24T12:35:58.697301+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:21.144883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:33:21.144883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:51.617214+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":590,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.260893+0000","last_change":"2024-03-24T12:42:06.797147+0000","last_active":"2024-03-24T12:42:50.260893+0000","last_peered":"2024-03-24T12:42:50.260893+0000","last_clean":"2024-03-24T12:42:50.260893+0000","last_became_active":"2024-03-24T12:39:34.748131+0000","last_became_peered":"2024-03-24T12:39:34.748131+0000","last_unstale":"2024-03-24T12:42:50.260893+0000","last_undegraded":"2024-03-24T12:42:50.260893+0000","last_fullsized":"2024-03-24T12:42:50.260893+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:06.797037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:06.797037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:23.995016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"207'24","reported_seq":628,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.261881+0000","last_change":"2024-03-24T12:42:15.815223+0000","last_active":"2024-03-24T12:42:50.261881+0000","last_peered":"2024-03-24T12:42:50.261881+0000","last_clean":"2024-03-24T12:42:50.261881+0000","last_became_active":"2024-03-24T12:39:35.101521+0000","last_became_peered":"2024-03-24T12:39:35.101521+0000","last_unstale":"2024-03-24T12:42:50.261881+0000","last_undegraded":"2024-03-24T12:42:50.261881+0000","last_fullsized":"2024-03-24T12:42:50.261881+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'24","last_scrub_stamp":"2024-03-24T12:42:15.815110+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:42:15.815110+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:22.366041+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":558,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.467356+0000","last_change":"2024-03-24T12:42:10.624592+0000","last_active":"2024-03-24T12:42:50.467356+0000","last_peered":"2024-03-24T12:42:50.467356+0000","last_clean":"2024-03-24T12:42:50.467356+0000","last_became_active":"2024-03-24T12:26:09.349477+0000","last_became_peered":"2024-03-24T12:26:09.349477+0000","last_unstale":"2024-03-24T12:42:50.467356+0000","last_undegraded":"2024-03-24T12:42:50.467356+0000","last_fullsized":"2024-03-24T12:42:50.467356+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:42:10.624471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:42:10.624471+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:21.443833+0000","scrub_duration":89,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"216'10","reported_seq":320,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264392+0000","last_change":"2024-03-24T12:42:17.636032+0000","last_active":"2024-03-24T12:42:50.264392+0000","last_peered":"2024-03-24T12:42:50.264392+0000","last_clean":"2024-03-24T12:42:50.264392+0000","last_became_active":"2024-03-24T12:36:52.065482+0000","last_became_peered":"2024-03-24T12:36:52.065482+0000","last_unstale":"2024-03-24T12:42:50.264392+0000","last_undegraded":"2024-03-24T12:42:50.264392+0000","last_fullsized":"2024-03-24T12:42:50.264392+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"216'10","last_scrub_stamp":"2024-03-24T12:42:17.635887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:17.635887+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:31.812206+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":571,"reported_epoch":319,"state":"active+clean+scrubbing","last_fresh":"2024-03-24T12:42:50.264441+0000","last_change":"2024-03-24T12:42:46.519429+0000","last_active":"2024-03-24T12:42:50.264441+0000","last_peered":"2024-03-24T12:42:50.264441+0000","last_clean":"2024-03-24T12:42:50.264441+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:42:50.264441+0000","last_undegraded":"2024-03-24T12:42:50.264441+0000","last_fullsized":"2024-03-24T12:42:50.264441+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:56.893413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:56.893413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"207'20","reported_seq":596,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.352357+0000","last_change":"2024-03-24T12:42:23.755669+0000","last_active":"2024-03-24T12:42:50.352357+0000","last_peered":"2024-03-24T12:42:50.352357+0000","last_clean":"2024-03-24T12:42:50.352357+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:42:50.352357+0000","last_undegraded":"2024-03-24T12:42:50.352357+0000","last_fullsized":"2024-03-24T12:42:50.352357+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:42:23.755427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:23.755427+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:50.660930+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":544,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.262637+0000","last_change":"2024-03-24T12:35:58.712757+0000","last_active":"2024-03-24T12:42:50.262637+0000","last_peered":"2024-03-24T12:42:50.262637+0000","last_clean":"2024-03-24T12:35:54.677408+0000","last_became_active":"2024-03-24T12:35:58.712757+0000","last_became_peered":"2024-03-24T12:35:58.712757+0000","last_unstale":"2024-03-24T12:42:50.262637+0000","last_undegraded":"2024-03-24T12:42:50.262637+0000","last_fullsized":"2024-03-24T12:35:58.700636+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:01.913287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:01.913287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":["4","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"93'9","reported_seq":765,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.352726+0000","last_change":"2024-03-24T12:42:47.601660+0000","last_active":"2024-03-24T12:42:50.352726+0000","last_peered":"2024-03-24T12:42:50.352726+0000","last_clean":"2024-03-24T12:42:50.352726+0000","last_became_active":"2024-03-24T12:26:07.337610+0000","last_became_peered":"2024-03-24T12:26:07.337610+0000","last_unstale":"2024-03-24T12:42:50.352726+0000","last_undegraded":"2024-03-24T12:42:50.352726+0000","last_fullsized":"2024-03-24T12:42:50.352726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'9","last_scrub_stamp":"2024-03-24T12:42:47.601304+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:42:47.601304+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:05.257774+0000","scrub_duration":841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":571,"reported_epoch":319,"state":"active+clean+scrubbing","last_fresh":"2024-03-24T12:42:50.468594+0000","last_change":"2024-03-24T12:42:47.156628+0000","last_active":"2024-03-24T12:42:50.468594+0000","last_peered":"2024-03-24T12:42:50.468594+0000","last_clean":"2024-03-24T12:42:50.468594+0000","last_became_active":"2024-03-24T12:26:03.168618+0000","last_became_peered":"2024-03-24T12:26:03.168618+0000","last_unstale":"2024-03-24T12:42:50.468594+0000","last_undegraded":"2024-03-24T12:42:50.468594+0000","last_fullsized":"2024-03-24T12:42:50.468594+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:53.346202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:53.346202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"207'19","reported_seq":316,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:50.264796+0000","last_change":"2024-03-24T12:35:58.751467+0000","last_active":"2024-03-24T12:42:50.264796+0000","last_peered":"2024-03-24T12:42:50.264796+0000","last_clean":"2024-03-24T12:27:16.240812+0000","last_became_active":"2024-03-24T12:35:58.751467+0000","last_became_peered":"2024-03-24T12:35:58.751467+0000","last_unstale":"2024-03-24T12:42:50.264796+0000","last_undegraded":"2024-03-24T12:35:58.700376+0000","last_fullsized":"2024-03-24T12:35:58.700265+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"88'13","last_scrub_stamp":"2024-03-24T12:35:08.002032+0000","last_deep_scrub":"88'13","last_deep_scrub_stamp":"2024-03-24T12:27:16.242776+0000","last_clean_scrub_stamp":"2024-03-24T12:35:08.002032+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":574,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349763+0000","last_change":"2024-03-24T12:42:13.990715+0000","last_active":"2024-03-24T12:42:50.349763+0000","last_peered":"2024-03-24T12:42:50.349763+0000","last_clean":"2024-03-24T12:42:50.349763+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:42:50.349763+0000","last_undegraded":"2024-03-24T12:42:50.349763+0000","last_fullsized":"2024-03-24T12:42:50.349763+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:13.990600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:42:13.990600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:36.941356+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":553,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.357395+0000","last_change":"2024-03-24T12:42:33.802250+0000","last_active":"2024-03-24T12:42:50.357395+0000","last_peered":"2024-03-24T12:42:50.357395+0000","last_clean":"2024-03-24T12:42:50.357395+0000","last_became_active":"2024-03-24T12:26:07.342672+0000","last_became_peered":"2024-03-24T12:26:07.342672+0000","last_unstale":"2024-03-24T12:42:50.357395+0000","last_undegraded":"2024-03-24T12:42:50.357395+0000","last_fullsized":"2024-03-24T12:42:50.357395+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:33.802138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:33.802138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:00.017438+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":591,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.265074+0000","last_change":"2024-03-24T12:42:20.558696+0000","last_active":"2024-03-24T12:42:50.265074+0000","last_peered":"2024-03-24T12:42:50.265074+0000","last_clean":"2024-03-24T12:42:50.265074+0000","last_became_active":"2024-03-24T12:39:34.755495+0000","last_became_peered":"2024-03-24T12:39:34.755495+0000","last_unstale":"2024-03-24T12:42:50.265074+0000","last_undegraded":"2024-03-24T12:42:50.265074+0000","last_fullsized":"2024-03-24T12:42:50.265074+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:20.558610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:20.558610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:35.271539+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"207'15","reported_seq":614,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.467735+0000","last_change":"2024-03-24T12:42:37.240267+0000","last_active":"2024-03-24T12:42:50.467735+0000","last_peered":"2024-03-24T12:42:50.467735+0000","last_clean":"2024-03-24T12:42:50.467735+0000","last_became_active":"2024-03-24T12:42:37.188617+0000","last_became_peered":"2024-03-24T12:42:37.188617+0000","last_unstale":"2024-03-24T12:42:50.467735+0000","last_undegraded":"2024-03-24T12:42:50.467735+0000","last_fullsized":"2024-03-24T12:42:50.467735+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:42:32.014662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:32.014662+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:58.941542+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":567,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264425+0000","last_change":"2024-03-24T12:42:10.653041+0000","last_active":"2024-03-24T12:42:50.264425+0000","last_peered":"2024-03-24T12:42:50.264425+0000","last_clean":"2024-03-24T12:42:50.264425+0000","last_became_active":"2024-03-24T12:39:35.100242+0000","last_became_peered":"2024-03-24T12:39:35.100242+0000","last_unstale":"2024-03-24T12:42:50.264425+0000","last_undegraded":"2024-03-24T12:42:50.264425+0000","last_fullsized":"2024-03-24T12:42:50.264425+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:10.652919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:55.271679+0000","last_clean_scrub_stamp":"2024-03-24T12:42:10.652919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:21.452935+0000","scrub_duration":49,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":240,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.352449+0000","last_change":"2024-03-24T12:35:58.714488+0000","last_active":"2024-03-24T12:42:50.352449+0000","last_peered":"2024-03-24T12:42:50.352449+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.714488+0000","last_became_peered":"2024-03-24T12:35:58.714488+0000","last_unstale":"2024-03-24T12:42:50.352449+0000","last_undegraded":"2024-03-24T12:42:50.352449+0000","last_fullsized":"2024-03-24T12:35:58.700893+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:10.995618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:10.995618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":533,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.263943+0000","last_change":"2024-03-24T12:42:25.822489+0000","last_active":"2024-03-24T12:42:50.263943+0000","last_peered":"2024-03-24T12:42:50.263943+0000","last_clean":"2024-03-24T12:42:50.263943+0000","last_became_active":"2024-03-24T12:39:35.120926+0000","last_became_peered":"2024-03-24T12:39:35.120926+0000","last_unstale":"2024-03-24T12:42:50.263943+0000","last_undegraded":"2024-03-24T12:42:50.263943+0000","last_fullsized":"2024-03-24T12:42:50.263943+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:25.822211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:25.822211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:33.608251+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,3],"avail_no_missing":["6","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"207'12","reported_seq":589,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.357582+0000","last_change":"2024-03-24T12:42:37.311334+0000","last_active":"2024-03-24T12:42:50.357582+0000","last_peered":"2024-03-24T12:42:50.357582+0000","last_clean":"2024-03-24T12:42:50.357582+0000","last_became_active":"2024-03-24T12:42:37.207944+0000","last_became_peered":"2024-03-24T12:42:37.207944+0000","last_unstale":"2024-03-24T12:42:50.357582+0000","last_undegraded":"2024-03-24T12:42:50.357582+0000","last_fullsized":"2024-03-24T12:42:50.357582+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-24T12:42:17.648076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:17.648076+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:24.717508+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.6","version":"319'8655","reported_seq":8744,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:51.675989+0000","last_change":"2024-03-24T12:41:04.289719+0000","last_active":"2024-03-24T12:42:51.675989+0000","last_peered":"2024-03-24T12:42:51.675989+0000","last_clean":"2024-03-24T12:40:50.114364+0000","last_became_active":"2024-03-24T12:40:51.131277+0000","last_became_peered":"2024-03-24T12:40:51.131277+0000","last_unstale":"2024-03-24T12:42:51.675989+0000","last_undegraded":"2024-03-24T12:41:04.288099+0000","last_fullsized":"2024-03-24T12:40:51.129259+0000","mapping_epoch":278,"log_start":"288'4104","ondisk_log_start":"288'4104","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:52.156565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463536128,"num_objects":7073,"num_object_clones":0,"num_object_copies":21219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8655,"num_write_kb":503296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":7073}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":215,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.266284+0000","last_change":"2024-03-24T12:42:11.842544+0000","last_active":"2024-03-24T12:42:50.266284+0000","last_peered":"2024-03-24T12:42:50.266284+0000","last_clean":"2024-03-24T12:42:50.266284+0000","last_became_active":"2024-03-24T12:39:35.103777+0000","last_became_peered":"2024-03-24T12:39:35.103777+0000","last_unstale":"2024-03-24T12:42:50.266284+0000","last_undegraded":"2024-03-24T12:42:50.266284+0000","last_fullsized":"2024-03-24T12:42:50.266284+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:11.842306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:42:11.842306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:31.941027+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,2],"avail_no_missing":["6","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":582,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264540+0000","last_change":"2024-03-24T12:42:02.652141+0000","last_active":"2024-03-24T12:42:50.264540+0000","last_peered":"2024-03-24T12:42:50.264540+0000","last_clean":"2024-03-24T12:42:50.264540+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:42:50.264540+0000","last_undegraded":"2024-03-24T12:42:50.264540+0000","last_fullsized":"2024-03-24T12:42:50.264540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:42:02.652009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:02.652009+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:18.245790+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"207'27","reported_seq":93,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.264153+0000","last_change":"2024-03-24T12:42:38.175044+0000","last_active":"2024-03-24T12:42:50.264153+0000","last_peered":"2024-03-24T12:42:50.264153+0000","last_clean":"2024-03-24T12:42:50.264153+0000","last_became_active":"2024-03-24T12:42:38.174245+0000","last_became_peered":"2024-03-24T12:42:38.174245+0000","last_unstale":"2024-03-24T12:42:50.264153+0000","last_undegraded":"2024-03-24T12:42:50.264153+0000","last_fullsized":"2024-03-24T12:42:50.264153+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-24T12:42:11.648832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:11.648832+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:39.148734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4,7],"avail_no_missing":["6","4","7"],"object_location_counts":[{"shards":"4,6,7","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.9","version":"258'10","reported_seq":803,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.265225+0000","last_change":"2024-03-24T12:42:03.632986+0000","last_active":"2024-03-24T12:42:50.265225+0000","last_peered":"2024-03-24T12:42:50.265225+0000","last_clean":"2024-03-24T12:42:50.265225+0000","last_became_active":"2024-03-24T12:39:34.753277+0000","last_became_peered":"2024-03-24T12:39:34.753277+0000","last_unstale":"2024-03-24T12:42:50.265225+0000","last_undegraded":"2024-03-24T12:42:50.265225+0000","last_fullsized":"2024-03-24T12:42:50.265225+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'10","last_scrub_stamp":"2024-03-24T12:42:03.632860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:03.632860+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:11.929336+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.7","version":"319'8952","reported_seq":9041,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:52.996606+0000","last_change":"2024-03-24T12:41:04.285775+0000","last_active":"2024-03-24T12:42:52.996606+0000","last_peered":"2024-03-24T12:42:52.996606+0000","last_clean":"2024-03-24T12:40:50.114364+0000","last_became_active":"2024-03-24T12:40:51.398233+0000","last_became_peered":"2024-03-24T12:40:51.398233+0000","last_unstale":"2024-03-24T12:42:52.996606+0000","last_undegraded":"2024-03-24T12:41:04.285284+0000","last_fullsized":"2024-03-24T12:40:51.133672+0000","mapping_epoch":278,"log_start":"286'3589","ondisk_log_start":"286'3589","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":5363,"log_dups_size":0,"ondisk_log_size":5363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465567744,"num_objects":7104,"num_object_clones":0,"num_object_copies":21312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7104,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8952,"num_write_kb":513792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[{"shards":"3,6","objects":7104}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":548,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.353674+0000","last_change":"2024-03-24T12:42:37.189497+0000","last_active":"2024-03-24T12:42:50.353674+0000","last_peered":"2024-03-24T12:42:50.353674+0000","last_clean":"2024-03-24T12:42:50.353674+0000","last_became_active":"2024-03-24T12:42:37.187442+0000","last_became_peered":"2024-03-24T12:42:37.187442+0000","last_unstale":"2024-03-24T12:42:50.353674+0000","last_undegraded":"2024-03-24T12:42:50.353674+0000","last_fullsized":"2024-03-24T12:42:50.353674+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:51.992860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:37:51.992860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":570,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264922+0000","last_change":"2024-03-24T12:42:04.658947+0000","last_active":"2024-03-24T12:42:50.264922+0000","last_peered":"2024-03-24T12:42:50.264922+0000","last_clean":"2024-03-24T12:42:50.264922+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:42:50.264922+0000","last_undegraded":"2024-03-24T12:42:50.264922+0000","last_fullsized":"2024-03-24T12:42:50.264922+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:04.658820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:04.658820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:18.867468+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"207'23","reported_seq":611,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.262183+0000","last_change":"2024-03-24T12:41:04.012667+0000","last_active":"2024-03-24T12:42:50.262183+0000","last_peered":"2024-03-24T12:42:50.262183+0000","last_clean":"2024-03-24T12:42:50.262183+0000","last_became_active":"2024-03-24T12:26:05.322138+0000","last_became_peered":"2024-03-24T12:26:05.322138+0000","last_unstale":"2024-03-24T12:42:50.262183+0000","last_undegraded":"2024-03-24T12:42:50.262183+0000","last_fullsized":"2024-03-24T12:42:50.262183+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'23","last_scrub_stamp":"2024-03-24T12:41:04.012539+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:41:04.012539+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":578,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.354662+0000","last_change":"2024-03-24T12:42:40.073242+0000","last_active":"2024-03-24T12:42:50.354662+0000","last_peered":"2024-03-24T12:42:50.354662+0000","last_clean":"2024-03-24T12:42:50.354662+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:42:50.354662+0000","last_undegraded":"2024-03-24T12:42:50.354662+0000","last_fullsized":"2024-03-24T12:42:50.354662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:40.073165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:40.073165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:04.032093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.4","version":"319'9344","reported_seq":9451,"reported_epoch":319,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T12:42:51.803868+0000","last_change":"2024-03-24T12:42:43.998407+0000","last_active":"2024-03-24T12:42:51.803868+0000","last_peered":"2024-03-24T12:42:51.803868+0000","last_clean":"2024-03-24T12:42:51.803868+0000","last_became_active":"2024-03-24T12:40:51.397617+0000","last_became_peered":"2024-03-24T12:40:51.397617+0000","last_unstale":"2024-03-24T12:42:51.803868+0000","last_undegraded":"2024-03-24T12:42:51.803868+0000","last_fullsized":"2024-03-24T12:42:51.803868+0000","mapping_epoch":278,"log_start":"296'4808","ondisk_log_start":"296'4808","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":823,"log_size":4536,"log_dups_size":0,"ondisk_log_size":4536,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 7s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463536128,"num_objects":7073,"num_object_clones":0,"num_object_copies":21219,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7073,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8639,"num_write_kb":502784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":566,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264007+0000","last_change":"2024-03-24T12:42:27.528368+0000","last_active":"2024-03-24T12:42:50.264007+0000","last_peered":"2024-03-24T12:42:50.264007+0000","last_clean":"2024-03-24T12:42:50.264007+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:42:50.264007+0000","last_undegraded":"2024-03-24T12:42:50.264007+0000","last_fullsized":"2024-03-24T12:42:50.264007+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:27.528248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:42:27.528248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:31.397327+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":549,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.350763+0000","last_change":"2024-03-24T12:35:58.711865+0000","last_active":"2024-03-24T12:42:50.350763+0000","last_peered":"2024-03-24T12:42:50.350763+0000","last_clean":"2024-03-24T12:35:54.681353+0000","last_became_active":"2024-03-24T12:35:58.711865+0000","last_became_peered":"2024-03-24T12:35:58.711865+0000","last_unstale":"2024-03-24T12:42:50.350763+0000","last_undegraded":"2024-03-24T12:42:50.350763+0000","last_fullsized":"2024-03-24T12:35:58.701950+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:16.019255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.019255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:52.013098+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":["2","0"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":575,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.264710+0000","last_change":"2024-03-24T12:42:08.627364+0000","last_active":"2024-03-24T12:42:50.264710+0000","last_peered":"2024-03-24T12:42:50.264710+0000","last_clean":"2024-03-24T12:42:50.264710+0000","last_became_active":"2024-03-24T12:39:35.119214+0000","last_became_peered":"2024-03-24T12:39:35.119214+0000","last_unstale":"2024-03-24T12:42:50.264710+0000","last_undegraded":"2024-03-24T12:42:50.264710+0000","last_fullsized":"2024-03-24T12:42:50.264710+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:08.627139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:42:08.627139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:11.979279+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6,3],"avail_no_missing":["7","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"207'35","reported_seq":597,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.266939+0000","last_change":"2024-03-24T12:42:22.032188+0000","last_active":"2024-03-24T12:42:50.266939+0000","last_peered":"2024-03-24T12:42:50.266939+0000","last_clean":"2024-03-24T12:42:50.266939+0000","last_became_active":"2024-03-24T12:39:35.101391+0000","last_became_peered":"2024-03-24T12:39:35.101391+0000","last_unstale":"2024-03-24T12:42:50.266939+0000","last_undegraded":"2024-03-24T12:42:50.266939+0000","last_fullsized":"2024-03-24T12:42:50.266939+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-24T12:42:22.032042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:22.032042+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:22.441946+0000","scrub_duration":206,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"319'8811","reported_seq":8984,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:54.142438+0000","last_change":"2024-03-24T12:42:53.072288+0000","last_active":"2024-03-24T12:42:54.142438+0000","last_peered":"2024-03-24T12:42:54.142438+0000","last_clean":"2024-03-24T12:42:54.142438+0000","last_became_active":"2024-03-24T12:40:51.397589+0000","last_became_peered":"2024-03-24T12:40:51.397589+0000","last_unstale":"2024-03-24T12:42:54.142438+0000","last_undegraded":"2024-03-24T12:42:54.142438+0000","last_fullsized":"2024-03-24T12:42:54.142438+0000","mapping_epoch":278,"log_start":"286'3386","ondisk_log_start":"286'3386","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"319'8774","last_scrub_stamp":"2024-03-24T12:42:53.071811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:42:53.071811+0000","objects_scrubbed":7229,"log_size":5425,"log_dups_size":0,"ondisk_log_size":5425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":7,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:06.220681+0000","scrub_duration":6055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466026496,"num_objects":7111,"num_object_clones":0,"num_object_copies":21333,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7111,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8811,"num_write_kb":509504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":558,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.351730+0000","last_change":"2024-03-24T12:42:23.989802+0000","last_active":"2024-03-24T12:42:50.351730+0000","last_peered":"2024-03-24T12:42:50.351730+0000","last_clean":"2024-03-24T12:42:50.351730+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:42:50.351730+0000","last_undegraded":"2024-03-24T12:42:50.351730+0000","last_fullsized":"2024-03-24T12:42:50.351730+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:23.989350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:42:23.989350+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:52.338400+0000","scrub_duration":226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":578,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349275+0000","last_change":"2024-03-24T12:42:36.035479+0000","last_active":"2024-03-24T12:42:50.349275+0000","last_peered":"2024-03-24T12:42:50.349275+0000","last_clean":"2024-03-24T12:42:50.349275+0000","last_became_active":"2024-03-24T12:26:07.329461+0000","last_became_peered":"2024-03-24T12:26:07.329461+0000","last_unstale":"2024-03-24T12:42:50.349275+0000","last_undegraded":"2024-03-24T12:42:50.349275+0000","last_fullsized":"2024-03-24T12:42:50.349275+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:36.035324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:38.596133+0000","last_clean_scrub_stamp":"2024-03-24T12:42:36.035324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:45.925121+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":320,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.352459+0000","last_change":"2024-03-24T12:40:03.059161+0000","last_active":"2024-03-24T12:42:50.352459+0000","last_peered":"2024-03-24T12:42:50.352459+0000","last_clean":"2024-03-24T12:42:50.352459+0000","last_became_active":"2024-03-24T12:39:35.119146+0000","last_became_peered":"2024-03-24T12:39:35.119146+0000","last_unstale":"2024-03-24T12:42:50.352459+0000","last_undegraded":"2024-03-24T12:42:50.352459+0000","last_fullsized":"2024-03-24T12:42:50.352459+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:40:03.054668+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:40:03.054668+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[{"shards":"1,3,6","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"207'15","reported_seq":607,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.467642+0000","last_change":"2024-03-24T12:42:19.733040+0000","last_active":"2024-03-24T12:42:50.467642+0000","last_peered":"2024-03-24T12:42:50.467642+0000","last_clean":"2024-03-24T12:42:50.467642+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:42:50.467642+0000","last_undegraded":"2024-03-24T12:42:50.467642+0000","last_fullsized":"2024-03-24T12:42:50.467642+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:42:19.732879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:19.732879+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:35.681702+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":363,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.351109+0000","last_change":"2024-03-24T12:42:03.576756+0000","last_active":"2024-03-24T12:42:50.351109+0000","last_peered":"2024-03-24T12:42:50.351109+0000","last_clean":"2024-03-24T12:42:50.351109+0000","last_became_active":"2024-03-24T12:41:52.954156+0000","last_became_peered":"2024-03-24T12:41:52.954156+0000","last_unstale":"2024-03-24T12:42:50.351109+0000","last_undegraded":"2024-03-24T12:42:50.351109+0000","last_fullsized":"2024-03-24T12:42:50.351109+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:03.576247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:03.576247+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:18.953767+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2,6],"avail_no_missing":["0","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"319'8807","reported_seq":8896,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:53.033439+0000","last_change":"2024-03-24T12:41:04.302350+0000","last_active":"2024-03-24T12:42:53.033439+0000","last_peered":"2024-03-24T12:42:53.033439+0000","last_clean":"2024-03-24T12:40:50.114364+0000","last_became_active":"2024-03-24T12:40:51.136373+0000","last_became_peered":"2024-03-24T12:40:51.136373+0000","last_unstale":"2024-03-24T12:42:53.033439+0000","last_undegraded":"2024-03-24T12:41:04.286161+0000","last_fullsized":"2024-03-24T12:40:51.132279+0000","mapping_epoch":278,"log_start":"286'3536","ondisk_log_start":"286'3536","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":5271,"log_dups_size":0,"ondisk_log_size":5271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474677248,"num_objects":7243,"num_object_clones":0,"num_object_copies":21729,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7243,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7243,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8807,"num_write_kb":513600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":7243}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":550,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.352636+0000","last_change":"2024-03-24T12:42:18.638726+0000","last_active":"2024-03-24T12:42:50.352636+0000","last_peered":"2024-03-24T12:42:50.352636+0000","last_clean":"2024-03-24T12:42:50.352636+0000","last_became_active":"2024-03-24T12:26:09.347558+0000","last_became_peered":"2024-03-24T12:26:09.347558+0000","last_unstale":"2024-03-24T12:42:50.352636+0000","last_undegraded":"2024-03-24T12:42:50.352636+0000","last_fullsized":"2024-03-24T12:42:50.352636+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:18.638598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:42:18.638598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:25.223424+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":296,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.356404+0000","last_change":"2024-03-24T12:42:08.567699+0000","last_active":"2024-03-24T12:42:50.356404+0000","last_peered":"2024-03-24T12:42:50.356404+0000","last_clean":"2024-03-24T12:42:50.356404+0000","last_became_active":"2024-03-24T12:39:35.101653+0000","last_became_peered":"2024-03-24T12:39:35.101653+0000","last_unstale":"2024-03-24T12:42:50.356404+0000","last_undegraded":"2024-03-24T12:42:50.356404+0000","last_fullsized":"2024-03-24T12:42:50.356404+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:42:08.567241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:08.567241+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:18.213412+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6,0],"avail_no_missing":["1","0","6"],"object_location_counts":[{"shards":"0,1,6","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":591,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.261678+0000","last_change":"2024-03-24T12:42:43.931063+0000","last_active":"2024-03-24T12:42:50.261678+0000","last_peered":"2024-03-24T12:42:50.261678+0000","last_clean":"2024-03-24T12:42:50.261678+0000","last_became_active":"2024-03-24T12:39:34.751547+0000","last_became_peered":"2024-03-24T12:39:34.751547+0000","last_unstale":"2024-03-24T12:42:50.261678+0000","last_undegraded":"2024-03-24T12:42:50.261678+0000","last_fullsized":"2024-03-24T12:42:50.261678+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:43.930950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:43.930950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:03.549420+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"248'51","reported_seq":736,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349380+0000","last_change":"2024-03-24T12:42:11.056886+0000","last_active":"2024-03-24T12:42:50.349380+0000","last_peered":"2024-03-24T12:42:50.349380+0000","last_clean":"2024-03-24T12:42:50.349380+0000","last_became_active":"2024-03-24T12:39:07.750351+0000","last_became_peered":"2024-03-24T12:39:07.750351+0000","last_unstale":"2024-03-24T12:42:50.349380+0000","last_undegraded":"2024-03-24T12:42:50.349380+0000","last_fullsized":"2024-03-24T12:42:50.349380+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'51","last_scrub_stamp":"2024-03-24T12:42:11.056740+0000","last_deep_scrub":"248'51","last_deep_scrub_stamp":"2024-03-24T12:39:47.606538+0000","last_clean_scrub_stamp":"2024-03-24T12:42:11.056740+0000","objects_scrubbed":10,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:11.764136+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":562,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.529336+0000","last_change":"2024-03-24T12:42:04.544218+0000","last_active":"2024-03-24T12:42:50.529336+0000","last_peered":"2024-03-24T12:42:50.529336+0000","last_clean":"2024-03-24T12:42:50.529336+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:42:50.529336+0000","last_undegraded":"2024-03-24T12:42:50.529336+0000","last_fullsized":"2024-03-24T12:42:50.529336+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:04.544105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:04.544105+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:31.300807+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":327,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.468276+0000","last_change":"2024-03-24T12:42:41.059197+0000","last_active":"2024-03-24T12:42:50.468276+0000","last_peered":"2024-03-24T12:42:50.468276+0000","last_clean":"2024-03-24T12:42:50.468276+0000","last_became_active":"2024-03-24T12:42:40.200767+0000","last_became_peered":"2024-03-24T12:42:40.200767+0000","last_unstale":"2024-03-24T12:42:50.468276+0000","last_undegraded":"2024-03-24T12:42:50.468276+0000","last_fullsized":"2024-03-24T12:42:50.468276+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:41.059079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:41.059079+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:49.290154+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"319'8676","reported_seq":8765,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:54.199566+0000","last_change":"2024-03-24T12:41:04.289686+0000","last_active":"2024-03-24T12:42:54.199566+0000","last_peered":"2024-03-24T12:42:54.199566+0000","last_clean":"2024-03-24T12:40:50.114364+0000","last_became_active":"2024-03-24T12:40:51.134872+0000","last_became_peered":"2024-03-24T12:40:51.134872+0000","last_unstale":"2024-03-24T12:42:54.199566+0000","last_undegraded":"2024-03-24T12:41:04.283226+0000","last_fullsized":"2024-03-24T12:40:51.131989+0000","mapping_epoch":278,"log_start":"286'3234","ondisk_log_start":"286'3234","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":5442,"log_dups_size":0,"ondisk_log_size":5442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459931648,"num_objects":7018,"num_object_clones":0,"num_object_copies":21054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7018,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8676,"num_write_kb":502208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":7018}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":587,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264931+0000","last_change":"2024-03-24T12:42:25.595016+0000","last_active":"2024-03-24T12:42:50.264931+0000","last_peered":"2024-03-24T12:42:50.264931+0000","last_clean":"2024-03-24T12:42:50.264931+0000","last_became_active":"2024-03-24T12:39:34.755964+0000","last_became_peered":"2024-03-24T12:39:34.755964+0000","last_unstale":"2024-03-24T12:42:50.264931+0000","last_undegraded":"2024-03-24T12:42:50.264931+0000","last_fullsized":"2024-03-24T12:42:50.264931+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:25.594892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:15.087115+0000","last_clean_scrub_stamp":"2024-03-24T12:42:25.594892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:38.294773+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":571,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.354820+0000","last_change":"2024-03-24T12:42:16.042702+0000","last_active":"2024-03-24T12:42:50.354820+0000","last_peered":"2024-03-24T12:42:50.354820+0000","last_clean":"2024-03-24T12:42:50.354820+0000","last_became_active":"2024-03-24T12:39:34.744167+0000","last_became_peered":"2024-03-24T12:39:34.744167+0000","last_unstale":"2024-03-24T12:42:50.354820+0000","last_undegraded":"2024-03-24T12:42:50.354820+0000","last_fullsized":"2024-03-24T12:42:50.354820+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:16.042423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:16.042423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:23.319084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":602,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349098+0000","last_change":"2024-03-24T12:42:42.859614+0000","last_active":"2024-03-24T12:42:50.349098+0000","last_peered":"2024-03-24T12:42:50.349098+0000","last_clean":"2024-03-24T12:42:50.349098+0000","last_became_active":"2024-03-24T12:26:03.158403+0000","last_became_peered":"2024-03-24T12:26:03.158403+0000","last_unstale":"2024-03-24T12:42:50.349098+0000","last_undegraded":"2024-03-24T12:42:50.349098+0000","last_fullsized":"2024-03-24T12:42:50.349098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:42.859504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:42.859504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:10.080949+0000","scrub_duration":1784,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"207'22","reported_seq":591,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:50.465873+0000","last_change":"2024-03-24T12:35:58.709644+0000","last_active":"2024-03-24T12:42:50.465873+0000","last_peered":"2024-03-24T12:42:50.465873+0000","last_clean":"2024-03-24T12:35:54.687814+0000","last_became_active":"2024-03-24T12:35:58.709644+0000","last_became_peered":"2024-03-24T12:35:58.709644+0000","last_unstale":"2024-03-24T12:42:50.465873+0000","last_undegraded":"2024-03-24T12:35:58.700232+0000","last_fullsized":"2024-03-24T12:35:58.699170+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'18","last_scrub_stamp":"2024-03-24T12:33:02.509218+0000","last_deep_scrub":"84'18","last_deep_scrub_stamp":"2024-03-24T12:31:08.237387+0000","last_clean_scrub_stamp":"2024-03-24T12:33:02.509218+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:51.617201+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[{"shards":"0,1","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":549,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.354236+0000","last_change":"2024-03-24T12:35:58.703902+0000","last_active":"2024-03-24T12:42:50.354236+0000","last_peered":"2024-03-24T12:42:50.354236+0000","last_clean":"2024-03-24T12:35:54.680479+0000","last_became_active":"2024-03-24T12:35:58.703902+0000","last_became_peered":"2024-03-24T12:35:58.703902+0000","last_unstale":"2024-03-24T12:42:50.354236+0000","last_undegraded":"2024-03-24T12:42:50.354236+0000","last_fullsized":"2024-03-24T12:35:58.701457+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:13.031712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:13.031712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:52.013054+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"319'8717","reported_seq":8892,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:52.953201+0000","last_change":"2024-03-24T12:42:42.854623+0000","last_active":"2024-03-24T12:42:52.953201+0000","last_peered":"2024-03-24T12:42:52.953201+0000","last_clean":"2024-03-24T12:42:52.953201+0000","last_became_active":"2024-03-24T12:40:51.136705+0000","last_became_peered":"2024-03-24T12:40:51.136705+0000","last_unstale":"2024-03-24T12:42:52.953201+0000","last_undegraded":"2024-03-24T12:42:52.953201+0000","last_fullsized":"2024-03-24T12:42:52.953201+0000","mapping_epoch":278,"log_start":"286'3363","ondisk_log_start":"286'3363","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'8407","last_scrub_stamp":"2024-03-24T12:42:42.854286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:42:42.854286+0000","objects_scrubbed":7451,"log_size":5354,"log_dups_size":0,"ondisk_log_size":5354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:06.512638+0000","scrub_duration":1927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":465764352,"num_objects":7107,"num_object_clones":0,"num_object_copies":21321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8717,"num_write_kb":506368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":240,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.357222+0000","last_change":"2024-03-24T12:35:58.713785+0000","last_active":"2024-03-24T12:42:50.357222+0000","last_peered":"2024-03-24T12:42:50.357222+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.713785+0000","last_became_peered":"2024-03-24T12:35:58.713785+0000","last_unstale":"2024-03-24T12:42:50.357222+0000","last_undegraded":"2024-03-24T12:42:50.357222+0000","last_fullsized":"2024-03-24T12:35:58.700902+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:18.057286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:35:18.057286+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":872,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264276+0000","last_change":"2024-03-24T12:42:06.681235+0000","last_active":"2024-03-24T12:42:50.264276+0000","last_peered":"2024-03-24T12:42:50.264276+0000","last_clean":"2024-03-24T12:42:50.264276+0000","last_became_active":"2024-03-24T12:41:47.922762+0000","last_became_peered":"2024-03-24T12:41:47.922762+0000","last_unstale":"2024-03-24T12:42:50.264276+0000","last_undegraded":"2024-03-24T12:42:50.264276+0000","last_fullsized":"2024-03-24T12:42:50.264276+0000","mapping_epoch":257,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:06.681135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:06.681135+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:26.668569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":568,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.349175+0000","last_change":"2024-03-24T12:39:53.978440+0000","last_active":"2024-03-24T12:42:50.349175+0000","last_peered":"2024-03-24T12:42:50.349175+0000","last_clean":"2024-03-24T12:42:50.349175+0000","last_became_active":"2024-03-24T12:39:35.128182+0000","last_became_peered":"2024-03-24T12:39:35.128182+0000","last_unstale":"2024-03-24T12:42:50.349175+0000","last_undegraded":"2024-03-24T12:42:50.349175+0000","last_fullsized":"2024-03-24T12:42:50.349175+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:53.978235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:39:53.978235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"258'30","reported_seq":213,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.315964+0000","last_change":"2024-03-24T12:42:32.914865+0000","last_active":"2024-03-24T12:42:50.315964+0000","last_peered":"2024-03-24T12:42:50.315964+0000","last_clean":"2024-03-24T12:42:50.315964+0000","last_became_active":"2024-03-24T12:39:34.753078+0000","last_became_peered":"2024-03-24T12:39:34.753078+0000","last_unstale":"2024-03-24T12:42:50.315964+0000","last_undegraded":"2024-03-24T12:42:50.315964+0000","last_fullsized":"2024-03-24T12:42:50.315964+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'30","last_scrub_stamp":"2024-03-24T12:42:32.914720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:32.914720+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:58.103591+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":542,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.303955+0000","last_change":"2024-03-24T12:35:58.711216+0000","last_active":"2024-03-24T12:42:50.303955+0000","last_peered":"2024-03-24T12:42:50.303955+0000","last_clean":"2024-03-24T12:35:54.688099+0000","last_became_active":"2024-03-24T12:35:58.711216+0000","last_became_peered":"2024-03-24T12:35:58.711216+0000","last_unstale":"2024-03-24T12:42:50.303955+0000","last_undegraded":"2024-03-24T12:42:50.303955+0000","last_fullsized":"2024-03-24T12:35:58.699124+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:47.802609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:47.802609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":["7","2"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":768,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.265706+0000","last_change":"2024-03-24T12:42:03.752923+0000","last_active":"2024-03-24T12:42:50.265706+0000","last_peered":"2024-03-24T12:42:50.265706+0000","last_clean":"2024-03-24T12:42:50.265706+0000","last_became_active":"2024-03-24T12:41:52.947112+0000","last_became_peered":"2024-03-24T12:41:52.947112+0000","last_unstale":"2024-03-24T12:42:50.265706+0000","last_undegraded":"2024-03-24T12:42:50.265706+0000","last_fullsized":"2024-03-24T12:42:50.265706+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:03.752830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:03.752830+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:16.384658+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.8","version":"319'8653","reported_seq":8745,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:51.680198+0000","last_change":"2024-03-24T12:40:51.136635+0000","last_active":"2024-03-24T12:42:51.680198+0000","last_peered":"2024-03-24T12:42:51.680198+0000","last_clean":"2024-03-24T12:42:51.680198+0000","last_became_active":"2024-03-24T12:40:51.136300+0000","last_became_peered":"2024-03-24T12:40:51.136300+0000","last_unstale":"2024-03-24T12:42:51.680198+0000","last_undegraded":"2024-03-24T12:42:51.680198+0000","last_fullsized":"2024-03-24T12:42:51.680198+0000","mapping_epoch":278,"log_start":"288'4145","ondisk_log_start":"288'4145","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":466550784,"num_objects":7119,"num_object_clones":0,"num_object_copies":21357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7119,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8653,"num_write_kb":504704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":554,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.356777+0000","last_change":"2024-03-24T12:35:58.717519+0000","last_active":"2024-03-24T12:42:50.356777+0000","last_peered":"2024-03-24T12:42:50.356777+0000","last_clean":"2024-03-24T12:35:54.687071+0000","last_became_active":"2024-03-24T12:35:58.717519+0000","last_became_peered":"2024-03-24T12:35:58.717519+0000","last_unstale":"2024-03-24T12:42:50.356777+0000","last_undegraded":"2024-03-24T12:42:50.356777+0000","last_fullsized":"2024-03-24T12:35:58.696848+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:59.974279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:34:59.974279+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":579,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.351519+0000","last_change":"2024-03-24T12:42:12.561279+0000","last_active":"2024-03-24T12:42:50.351519+0000","last_peered":"2024-03-24T12:42:50.351519+0000","last_clean":"2024-03-24T12:42:50.351519+0000","last_became_active":"2024-03-24T12:26:03.167233+0000","last_became_peered":"2024-03-24T12:26:03.167233+0000","last_unstale":"2024-03-24T12:42:50.351519+0000","last_undegraded":"2024-03-24T12:42:50.351519+0000","last_fullsized":"2024-03-24T12:42:50.351519+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:12.561140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:13.629968+0000","last_clean_scrub_stamp":"2024-03-24T12:42:12.561140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:37.809732+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1599,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:51.666209+0000","last_change":"2024-03-24T12:42:44.455954+0000","last_active":"2024-03-24T12:42:51.666209+0000","last_peered":"2024-03-24T12:42:51.666209+0000","last_clean":"2024-03-24T12:42:51.666209+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:42:51.666209+0000","last_undegraded":"2024-03-24T12:42:51.666209+0000","last_fullsized":"2024-03-24T12:42:51.666209+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:42:44.455813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:44.455813+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:07.032362+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1002,"num_read_kb":997,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"260'30","reported_seq":609,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.467458+0000","last_change":"2024-03-24T12:42:05.550426+0000","last_active":"2024-03-24T12:42:50.467458+0000","last_peered":"2024-03-24T12:42:50.467458+0000","last_clean":"2024-03-24T12:42:50.467458+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:42:50.467458+0000","last_undegraded":"2024-03-24T12:42:50.467458+0000","last_fullsized":"2024-03-24T12:42:50.467458+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'30","last_scrub_stamp":"2024-03-24T12:42:05.550286+0000","last_deep_scrub":"260'30","last_deep_scrub_stamp":"2024-03-24T12:39:51.288276+0000","last_clean_scrub_stamp":"2024-03-24T12:42:05.550286+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:27.879671+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":241,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.466238+0000","last_change":"2024-03-24T12:35:58.708969+0000","last_active":"2024-03-24T12:42:50.466238+0000","last_peered":"2024-03-24T12:42:50.466238+0000","last_clean":"2024-03-24T12:26:06.312270+0000","last_became_active":"2024-03-24T12:35:58.708969+0000","last_became_peered":"2024-03-24T12:35:58.708969+0000","last_unstale":"2024-03-24T12:42:50.466238+0000","last_undegraded":"2024-03-24T12:42:50.466238+0000","last_fullsized":"2024-03-24T12:35:58.699657+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:52.136918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:34:52.136918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:51.617266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":768,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.268197+0000","last_change":"2024-03-24T12:42:04.760447+0000","last_active":"2024-03-24T12:42:50.268197+0000","last_peered":"2024-03-24T12:42:50.268197+0000","last_clean":"2024-03-24T12:42:50.268197+0000","last_became_active":"2024-03-24T12:41:52.960995+0000","last_became_peered":"2024-03-24T12:41:52.960995+0000","last_unstale":"2024-03-24T12:42:50.268197+0000","last_undegraded":"2024-03-24T12:42:50.268197+0000","last_fullsized":"2024-03-24T12:42:50.268197+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:04.760318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:04.760318+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:14.065490+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.d","version":"319'10984","reported_seq":11190,"reported_epoch":319,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T12:42:53.055892+0000","last_change":"2024-03-24T12:42:39.872027+0000","last_active":"2024-03-24T12:42:53.055892+0000","last_peered":"2024-03-24T12:42:53.055892+0000","last_clean":"2024-03-24T12:42:53.055892+0000","last_became_active":"2024-03-24T12:40:51.138950+0000","last_became_peered":"2024-03-24T12:40:51.138950+0000","last_unstale":"2024-03-24T12:42:53.055892+0000","last_undegraded":"2024-03-24T12:42:53.055892+0000","last_fullsized":"2024-03-24T12:42:53.055892+0000","mapping_epoch":278,"log_start":"302'5639","ondisk_log_start":"302'5639","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":2685,"log_size":5345,"log_dups_size":0,"ondisk_log_size":5345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 13s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463208472,"num_objects":7069,"num_object_clones":0,"num_object_copies":21207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7069,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8737,"num_write_kb":505729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":587,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.356244+0000","last_change":"2024-03-24T12:42:12.561111+0000","last_active":"2024-03-24T12:42:50.356244+0000","last_peered":"2024-03-24T12:42:50.356244+0000","last_clean":"2024-03-24T12:42:50.356244+0000","last_became_active":"2024-03-24T12:39:34.751275+0000","last_became_peered":"2024-03-24T12:39:34.751275+0000","last_unstale":"2024-03-24T12:42:50.356244+0000","last_undegraded":"2024-03-24T12:42:50.356244+0000","last_fullsized":"2024-03-24T12:42:50.356244+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:12.560906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:42:12.560906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:37.386521+0000","scrub_duration":71,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":472,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.349030+0000","last_change":"2024-03-24T12:39:07.747764+0000","last_active":"2024-03-24T12:42:50.349030+0000","last_peered":"2024-03-24T12:42:50.349030+0000","last_clean":"2024-03-24T12:35:54.687290+0000","last_became_active":"2024-03-24T12:39:07.747764+0000","last_became_peered":"2024-03-24T12:39:07.747764+0000","last_unstale":"2024-03-24T12:42:50.349030+0000","last_undegraded":"2024-03-24T12:42:50.349030+0000","last_fullsized":"2024-03-24T12:39:07.743810+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:34:54.953570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:34:54.953570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:52.013086+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.3","version":"207'37","reported_seq":605,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:50.467844+0000","last_change":"2024-03-24T12:35:58.711927+0000","last_active":"2024-03-24T12:42:50.467844+0000","last_peered":"2024-03-24T12:42:50.467844+0000","last_clean":"2024-03-24T12:35:54.682985+0000","last_became_active":"2024-03-24T12:35:58.711927+0000","last_became_peered":"2024-03-24T12:35:58.711927+0000","last_unstale":"2024-03-24T12:42:50.467844+0000","last_undegraded":"2024-03-24T12:35:58.703907+0000","last_fullsized":"2024-03-24T12:35:58.703783+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'27","last_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_deep_scrub":"195'27","last_deep_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_clean_scrub_stamp":"2024-03-24T12:35:16.981148+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":["0","7"],"object_location_counts":[{"shards":"0,7","objects":8}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":525,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.267909+0000","last_change":"2024-03-24T12:42:31.883699+0000","last_active":"2024-03-24T12:42:50.267909+0000","last_peered":"2024-03-24T12:42:50.267909+0000","last_clean":"2024-03-24T12:42:50.267909+0000","last_became_active":"2024-03-24T12:39:35.099711+0000","last_became_peered":"2024-03-24T12:39:35.099711+0000","last_unstale":"2024-03-24T12:42:50.267909+0000","last_undegraded":"2024-03-24T12:42:50.267909+0000","last_fullsized":"2024-03-24T12:42:50.267909+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:31.883568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:42:31.883568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:55.659971+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0,2],"avail_no_missing":["6","0","2"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":768,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.266159+0000","last_change":"2024-03-24T12:42:05.783856+0000","last_active":"2024-03-24T12:42:50.266159+0000","last_peered":"2024-03-24T12:42:50.266159+0000","last_clean":"2024-03-24T12:42:50.266159+0000","last_became_active":"2024-03-24T12:41:52.961504+0000","last_became_peered":"2024-03-24T12:41:52.961504+0000","last_unstale":"2024-03-24T12:42:50.266159+0000","last_undegraded":"2024-03-24T12:42:50.266159+0000","last_fullsized":"2024-03-24T12:42:50.266159+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:05.783609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:05.783609+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:06.667204+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.c","version":"319'10650","reported_seq":10834,"reported_epoch":319,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T12:42:52.688983+0000","last_change":"2024-03-24T12:42:39.433861+0000","last_active":"2024-03-24T12:42:52.688983+0000","last_peered":"2024-03-24T12:42:52.688983+0000","last_clean":"2024-03-24T12:42:52.688983+0000","last_became_active":"2024-03-24T12:40:51.138108+0000","last_became_peered":"2024-03-24T12:40:51.138108+0000","last_unstale":"2024-03-24T12:42:52.688983+0000","last_undegraded":"2024-03-24T12:42:52.688983+0000","last_fullsized":"2024-03-24T12:42:52.688983+0000","mapping_epoch":278,"log_start":"299'5138","ondisk_log_start":"299'5138","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":2198,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 13s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467468288,"num_objects":7133,"num_object_clones":0,"num_object_copies":21399,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7133,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8845,"num_write_kb":511296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.351823+0000","last_change":"2024-03-24T12:42:06.595195+0000","last_active":"2024-03-24T12:42:50.351823+0000","last_peered":"2024-03-24T12:42:50.351823+0000","last_clean":"2024-03-24T12:42:50.351823+0000","last_became_active":"2024-03-24T12:26:09.340875+0000","last_became_peered":"2024-03-24T12:26:09.340875+0000","last_unstale":"2024-03-24T12:42:50.351823+0000","last_undegraded":"2024-03-24T12:42:50.351823+0000","last_fullsized":"2024-03-24T12:42:50.351823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:06.595072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:42:06.595072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:18.887564+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":240,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.352939+0000","last_change":"2024-03-24T12:35:58.711061+0000","last_active":"2024-03-24T12:42:50.352939+0000","last_peered":"2024-03-24T12:42:50.352939+0000","last_clean":"2024-03-24T12:26:02.143860+0000","last_became_active":"2024-03-24T12:35:58.711061+0000","last_became_peered":"2024-03-24T12:35:58.711061+0000","last_unstale":"2024-03-24T12:42:50.352939+0000","last_undegraded":"2024-03-24T12:42:50.352939+0000","last_fullsized":"2024-03-24T12:35:58.704709+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:35:00.924106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.924106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.2","version":"207'25","reported_seq":589,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:50.356618+0000","last_change":"2024-03-24T12:35:58.704200+0000","last_active":"2024-03-24T12:42:50.356618+0000","last_peered":"2024-03-24T12:42:50.356618+0000","last_clean":"2024-03-24T12:35:54.687582+0000","last_became_active":"2024-03-24T12:35:58.704200+0000","last_became_peered":"2024-03-24T12:35:58.704200+0000","last_unstale":"2024-03-24T12:42:50.356618+0000","last_undegraded":"2024-03-24T12:35:58.697842+0000","last_fullsized":"2024-03-24T12:35:58.697725+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"138'15","last_scrub_stamp":"2024-03-24T12:33:07.518368+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:33:07.518368+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:51.617182+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"318'14","reported_seq":793,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.467320+0000","last_change":"2024-03-24T12:42:45.094308+0000","last_active":"2024-03-24T12:42:50.467320+0000","last_peered":"2024-03-24T12:42:50.467320+0000","last_clean":"2024-03-24T12:42:50.467320+0000","last_became_active":"2024-03-24T12:39:35.135410+0000","last_became_peered":"2024-03-24T12:39:35.135410+0000","last_unstale":"2024-03-24T12:42:50.467320+0000","last_undegraded":"2024-03-24T12:42:50.467320+0000","last_fullsized":"2024-03-24T12:42:50.467320+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'14","last_scrub_stamp":"2024-03-24T12:42:45.093908+0000","last_deep_scrub":"318'14","last_deep_scrub_stamp":"2024-03-24T12:42:45.093908+0000","last_clean_scrub_stamp":"2024-03-24T12:42:45.093908+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:54.548481+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,3],"avail_no_missing":["0","3","6"],"object_location_counts":[{"shards":"0,3,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":357,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.352027+0000","last_change":"2024-03-24T12:42:04.608057+0000","last_active":"2024-03-24T12:42:50.352027+0000","last_peered":"2024-03-24T12:42:50.352027+0000","last_clean":"2024-03-24T12:42:50.352027+0000","last_became_active":"2024-03-24T12:41:52.954632+0000","last_became_peered":"2024-03-24T12:41:52.954632+0000","last_unstale":"2024-03-24T12:42:50.352027+0000","last_undegraded":"2024-03-24T12:42:50.352027+0000","last_fullsized":"2024-03-24T12:42:50.352027+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:04.607795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:04.607795+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:24.869000+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,0],"avail_no_missing":["3","0","7"],"object_location_counts":[{"shards":"0,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"319'11593","reported_seq":11823,"reported_epoch":319,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-24T12:42:53.066545+0000","last_change":"2024-03-24T12:42:36.882134+0000","last_active":"2024-03-24T12:42:53.066545+0000","last_peered":"2024-03-24T12:42:53.066545+0000","last_clean":"2024-03-24T12:42:53.066545+0000","last_became_active":"2024-03-24T12:40:51.398299+0000","last_became_peered":"2024-03-24T12:40:51.398299+0000","last_unstale":"2024-03-24T12:42:53.066545+0000","last_undegraded":"2024-03-24T12:42:53.066545+0000","last_fullsized":"2024-03-24T12:42:53.066545+0000","mapping_epoch":278,"log_start":"305'6303","ondisk_log_start":"305'6303","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":3524,"log_size":5290,"log_dups_size":0,"ondisk_log_size":5290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 16s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":467140608,"num_objects":7128,"num_object_clones":0,"num_object_copies":21384,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7128,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8684,"num_write_kb":505984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":571,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264698+0000","last_change":"2024-03-24T12:40:59.957298+0000","last_active":"2024-03-24T12:42:50.264698+0000","last_peered":"2024-03-24T12:42:50.264698+0000","last_clean":"2024-03-24T12:42:50.264698+0000","last_became_active":"2024-03-24T12:39:34.756439+0000","last_became_peered":"2024-03-24T12:39:34.756439+0000","last_unstale":"2024-03-24T12:42:50.264698+0000","last_undegraded":"2024-03-24T12:42:50.264698+0000","last_fullsized":"2024-03-24T12:42:50.264698+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:59.957191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:40:59.957191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":579,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.467547+0000","last_change":"2024-03-24T12:42:38.018922+0000","last_active":"2024-03-24T12:42:50.467547+0000","last_peered":"2024-03-24T12:42:50.467547+0000","last_clean":"2024-03-24T12:42:50.467547+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:42:50.467547+0000","last_undegraded":"2024-03-24T12:42:50.467547+0000","last_fullsized":"2024-03-24T12:42:50.467547+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:42:38.018786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:38.018786+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:53.920476+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"207'35","reported_seq":645,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.261886+0000","last_change":"2024-03-24T12:42:23.775222+0000","last_active":"2024-03-24T12:42:50.261886+0000","last_peered":"2024-03-24T12:42:50.261886+0000","last_clean":"2024-03-24T12:42:50.261886+0000","last_became_active":"2024-03-24T12:39:35.103040+0000","last_became_peered":"2024-03-24T12:39:35.103040+0000","last_unstale":"2024-03-24T12:42:50.261886+0000","last_undegraded":"2024-03-24T12:42:50.261886+0000","last_fullsized":"2024-03-24T12:42:50.261886+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-24T12:42:23.774969+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:42:23.774969+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:32.086477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,0],"avail_no_missing":["4","0","6"],"object_location_counts":[{"shards":"0,4,6","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":765,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349497+0000","last_change":"2024-03-24T12:40:16.189646+0000","last_active":"2024-03-24T12:42:50.349497+0000","last_peered":"2024-03-24T12:42:50.349497+0000","last_clean":"2024-03-24T12:42:50.349497+0000","last_became_active":"2024-03-24T12:26:07.335165+0000","last_became_peered":"2024-03-24T12:26:07.335165+0000","last_unstale":"2024-03-24T12:42:50.349497+0000","last_undegraded":"2024-03-24T12:42:50.349497+0000","last_fullsized":"2024-03-24T12:42:50.349497+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:40:16.189517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:40:16.189517+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":357,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.261349+0000","last_change":"2024-03-24T12:42:04.733820+0000","last_active":"2024-03-24T12:42:50.261349+0000","last_peered":"2024-03-24T12:42:50.261349+0000","last_clean":"2024-03-24T12:42:50.261349+0000","last_became_active":"2024-03-24T12:41:52.948500+0000","last_became_peered":"2024-03-24T12:41:52.948500+0000","last_unstale":"2024-03-24T12:42:50.261349+0000","last_undegraded":"2024-03-24T12:42:50.261349+0000","last_fullsized":"2024-03-24T12:42:50.261349+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:04.733707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:04.733707+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:15.147234+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.e","version":"319'8773","reported_seq":8946,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:52.541163+0000","last_change":"2024-03-24T12:42:47.598241+0000","last_active":"2024-03-24T12:42:52.541163+0000","last_peered":"2024-03-24T12:42:52.541163+0000","last_clean":"2024-03-24T12:42:52.541163+0000","last_became_active":"2024-03-24T12:40:51.137734+0000","last_became_peered":"2024-03-24T12:40:51.137734+0000","last_unstale":"2024-03-24T12:42:52.541163+0000","last_undegraded":"2024-03-24T12:42:52.541163+0000","last_fullsized":"2024-03-24T12:42:52.541163+0000","mapping_epoch":278,"log_start":"292'4587","ondisk_log_start":"292'4587","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'8630","last_scrub_stamp":"2024-03-24T12:42:47.598074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:42:47.598074+0000","objects_scrubbed":7347,"log_size":4186,"log_dups_size":0,"ondisk_log_size":4186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:58.606940+0000","scrub_duration":1639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470089728,"num_objects":7173,"num_object_clones":0,"num_object_copies":21519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7173,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8773,"num_write_kb":510272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":559,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.260943+0000","last_change":"2024-03-24T12:41:04.979281+0000","last_active":"2024-03-24T12:42:50.260943+0000","last_peered":"2024-03-24T12:42:50.260943+0000","last_clean":"2024-03-24T12:42:50.260943+0000","last_became_active":"2024-03-24T12:26:09.352577+0000","last_became_peered":"2024-03-24T12:26:09.352577+0000","last_unstale":"2024-03-24T12:42:50.260943+0000","last_undegraded":"2024-03-24T12:42:50.260943+0000","last_fullsized":"2024-03-24T12:42:50.260943+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:41:04.979163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:41:04.979163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":554,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.357472+0000","last_change":"2024-03-24T12:37:30.870003+0000","last_active":"2024-03-24T12:42:50.357472+0000","last_peered":"2024-03-24T12:42:50.357472+0000","last_clean":"2024-03-24T12:42:50.357472+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:42:50.357472+0000","last_undegraded":"2024-03-24T12:42:50.357472+0000","last_fullsized":"2024-03-24T12:42:50.357472+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:30.869794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:30.869794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"207'30","reported_seq":641,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349414+0000","last_change":"2024-03-24T12:42:37.694203+0000","last_active":"2024-03-24T12:42:50.349414+0000","last_peered":"2024-03-24T12:42:50.349414+0000","last_clean":"2024-03-24T12:42:50.349414+0000","last_became_active":"2024-03-24T12:42:37.188834+0000","last_became_peered":"2024-03-24T12:42:37.188834+0000","last_unstale":"2024-03-24T12:42:50.349414+0000","last_undegraded":"2024-03-24T12:42:50.349414+0000","last_fullsized":"2024-03-24T12:42:50.349414+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'30","last_scrub_stamp":"2024-03-24T12:42:37.694052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:37.694052+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:56.139296+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":577,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.261654+0000","last_change":"2024-03-24T12:42:37.184483+0000","last_active":"2024-03-24T12:42:50.261654+0000","last_peered":"2024-03-24T12:42:50.261654+0000","last_clean":"2024-03-24T12:42:50.261654+0000","last_became_active":"2024-03-24T12:42:37.184112+0000","last_became_peered":"2024-03-24T12:42:37.184112+0000","last_unstale":"2024-03-24T12:42:50.261654+0000","last_undegraded":"2024-03-24T12:42:50.261654+0000","last_fullsized":"2024-03-24T12:42:50.261654+0000","mapping_epoch":313,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":314,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:57.941616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:40:57.941616+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":356,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.348859+0000","last_change":"2024-03-24T12:42:03.892872+0000","last_active":"2024-03-24T12:42:50.348859+0000","last_peered":"2024-03-24T12:42:50.348859+0000","last_clean":"2024-03-24T12:42:50.348859+0000","last_became_active":"2024-03-24T12:41:52.957006+0000","last_became_peered":"2024-03-24T12:41:52.957006+0000","last_unstale":"2024-03-24T12:42:50.348859+0000","last_undegraded":"2024-03-24T12:42:50.348859+0000","last_fullsized":"2024-03-24T12:42:50.348859+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:03.892619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:03.892619+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:25.016061+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6,0],"avail_no_missing":["2","0","6"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1","version":"319'8754","reported_seq":8847,"reported_epoch":319,"state":"active+clean+scrubbing","last_fresh":"2024-03-24T12:42:52.496207+0000","last_change":"2024-03-24T12:42:48.889327+0000","last_active":"2024-03-24T12:42:52.496207+0000","last_peered":"2024-03-24T12:42:52.496207+0000","last_clean":"2024-03-24T12:42:52.496207+0000","last_became_active":"2024-03-24T12:40:51.135277+0000","last_became_peered":"2024-03-24T12:40:51.135277+0000","last_unstale":"2024-03-24T12:42:52.496207+0000","last_undegraded":"2024-03-24T12:42:52.496207+0000","last_fullsized":"2024-03-24T12:42:52.496207+0000","mapping_epoch":278,"log_start":"292'4511","ondisk_log_start":"292'4511","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":4243,"log_dups_size":0,"ondisk_log_size":4243,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":460324864,"num_objects":7024,"num_object_clones":0,"num_object_copies":21072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8754,"num_write_kb":504896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":240,"reported_epoch":319,"state":"active+undersized","last_fresh":"2024-03-24T12:42:50.465379+0000","last_change":"2024-03-24T12:35:58.711067+0000","last_active":"2024-03-24T12:42:50.465379+0000","last_peered":"2024-03-24T12:42:50.465379+0000","last_clean":"2024-03-24T12:26:08.325526+0000","last_became_active":"2024-03-24T12:35:58.711067+0000","last_became_peered":"2024-03-24T12:35:58.711067+0000","last_unstale":"2024-03-24T12:42:50.465379+0000","last_undegraded":"2024-03-24T12:42:50.465379+0000","last_fullsized":"2024-03-24T12:35:58.696416+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:33:36.210440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:33:36.210440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:42:51.617239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":["1","0"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"205'10","reported_seq":761,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:50.267158+0000","last_change":"2024-03-24T12:35:58.712862+0000","last_active":"2024-03-24T12:42:50.267158+0000","last_peered":"2024-03-24T12:42:50.267158+0000","last_clean":"2024-03-24T12:35:54.678342+0000","last_became_active":"2024-03-24T12:35:58.712862+0000","last_became_peered":"2024-03-24T12:35:58.712862+0000","last_unstale":"2024-03-24T12:42:50.267158+0000","last_undegraded":"2024-03-24T12:35:58.700712+0000","last_fullsized":"2024-03-24T12:35:58.700587+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"140'9","last_scrub_stamp":"2024-03-24T12:34:44.703379+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:34:44.703379+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":["4","0"],"object_location_counts":[{"shards":"0,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":562,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.349315+0000","last_change":"2024-03-24T12:42:42.772367+0000","last_active":"2024-03-24T12:42:50.349315+0000","last_peered":"2024-03-24T12:42:50.349315+0000","last_clean":"2024-03-24T12:42:50.349315+0000","last_became_active":"2024-03-24T12:26:03.163330+0000","last_became_peered":"2024-03-24T12:26:03.163330+0000","last_unstale":"2024-03-24T12:42:50.349315+0000","last_undegraded":"2024-03-24T12:42:50.349315+0000","last_fullsized":"2024-03-24T12:42:50.349315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:42.772243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:42.772243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:02.336742+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"207'34","reported_seq":607,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.371823+0000","last_change":"2024-03-24T12:42:02.926934+0000","last_active":"2024-03-24T12:42:50.371823+0000","last_peered":"2024-03-24T12:42:50.371823+0000","last_clean":"2024-03-24T12:42:50.371823+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:42:50.371823+0000","last_undegraded":"2024-03-24T12:42:50.371823+0000","last_fullsized":"2024-03-24T12:42:50.371823+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-24T12:42:02.926772+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:42:02.926772+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:22.634774+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.0","version":"319'8885","reported_seq":9065,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:53.048157+0000","last_change":"2024-03-24T12:42:39.540895+0000","last_active":"2024-03-24T12:42:53.048157+0000","last_peered":"2024-03-24T12:42:53.048157+0000","last_clean":"2024-03-24T12:42:53.048157+0000","last_became_active":"2024-03-24T12:40:51.138083+0000","last_became_peered":"2024-03-24T12:40:51.138083+0000","last_unstale":"2024-03-24T12:42:53.048157+0000","last_undegraded":"2024-03-24T12:42:53.048157+0000","last_fullsized":"2024-03-24T12:42:53.048157+0000","mapping_epoch":278,"log_start":"286'3543","ondisk_log_start":"286'3543","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"316'8449","last_scrub_stamp":"2024-03-24T12:42:39.540795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:42:39.540795+0000","objects_scrubbed":7829,"log_size":5342,"log_dups_size":0,"ondisk_log_size":5342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:08.023341+0000","scrub_duration":1646,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":480706560,"num_objects":7335,"num_object_clones":0,"num_object_copies":22005,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7335,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8885,"num_write_kb":519040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":547,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.352325+0000","last_change":"2024-03-24T12:39:55.001557+0000","last_active":"2024-03-24T12:42:50.352325+0000","last_peered":"2024-03-24T12:42:50.352325+0000","last_clean":"2024-03-24T12:42:50.352325+0000","last_became_active":"2024-03-24T12:39:35.129779+0000","last_became_peered":"2024-03-24T12:39:35.129779+0000","last_unstale":"2024-03-24T12:42:50.352325+0000","last_undegraded":"2024-03-24T12:42:50.352325+0000","last_fullsized":"2024-03-24T12:42:50.352325+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:39:55.000906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:39:55.000906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,7],"avail_no_missing":["3","6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":267,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.262520+0000","last_change":"2024-03-24T12:42:19.697781+0000","last_active":"2024-03-24T12:42:50.262520+0000","last_peered":"2024-03-24T12:42:50.262520+0000","last_clean":"2024-03-24T12:42:50.262520+0000","last_became_active":"2024-03-24T12:39:35.106802+0000","last_became_peered":"2024-03-24T12:39:35.106802+0000","last_unstale":"2024-03-24T12:42:50.262520+0000","last_undegraded":"2024-03-24T12:42:50.262520+0000","last_fullsized":"2024-03-24T12:42:50.262520+0000","mapping_epoch":255,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":256,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:19.697555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:19.697555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:34.392572+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6,2],"avail_no_missing":["4","2","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":570,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.261737+0000","last_change":"2024-03-24T12:40:55.957992+0000","last_active":"2024-03-24T12:42:50.261737+0000","last_peered":"2024-03-24T12:42:50.261737+0000","last_clean":"2024-03-24T12:42:50.261737+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:42:50.261737+0000","last_undegraded":"2024-03-24T12:42:50.261737+0000","last_fullsized":"2024-03-24T12:42:50.261737+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:55.957883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:40:55.957883+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"207'19","reported_seq":641,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.266891+0000","last_change":"2024-03-24T12:42:40.867242+0000","last_active":"2024-03-24T12:42:50.266891+0000","last_peered":"2024-03-24T12:42:50.266891+0000","last_clean":"2024-03-24T12:42:50.266891+0000","last_became_active":"2024-03-24T12:39:34.752219+0000","last_became_peered":"2024-03-24T12:39:34.752219+0000","last_unstale":"2024-03-24T12:42:50.266891+0000","last_undegraded":"2024-03-24T12:42:50.266891+0000","last_fullsized":"2024-03-24T12:42:50.266891+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:42:40.867095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:42:40.867095+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:50.783112+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":355,"reported_epoch":319,"state":"active+clean+remapped","last_fresh":"2024-03-24T12:42:50.354441+0000","last_change":"2024-03-24T12:42:04.891272+0000","last_active":"2024-03-24T12:42:50.354441+0000","last_peered":"2024-03-24T12:42:50.354441+0000","last_clean":"2024-03-24T12:42:50.354441+0000","last_became_active":"2024-03-24T12:41:52.965224+0000","last_became_peered":"2024-03-24T12:41:52.965224+0000","last_unstale":"2024-03-24T12:42:50.354441+0000","last_undegraded":"2024-03-24T12:42:50.354441+0000","last_fullsized":"2024-03-24T12:42:50.354441+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:04.891014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:04.891014+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:12.813106+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3,0],"avail_no_missing":["2","0","3"],"object_location_counts":[{"shards":"0,2,3","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"319'8746","reported_seq":8835,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:54.054062+0000","last_change":"2024-03-24T12:41:04.286472+0000","last_active":"2024-03-24T12:42:54.054062+0000","last_peered":"2024-03-24T12:42:54.054062+0000","last_clean":"2024-03-24T12:40:50.114364+0000","last_became_active":"2024-03-24T12:40:51.397675+0000","last_became_peered":"2024-03-24T12:40:51.397675+0000","last_unstale":"2024-03-24T12:42:54.054062+0000","last_undegraded":"2024-03-24T12:41:04.282306+0000","last_fullsized":"2024-03-24T12:40:51.132010+0000","mapping_epoch":278,"log_start":"286'3326","ondisk_log_start":"286'3326","created":278,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:40:50.114364+0000","objects_scrubbed":0,"log_size":5420,"log_dups_size":0,"ondisk_log_size":5420,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459538432,"num_objects":7012,"num_object_clones":0,"num_object_copies":21036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7012,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8746,"num_write_kb":504256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":7012}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":574,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.371652+0000","last_change":"2024-03-24T12:42:17.213244+0000","last_active":"2024-03-24T12:42:50.371652+0000","last_peered":"2024-03-24T12:42:50.371652+0000","last_clean":"2024-03-24T12:42:50.371652+0000","last_became_active":"2024-03-24T12:26:09.347716+0000","last_became_peered":"2024-03-24T12:26:09.347716+0000","last_unstale":"2024-03-24T12:42:50.371652+0000","last_undegraded":"2024-03-24T12:42:50.371652+0000","last_fullsized":"2024-03-24T12:42:50.371652+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:17.213137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:53.627010+0000","last_clean_scrub_stamp":"2024-03-24T12:42:17.213137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:43.268369+0000","scrub_duration":227,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":171,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.260976+0000","last_change":"2024-03-24T12:42:39.834005+0000","last_active":"2024-03-24T12:42:50.260976+0000","last_peered":"2024-03-24T12:42:50.260976+0000","last_clean":"2024-03-24T12:42:50.260976+0000","last_became_active":"2024-03-24T12:39:07.753551+0000","last_became_peered":"2024-03-24T12:39:07.753551+0000","last_unstale":"2024-03-24T12:42:50.260976+0000","last_undegraded":"2024-03-24T12:42:50.260976+0000","last_fullsized":"2024-03-24T12:42:50.260976+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:39.833885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:42:39.833885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:53.030633+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":156,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.265630+0000","last_change":"2024-03-24T12:42:28.910820+0000","last_active":"2024-03-24T12:42:50.265630+0000","last_peered":"2024-03-24T12:42:50.265630+0000","last_clean":"2024-03-24T12:42:50.265630+0000","last_became_active":"2024-03-24T12:39:34.745353+0000","last_became_peered":"2024-03-24T12:39:34.745353+0000","last_unstale":"2024-03-24T12:42:50.265630+0000","last_undegraded":"2024-03-24T12:42:50.265630+0000","last_fullsized":"2024-03-24T12:42:50.265630+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:28.910598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:42:28.910598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:38.885881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"207'39","reported_seq":631,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.262635+0000","last_change":"2024-03-24T12:42:24.777315+0000","last_active":"2024-03-24T12:42:50.262635+0000","last_peered":"2024-03-24T12:42:50.262635+0000","last_clean":"2024-03-24T12:42:50.262635+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:42:50.262635+0000","last_undegraded":"2024-03-24T12:42:50.262635+0000","last_fullsized":"2024-03-24T12:42:50.262635+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'39","last_scrub_stamp":"2024-03-24T12:42:24.777172+0000","last_deep_scrub":"207'39","last_deep_scrub_stamp":"2024-03-24T12:36:22.669686+0000","last_clean_scrub_stamp":"2024-03-24T12:42:24.777172+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:28.260079+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":879,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.264015+0000","last_change":"2024-03-24T12:42:07.642011+0000","last_active":"2024-03-24T12:42:50.264015+0000","last_peered":"2024-03-24T12:42:50.264015+0000","last_clean":"2024-03-24T12:42:50.264015+0000","last_became_active":"2024-03-24T12:41:52.953388+0000","last_became_peered":"2024-03-24T12:41:52.953388+0000","last_unstale":"2024-03-24T12:42:50.264015+0000","last_undegraded":"2024-03-24T12:42:50.264015+0000","last_fullsized":"2024-03-24T12:42:50.264015+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:42:07.641876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:42:07.641876+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:11.042882+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.2","version":"319'8846","reported_seq":9018,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:54.781528+0000","last_change":"2024-03-24T12:42:24.452745+0000","last_active":"2024-03-24T12:42:54.781528+0000","last_peered":"2024-03-24T12:42:54.781528+0000","last_clean":"2024-03-24T12:42:54.781528+0000","last_became_active":"2024-03-24T12:40:51.130578+0000","last_became_peered":"2024-03-24T12:40:51.130578+0000","last_unstale":"2024-03-24T12:42:54.781528+0000","last_undegraded":"2024-03-24T12:42:54.781528+0000","last_fullsized":"2024-03-24T12:42:54.781528+0000","mapping_epoch":278,"log_start":"287'3993","ondisk_log_start":"287'3993","created":278,"last_epoch_clean":279,"parent":"0.0","parent_split_bits":0,"last_scrub":"308'7261","last_scrub_stamp":"2024-03-24T12:42:24.452542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:50.114364+0000","last_clean_scrub_stamp":"2024-03-24T12:42:24.452542+0000","objects_scrubbed":7182,"log_size":4853,"log_dups_size":0,"ondisk_log_size":4853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:43:42.743552+0000","scrub_duration":2899,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461504512,"num_objects":7042,"num_object_clones":0,"num_object_copies":21126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8846,"num_write_kb":508416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":574,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.267348+0000","last_change":"2024-03-24T12:42:45.013856+0000","last_active":"2024-03-24T12:42:50.267348+0000","last_peered":"2024-03-24T12:42:50.267348+0000","last_clean":"2024-03-24T12:42:50.267348+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:42:50.267348+0000","last_undegraded":"2024-03-24T12:42:50.267348+0000","last_fullsized":"2024-03-24T12:42:50.267348+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:42:45.013715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:42:45.013715+0000","last_clean_scrub_stamp":"2024-03-24T12:42:45.013715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:44:02.389554+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"205'9","reported_seq":761,"reported_epoch":319,"state":"active+undersized+degraded","last_fresh":"2024-03-24T12:42:50.350752+0000","last_change":"2024-03-24T12:35:58.752262+0000","last_active":"2024-03-24T12:42:50.350752+0000","last_peered":"2024-03-24T12:42:50.350752+0000","last_clean":"2024-03-24T12:35:54.684037+0000","last_became_active":"2024-03-24T12:35:58.752262+0000","last_became_peered":"2024-03-24T12:35:58.752262+0000","last_unstale":"2024-03-24T12:42:50.350752+0000","last_undegraded":"2024-03-24T12:35:58.701740+0000","last_fullsized":"2024-03-24T12:35:58.701602+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-24T12:35:00.998713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:35:00.998713+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":["2","7"],"object_location_counts":[{"shards":"2,7","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":554,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.353553+0000","last_change":"2024-03-24T12:37:38.861709+0000","last_active":"2024-03-24T12:42:50.353553+0000","last_peered":"2024-03-24T12:42:50.353553+0000","last_clean":"2024-03-24T12:42:50.353553+0000","last_became_active":"2024-03-24T12:26:03.161945+0000","last_became_peered":"2024-03-24T12:26:03.161945+0000","last_unstale":"2024-03-24T12:42:50.353553+0000","last_undegraded":"2024-03-24T12:42:50.353553+0000","last_fullsized":"2024-03-24T12:42:50.353553+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:37:38.861597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:37:38.861597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"207'20","reported_seq":640,"reported_epoch":319,"state":"active+clean","last_fresh":"2024-03-24T12:42:50.261060+0000","last_change":"2024-03-24T12:41:06.988179+0000","last_active":"2024-03-24T12:42:50.261060+0000","last_peered":"2024-03-24T12:42:50.261060+0000","last_clean":"2024-03-24T12:42:50.261060+0000","last_became_active":"2024-03-24T12:39:34.759881+0000","last_became_peered":"2024-03-24T12:39:34.759881+0000","last_unstale":"2024-03-24T12:42:50.261060+0000","last_undegraded":"2024-03-24T12:42:50.261060+0000","last_fullsized":"2024-03-24T12:42:50.261060+0000","mapping_epoch":254,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:41:06.988045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:41:06.988045+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":7455571992,"num_objects":113764,"num_object_clones":0,"num_object_copies":341292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35450,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113764,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":140180,"num_write_kb":8126145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20056518656,"data_stored":20056506440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":80641,"ondisk_log_size":80641,"up":43,"acting":43,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1002,"num_read_kb":997,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":8,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":29,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":5246,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":85,"acting":90,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":23,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1122,"num_read_kb":879,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":78,"num_bytes_recovered":1144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430080,"data_stored":11436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":759,"ondisk_log_size":759,"up":88,"acting":92,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":83,"acting":87,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":82,"acting":86,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037966,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2405152,"kb_used_data":1814488,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":91340000,"statfs":{"total":95995035648,"available":93532160000,"internally_reserved":0,"allocated":1858035712,"data_stored":1857334330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":604830154},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,0,0,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332308,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3726680,"kb_used_data":3222032,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":90018472,"statfs":{"total":95995035648,"available":92178915328,"internally_reserved":0,"allocated":3299360768,"data_stored":3298667344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":516684231},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,2,3,0,0,2,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575087,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1930452,"kb_used_data":1371724,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":91814700,"statfs":{"total":95995035648,"available":94018252800,"internally_reserved":0,"allocated":1404645376,"data_stored":1403892561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738611,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5302720,"kb_used_data":4573240,"kb_used_omap":1,"kb_used_meta":729470,"kb_avail":88442432,"statfs":{"total":95995035648,"available":90565050368,"internally_reserved":0,"allocated":4682997760,"data_stored":4682264091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":746977733},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,11,1,10,3,3],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411561,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3371292,"kb_used_data":2718164,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":90373860,"statfs":{"total":95995035648,"available":92542832640,"internally_reserved":0,"allocated":2783399936,"data_stored":2782702506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":668793285},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,3,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117218,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2936452,"kb_used_data":2283320,"kb_used_omap":1,"kb_used_meta":653118,"kb_avail":90808700,"statfs":{"total":95995035648,"available":92988108800,"internally_reserved":0,"allocated":2338119680,"data_stored":2337386720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":668793287},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920990,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4182904,"kb_used_data":3660268,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":89562248,"statfs":{"total":95995035648,"available":91711741952,"internally_reserved":0,"allocated":3748114432,"data_stored":3747392176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":42,"seq":180388626566,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957988,"kb_used_data":688028,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":92787164,"statfs":{"total":95995035648,"available":95014055936,"internally_reserved":0,"allocated":704540672,"data_stored":704173829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4676456448,"data_stored":4676452376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1401356288,"data_stored":1401356288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2776956928,"data_stored":2776956928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2326069248,"data_stored":2326069248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3734638592,"data_stored":3734634520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3286044672,"data_stored":3286040600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1854996480,"data_stored":1854996480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:43:00.878 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-24T12:43:00.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 26 2024-03-24T12:43:00.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:00.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:00.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:00.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:01.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:01.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:01.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:01.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:01.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:00 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:43:01.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:00 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:43:01.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:00 smithi190 ceph-mon[33748]: osdmap e321: 8 total, 7 up, 8 in 2024-03-24T12:43:01.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:00 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 35024/336369 objects degraded (10.412%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:01.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:00 smithi190 ceph-mon[33748]: osdmap e322: 8 total, 7 up, 8 in 2024-03-24T12:43:01.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:00 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:43:01.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:00 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:43:01.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:00 smithi029 ceph-mon[29919]: osdmap e321: 8 total, 7 up, 8 in 2024-03-24T12:43:01.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:00 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 35024/336369 objects degraded (10.412%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:01.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:00 smithi029 ceph-mon[29919]: osdmap e322: 8 total, 7 up, 8 in 2024-03-24T12:43:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:00 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:43:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:00 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:43:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:00 smithi029 ceph-mon[41500]: osdmap e321: 8 total, 7 up, 8 in 2024-03-24T12:43:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:00 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 35024/336369 objects degraded (10.412%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:01.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:00 smithi029 ceph-mon[41500]: osdmap e322: 8 total, 7 up, 8 in 2024-03-24T12:43:01.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:01.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:01.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:01.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:01.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:01.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:01.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:01.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:01.500 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:01.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:01.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:01.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:01.631 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:01.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:01.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:01.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:01.765 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:01.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:01.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:01.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:01.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:01.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:02.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:02.038 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:02.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:02.139 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:02.164 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:02.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:02.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:02.316 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:02.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:02.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:02 smithi029 ceph-mon[29919]: pgmap v729: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 427 op/s; 34751/334686 objects degraded (10.383%); 28/334686 objects misplaced (0.008%) 2024-03-24T12:43:02.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:02 smithi029 ceph-mon[41500]: pgmap v729: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 427 op/s; 34751/334686 objects degraded (10.383%); 28/334686 objects misplaced (0.008%) 2024-03-24T12:43:02.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:02.455 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:02.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:02.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:02.601 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:02.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:02 smithi190 ceph-mon[33748]: pgmap v729: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 427 op/s; 34751/334686 objects degraded (10.383%); 28/334686 objects misplaced (0.008%) 2024-03-24T12:43:02.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:02.736 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:02.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:02.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:02.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:02.891 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:02.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:02.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:03.023 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:03.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:03.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:03.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:03.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:03.162 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:03.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:03.224 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:43:03.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:03.293 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:03.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:03.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:03.397 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:03.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:03.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:03.535 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:03.565 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:03.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:03.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:03.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:03.709 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:03.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:03.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:03.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:03.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:03.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:03.964 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:43:03.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:03.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:03.983 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:43:03.983 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:43:03.983 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:43:03.983 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:43:03.983 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.983 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:133729 8.0 8:0310b530:::benchmark_data_smithi029_108893_object19869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e190)", 2024-03-24T12:43:03.983 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:34:48.330728+0000", 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 495.51671525299997, 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.011871341, 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 133729 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:48.330728+0000", 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:03.984 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:48.330728+0000", 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:48.330730+0000", 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.61e-06 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:03.985 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:48.330736+0000", 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0260000000000004e-06 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:48.330753+0000", 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6985000000000001e-05 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:48.330762+0000", 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3540000000000006e-06 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.986 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:48.339256+0000", 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0084942260000000006 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:48.339265+0000", 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1200000000000008e-06 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:49.335260+0000", 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.99599477800000003 2024-03-24T12:43:03.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:49.335401+0000", 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000140758 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,5", 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:49.335488+0000", 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6774999999999995e-05 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:03.988 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:49.339490+0000", 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.004002294 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:49.340893+0000", 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001403465 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:49.342511+0000", 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0016172000000000001 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.989 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:03.990 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:49.342565+0000", 2024-03-24T12:43:03.990 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4435000000000003e-05 2024-03-24T12:43:03.990 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.990 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.990 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:03.990 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:34:49.342599+0000", 2024-03-24T12:43:03.990 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4314999999999999e-05 2024-03-24T12:43:03.990 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:03.990 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:206537 8.0 8:0ce22cc9:::benchmark_data_smithi029_108893_object92677:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e203)", 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:57.700761+0000", 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 426.14668188100001, 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0934872710000001, 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:43:03.991 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 206537 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700761+0000", 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700761+0000", 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.992 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700763+0000", 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2300000000000001e-06 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700766+0000", 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4000000000000001e-06 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700774+0000", 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4100000000000008e-06 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.993 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700784+0000", 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8730000000000002e-06 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711043+0000", 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010258478 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711052+0000", 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2299999999999997e-06 2024-03-24T12:43:03.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.778190+0000", 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0671383839999999 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.779149+0000", 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00095899099999999997 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2", 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.779275+0000", 2024-03-24T12:43:03.995 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012562900000000001 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.785020+0000", 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0057455680000000004 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.794130+0000", 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0091095369999999991 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.794160+0000", 2024-03-24T12:43:03.996 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0341999999999999e-05 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.794249+0000", 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8199e-05 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:206531 8.8 8:158724a5:::benchmark_data_smithi029_108893_object92671:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:57.700796+0000", 2024-03-24T12:43:03.997 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 426.14664705799999, 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0814151139999999, 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 206531 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700796+0000", 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:03.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700796+0000", 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700797+0000", 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7e-07 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700799+0000", 2024-03-24T12:43:03.999 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.649e-06 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700806+0000", 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1169999999999998e-06 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700812+0000", 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.998e-06 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.000 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711080+0000", 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010267422999999999 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711087+0000", 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.888e-06 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.768780+0000", 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0576933260000001 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.001 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.771548+0000", 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002767734 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 7", 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.771942+0000", 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00039435499999999999 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.778111+0000", 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0061692530000000004 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.002 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.004 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.004 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.782046+0000", 2024-03-24T12:43:04.004 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0039349010000000002 2024-03-24T12:43:04.004 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.004 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.004 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.782093+0000", 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7191999999999998e-05 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.782211+0000", 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011780799999999999 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.005 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:206536 8.8 8:1d153299:::benchmark_data_smithi029_108893_object92676:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:57.700820+0000", 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 426.14662315999999, 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0933002700000001, 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 206536 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.006 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.007 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.007 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700820+0000", 2024-03-24T12:43:04.007 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.007 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.007 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700820+0000", 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700820+0000", 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0600000000000001e-07 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700823+0000", 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4260000000000002e-06 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.008 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700829+0000", 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2099999999999998e-06 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.700835+0000", 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4879999999999998e-06 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711119+0000", 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010284941000000001 2024-03-24T12:43:04.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.711127+0000", 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7559999999999999e-06 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.778145+0000", 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0670174139999999 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.778262+0000", 2024-03-24T12:43:04.010 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000117243 2024-03-24T12:43:04.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.011 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.011 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 7", 2024-03-24T12:43:04.012 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.778384+0000", 2024-03-24T12:43:04.012 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000121873 2024-03-24T12:43:04.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.013 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.013 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.013 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.782034+0000", 2024-03-24T12:43:04.014 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0036501390000000002 2024-03-24T12:43:04.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.014 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.014 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.014 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.793917+0000", 2024-03-24T12:43:04.014 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.011883060000000001 2024-03-24T12:43:04.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.794029+0000", 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011179300000000001 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.794120+0000", 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1520999999999993e-05 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.015 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24815.0:206549 8.0 8:0b3c95fa:::benchmark_data_smithi029_108893_object92689:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e204)", 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:35:57.721395+0000", 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 426.12604827500002, 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0730424190000001, 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24815", 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/523697355", 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 206549 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.016 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.017 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.017 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.721395+0000", 2024-03-24T12:43:04.017 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.017 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.017 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.017 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.721395+0000", 2024-03-24T12:43:04.017 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.721396+0000", 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.28e-06 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.721401+0000", 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1329999999999998e-06 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.018 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.721412+0000", 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1071e-05 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.721420+0000", 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5800000000000003e-06 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.721467+0000", 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6743999999999998e-05 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.019 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:57.721472+0000", 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5130000000000004e-06 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.782242+0000", 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0607700790000001 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.782349+0000", 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106963 2024-03-24T12:43:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2", 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.782465+0000", 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115419 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.793959+0000", 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.011494262 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.794257+0000", 2024-03-24T12:43:04.021 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029794700000000001 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.794341+0000", 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3679999999999998e-05 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:35:58.794437+0000", 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6748000000000003e-05 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.022 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104326 9.1 9:8c714a94:::benchmark_data_smithi029_134815_object104325:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.119854+0000", 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 293.72758892000002, 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0492705629999999, 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.023 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104326 2024-03-24T12:43:04.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.024 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.024 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.024 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.024 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.119854+0000", 2024-03-24T12:43:04.024 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.024 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.024 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.119854+0000", 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.119856+0000", 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.074e-06 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.119881+0000", 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4236999999999999e-05 2024-03-24T12:43:04.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.119896+0000", 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5909e-05 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.119905+0000", 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2099999999999993e-06 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.026 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.120026+0000", 2024-03-24T12:43:04.027 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000121728 2024-03-24T12:43:04.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.027 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.027 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.120038+0000", 2024-03-24T12:43:04.027 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1294999999999999e-05 2024-03-24T12:43:04.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.027 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.027 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090187+0000", 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.97014882099999999 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090201+0000", 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4148000000000001e-05 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.161256+0000", 2024-03-24T12:43:04.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0710551079999999 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.161425+0000", 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016903600000000001 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.161963+0000", 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00053837600000000002 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.029 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.168599+0000", 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.006636009 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.168678+0000", 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8461000000000007e-05 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.168735+0000", 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7080000000000002e-05 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.168959+0000", 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000224217 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.169016+0000", 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7027000000000001e-05 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.169125+0000", 2024-03-24T12:43:04.031 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010882699999999999 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104329 9.1 9:843fadc3:::benchmark_data_smithi029_134815_object104328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.124348+0000", 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 293.72309544199999, 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0610869080000001, 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.032 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104329 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.124348+0000", 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.124348+0000", 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.034 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.034 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.034 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.124349+0000", 2024-03-24T12:43:04.034 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1310000000000001e-06 2024-03-24T12:43:04.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.034 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.034 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.034 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.124372+0000", 2024-03-24T12:43:04.034 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2679000000000001e-05 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.124384+0000", 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2519e-05 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.124393+0000", 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7350000000000002e-06 2024-03-24T12:43:04.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.124472+0000", 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9338999999999994e-05 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.124483+0000", 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0499000000000001e-05 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090235+0000", 2024-03-24T12:43:04.036 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.965752049 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090244+0000", 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9609999999999994e-06 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.169172+0000", 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0789279920000001 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.037 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.169324+0000", 2024-03-24T12:43:04.038 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015285799999999999 2024-03-24T12:43:04.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.038 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.038 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.038 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.169547+0000", 2024-03-24T12:43:04.038 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00022271899999999999 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.175243+0000", 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0056953380000000003 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.175293+0000", 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0223999999999999e-05 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.039 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.175350+0000", 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7271000000000001e-05 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185252+0000", 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0099023649999999998 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185310+0000", 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7986999999999999e-05 2024-03-24T12:43:04.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185435+0000", 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012424199999999999 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104335 9.1 9:87fb922f:::benchmark_data_smithi029_134815_object104334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.128944+0000", 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 293.71849908299998, 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.05671684, 2024-03-24T12:43:04.041 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104335 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128944+0000", 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.042 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128944+0000", 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128946+0000", 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.502e-06 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128968+0000", 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2821999999999999e-05 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.043 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128982+0000", 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3115999999999999e-05 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128990+0000", 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1429999999999998e-06 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129056+0000", 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6476999999999994e-05 2024-03-24T12:43:04.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.045 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.045 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.045 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129067+0000", 2024-03-24T12:43:04.045 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1276e-05 2024-03-24T12:43:04.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.045 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.045 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.045 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090270+0000", 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.96120262400000001 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090279+0000", 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4129999999999995e-06 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.172066+0000", 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0817869879999999 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.046 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.172239+0000", 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017228100000000001 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.172430+0000", 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019171699999999999 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185194+0000", 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012763864 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.047 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185239+0000", 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4696e-05 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185445+0000", 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020580699999999999 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185491+0000", 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6641999999999998e-05 2024-03-24T12:43:04.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.050 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.050 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.050 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185549+0000", 2024-03-24T12:43:04.050 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7065999999999999e-05 2024-03-24T12:43:04.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.050 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185661+0000", 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011240599999999999 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104348 9.1 9:8b4995a9:::benchmark_data_smithi029_134815_object104347:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.146488+0000", 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 293.70095492199999, 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.04818254, 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.051 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104348 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.146488+0000", 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.146488+0000", 2024-03-24T12:43:04.052 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.146490+0000", 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.39e-06 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.146512+0000", 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2090999999999999e-05 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.053 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.146527+0000", 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4939e-05 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.146535+0000", 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1410000000000005e-06 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.146602+0000", 2024-03-24T12:43:04.054 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7038000000000004e-05 2024-03-24T12:43:04.055 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.055 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.055 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.055 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.146612+0000", 2024-03-24T12:43:04.055 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9029999999999999e-06 2024-03-24T12:43:04.055 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.055 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.055 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.055 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090306+0000", 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94369384099999998 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090314+0000", 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6770000000000001e-06 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.182315+0000", 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0920002820000001 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.056 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.182472+0000", 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000157539 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.182658+0000", 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018624599999999999 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185450+0000", 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027921399999999998 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.188703+0000", 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0032526790000000001 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194436+0000", 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0057323900000000004 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194476+0000", 2024-03-24T12:43:04.058 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0648000000000001e-05 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194537+0000", 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1273999999999996e-05 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194671+0000", 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013332200000000001 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.059 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.062 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.062 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104350 9.1 9:80fe91c1:::benchmark_data_smithi029_134815_object104349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:43:04.062 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.147151+0000", 2024-03-24T12:43:04.062 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 293.70029249999999, 2024-03-24T12:43:04.062 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0516863349999999, 2024-03-24T12:43:04.062 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.062 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.062 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.062 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104350 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.147151+0000", 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.147151+0000", 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.147152+0000", 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0100000000000001e-06 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.147173+0000", 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0988e-05 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.064 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.147183+0000", 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0739e-05 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.147202+0000", 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8403999999999999e-05 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.147257+0000", 2024-03-24T12:43:04.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5003999999999997e-05 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.147266+0000", 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3950000000000007e-06 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090341+0000", 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94307496400000002 2024-03-24T12:43:04.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090350+0000", 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5429999999999994e-06 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185764+0000", 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.095414364 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.185930+0000", 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016548800000000001 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.186114+0000", 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018452799999999999 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194351+0000", 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0082371939999999998 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.068 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194748+0000", 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00039659800000000001 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194785+0000", 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7116000000000002e-05 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.198464+0000", 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003678944 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.198522+0000", 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7964000000000003e-05 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.198837+0000", 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000315092 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104367 9.1 9:806c6c03:::benchmark_data_smithi029_134815_object104366:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.185825+0000", 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 293.66161782199998, 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0132577820000002, 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104367 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.071 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.185825+0000", 2024-03-24T12:43:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.185825+0000", 2024-03-24T12:43:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.185827+0000", 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1179999999999999e-06 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.185855+0000", 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7271e-05 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.185869+0000", 2024-03-24T12:43:04.073 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4708e-05 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.185880+0000", 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.042e-05 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.185960+0000", 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9803999999999997e-05 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.185972+0000", 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2173e-05 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090376+0000", 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.90440379800000004 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090384+0000", 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8009999999999993e-06 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.075 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.188741+0000", 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0983563240000001 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.188890+0000", 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014924900000000001 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.189075+0000", 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018514700000000001 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194397+0000", 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0053220079999999996 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.197349+0000", 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002952222 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.197390+0000", 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0503000000000002e-05 2024-03-24T12:43:04.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.198899+0000", 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001509568 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.198964+0000", 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4319000000000005e-05 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199083+0000", 2024-03-24T12:43:04.078 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000119349 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104377 9.1 9:82b4f125:::benchmark_data_smithi029_134815_object104376:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.213638+0000", 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 293.63380523199999, 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9856139239999999, 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.079 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104377 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.213638+0000", 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.213638+0000", 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.213639+0000", 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.057e-06 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.213664+0000", 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.5228e-05 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.081 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.213676+0000", 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2191999999999999e-05 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.213685+0000", 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8619999999999992e-06 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.213740+0000", 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4991999999999998e-05 2024-03-24T12:43:04.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.213750+0000", 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5100000000000004e-06 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090411+0000", 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.87666082700000003 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.090419+0000", 2024-03-24T12:43:04.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1890000000000007e-06 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.191703+0000", 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.101284119 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.191855+0000", 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015183500000000001 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.084 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.192036+0000", 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018170600000000001 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194710+0000", 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026738230000000001 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.198432+0000", 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0037219929999999998 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.198886+0000", 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00045387999999999998 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199093+0000", 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000206887 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199151+0000", 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8056000000000001e-05 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199252+0000", 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010076799999999999 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172217 9.1 9:81a84da0:::benchmark_data_smithi029_134815_object60151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.805961+0000", 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 210.041482621, 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4110725230000001, 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.087 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172217 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.805961+0000", 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.088 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.805961+0000", 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.805962+0000", 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.017e-06 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.805967+0000", 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8499999999999999e-06 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.089 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.805978+0000", 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0502e-05 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.805986+0000", 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5189999999999992e-06 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.806058+0000", 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1526000000000006e-05 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.806069+0000", 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0913e-05 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.746088+0000", 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94001873899999999 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.746102+0000", 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4294000000000001e-05 2024-03-24T12:43:04.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.211489+0000", 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.46538688099999997 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.211600+0000", 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000110807 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.211721+0000", 2024-03-24T12:43:04.092 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000121014 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.216702+0000", 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0049817419999999999 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.216759+0000", 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6570000000000002e-05 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.093 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.216774+0000", 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5492999999999999e-05 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.216817+0000", 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3027e-05 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.216903+0000", 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5329999999999998e-05 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.094 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.217033+0000", 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013029900000000001 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172233 9.1 9:88c077ed:::benchmark_data_smithi029_134815_object60167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.882689+0000", 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 209.964753691, 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3403986569999999, 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.095 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172233 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.882689+0000", 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.096 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.882689+0000", 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.882690+0000", 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7999999999999995e-07 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.882694+0000", 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.788e-06 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.097 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.882706+0000", 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.164e-05 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.882714+0000", 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0509999999999997e-06 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.882779+0000", 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5413999999999994e-05 2024-03-24T12:43:04.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.882789+0000", 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.024e-05 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.746130+0000", 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.86334034599999998 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.746138+0000", 2024-03-24T12:43:04.099 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1620000000000008e-06 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.217086+0000", 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.47094847400000001 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.217231+0000", 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000144785 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.100 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.217387+0000", 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015583599999999999 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.222360+0000", 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0049726329999999997 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.222394+0000", 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4910000000000003e-05 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.101 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.222434+0000", 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9390000000000001e-05 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.222785+0000", 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00035117299999999998 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.222958+0000", 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000172826 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.102 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.223088+0000", 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000130309 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172236 9.1 9:876a8dec:::benchmark_data_smithi029_134815_object60170:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.887373+0000", 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 209.96006978, 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3421340310000001, 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.103 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172236 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.887373+0000", 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.104 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.887373+0000", 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.887374+0000", 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2699999999999998e-07 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.887379+0000", 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3889999999999998e-06 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.105 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.887389+0000", 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8470000000000008e-06 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.887395+0000", 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.117e-06 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.887441+0000", 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5908000000000002e-05 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.106 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.887448+0000", 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.939e-06 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.746160+0000", 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.85871231000000003 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.746167+0000", 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2899999999999997e-06 2024-03-24T12:43:04.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.219971+0000", 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.47380402300000002 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.220095+0000", 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000124339 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.220229+0000", 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133869 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.222708+0000", 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0024787810000000002 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.223135+0000", 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00042639899999999999 2024-03-24T12:43:04.109 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.223314+0000", 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017932000000000001 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.229364+0000", 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060498380000000001 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.110 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.229413+0000", 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9215000000000002e-05 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.229507+0000", 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4519999999999996e-05 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172244 9.1 9:8bbb4662:::benchmark_data_smithi029_134815_object60178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.905660+0000", 2024-03-24T12:43:04.111 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 209.941783122, 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3241364789999999, 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172244 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.905660+0000", 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.905660+0000", 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.905661+0000", 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3200000000000004e-07 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.905664+0000", 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.3249999999999999e-06 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.905673+0000", 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8140000000000007e-06 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.905679+0000", 2024-03-24T12:43:04.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6710000000000004e-06 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.905730+0000", 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1780000000000002e-05 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.905740+0000", 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0046e-05 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.746188+0000", 2024-03-24T12:43:04.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.84044739400000001 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.746196+0000", 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5750000000000001e-06 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.223347+0000", 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.477150134 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.223455+0000", 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010882000000000001 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4,6,7", 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.223577+0000", 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000121242 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.229342+0000", 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0057654439999999998 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.229517+0000", 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017541799999999999 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.229581+0000", 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3301999999999998e-05 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.229633+0000", 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2157999999999998e-05 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.119 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.119 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.229686+0000", 2024-03-24T12:43:04.119 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2714999999999999e-05 2024-03-24T12:43:04.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.119 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.119 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.229797+0000", 2024-03-24T12:43:04.119 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000110909 2024-03-24T12:43:04.119 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.119 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.120 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.120 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:114602 10.3 e308/278 10:ce14094c:::benchmark_data_smithi029_160664_object114601:head v 308'7110, mlcod=308'7108)", 2024-03-24T12:43:04.120 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:42:22.373496+0000", 2024-03-24T12:43:04.120 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 41.473946924000003, 2024-03-24T12:43:04.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.415737315, 2024-03-24T12:43:04.121 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.121 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.121 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.121 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.121 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373496+0000", 2024-03-24T12:43:04.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.122 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.122 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373496+0000", 2024-03-24T12:43:04.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373497+0000", 2024-03-24T12:43:04.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.001e-06 2024-03-24T12:43:04.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373530+0000", 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.3049000000000002e-05 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373546+0000", 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5488000000000001e-05 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373555+0000", 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5300000000000002e-06 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373642+0000", 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6261000000000006e-05 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373666+0000", 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4859999999999999e-05 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.789165+0000", 2024-03-24T12:43:04.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.41549841100000001 2024-03-24T12:43:04.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.126 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.126 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.789234+0000", 2024-03-24T12:43:04.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8714999999999995e-05 2024-03-24T12:43:04.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.126 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:114609 10.f e308/278 10:f32b54ec:::benchmark_data_smithi029_160664_object114608:head v 308'7121, mlcod=308'7119)", 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:42:22.375049+0000", 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 41.472394270999999, 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40513417200000001, 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.375049+0000", 2024-03-24T12:43:04.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.375049+0000", 2024-03-24T12:43:04.129 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.129 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.129 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.129 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.375050+0000", 2024-03-24T12:43:04.129 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7999999999999993e-07 2024-03-24T12:43:04.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.130 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.130 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.375093+0000", 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2713999999999998e-05 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.375103+0000", 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0095000000000001e-05 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.375110+0000", 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1450000000000002e-06 2024-03-24T12:43:04.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.375167+0000", 2024-03-24T12:43:04.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7598e-05 2024-03-24T12:43:04.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.375187+0000", 2024-03-24T12:43:04.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9548999999999999e-05 2024-03-24T12:43:04.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.780135+0000", 2024-03-24T12:43:04.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40494760699999999 2024-03-24T12:43:04.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.780183+0000", 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8483999999999999e-05 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:114612 10.7 e308/278 10:ed96140a:::benchmark_data_smithi029_160664_object114611:head v 308'7298, mlcod=308'7296)", 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:42:22.377967+0000", 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 41.469476092999997, 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40224274100000001, 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.134 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.135 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.135 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.377967+0000", 2024-03-24T12:43:04.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.135 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.135 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.377967+0000", 2024-03-24T12:43:04.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.136 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.377968+0000", 2024-03-24T12:43:04.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8200000000000008e-07 2024-03-24T12:43:04.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.137 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.377998+0000", 2024-03-24T12:43:04.137 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0298e-05 2024-03-24T12:43:04.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.137 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.137 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.378009+0000", 2024-03-24T12:43:04.137 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0798999999999999e-05 2024-03-24T12:43:04.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.138 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.138 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.138 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.378017+0000", 2024-03-24T12:43:04.138 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.486e-06 2024-03-24T12:43:04.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.138 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.139 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.139 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.773955+0000", 2024-03-24T12:43:04.139 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.39593849399999997 2024-03-24T12:43:04.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.139 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.139 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.773977+0000", 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2123000000000001e-05 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.780187+0000", 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0062102069999999997 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.780210+0000", 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2351999999999999e-05 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.140 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:114615 10.3 e308/278 10:cd958ff1:::benchmark_data_smithi029_160664_object114614:head v 308'7111, mlcod=308'7108)", 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:42:22.382395+0000", 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 41.465047937999998, 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.41094538400000002, 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.382395+0000", 2024-03-24T12:43:04.141 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.382395+0000", 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.382396+0000", 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7600000000000006e-07 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.142 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.382422+0000", 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.5834e-05 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.773203+0000", 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.39078066099999997 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.773210+0000", 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5680000000000002e-06 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.773621+0000", 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041051999999999999 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.773649+0000", 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7761000000000001e-05 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.792884+0000", 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.019235845000000001 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:43:04.144 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:43:04.145 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.793341+0000", 2024-03-24T12:43:04.145 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000456219 2024-03-24T12:43:04.145 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.145 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.145 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.145 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:43:04.145 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:43:04.145 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:43:04.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:04.147 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:43:04.147 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:43:04.147 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:43:04.147 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:43:04.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:04.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:04.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:04.188 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:43:04.225 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:43:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:43:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:43:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:43:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:43:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:43:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:43:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:43:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:43:04.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:04.320 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:04.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:04.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:04 smithi190 ceph-mon[33748]: pgmap v730: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 431 op/s; 34608/332079 objects degraded (10.422%); 28/332079 objects misplaced (0.008%) 2024-03-24T12:43:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:04 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3004623957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:43:04.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:04 smithi029 ceph-mon[29919]: pgmap v730: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 431 op/s; 34608/332079 objects degraded (10.422%); 28/332079 objects misplaced (0.008%) 2024-03-24T12:43:04.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:04.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:04.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:04 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3004623957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:43:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:04 smithi029 ceph-mon[41500]: pgmap v730: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 431 op/s; 34608/332079 objects degraded (10.422%); 28/332079 objects misplaced (0.008%) 2024-03-24T12:43:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:04.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:04 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3004623957' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:43:04.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:04.467 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:04.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:04.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:04.605 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:04.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:04.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:04.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:04.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:04.758 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:04.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:04.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:04.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:04.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:05.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:05.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:05.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:05.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:05.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:05.179 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:05.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:05.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:05.317 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:05.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:05.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:05.419 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:05.461 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:05.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:05.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:05.563 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:05.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:05.597 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:05.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:05.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:05.740 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:05.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:05.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:05.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:05.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:05.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:05.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:43:05.945 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:43:05.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:06.006 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:06.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:06.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:06.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:06.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:06.142 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:06.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:06.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:06 smithi029 ceph-mon[29919]: pgmap v731: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 570 op/s; 34031/328215 objects degraded (10.369%); 28/328215 objects misplaced (0.009%) 2024-03-24T12:43:06.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:06 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:43:06.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:06 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:43:06.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:06 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 34031/328215 objects degraded (10.369%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:06.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:06 smithi029 ceph-mon[41500]: pgmap v731: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 570 op/s; 34031/328215 objects degraded (10.369%); 28/328215 objects misplaced (0.009%) 2024-03-24T12:43:06.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:06 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:43:06.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:06 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:43:06.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:06 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 34031/328215 objects degraded (10.369%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:06.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:06.267 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:06.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:06.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:06 smithi190 ceph-mon[33748]: pgmap v731: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 570 op/s; 34031/328215 objects degraded (10.369%); 28/328215 objects misplaced (0.009%) 2024-03-24T12:43:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:06 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:43:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:06 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:43:06.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:06 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 34031/328215 objects degraded (10.369%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:06.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:06.403 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:06.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:06.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:06.536 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:06.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:06.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:06.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:06.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:06.749 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:06.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:06.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:06.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:06.883 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:06.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:43:06.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:07.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:07.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:07.140 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:07.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:07.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:07.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:07.305 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:07.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:07.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:07 smithi190 ceph-mon[33748]: osdmap e323: 8 total, 7 up, 8 in 2024-03-24T12:43:07.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:07.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:07 smithi029 ceph-mon[29919]: osdmap e323: 8 total, 7 up, 8 in 2024-03-24T12:43:07.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:07 smithi029 ceph-mon[41500]: osdmap e323: 8 total, 7 up, 8 in 2024-03-24T12:43:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:07.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:07.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:07.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T12:43:07.546 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:07.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:07.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:07.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:07.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:07.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:07.733 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:07.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:07.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:07.863 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:43:07.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:07.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:07.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:07.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:43:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:43:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:43:07.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:43:07.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:08.013 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:08.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:08.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:08.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:08.166 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:08.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:08.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:08.302 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:08.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:08.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:08 smithi029 ceph-mon[29919]: pgmap v733: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 330 op/s; 34031/327846 objects degraded (10.380%); 28/327846 objects misplaced (0.009%) 2024-03-24T12:43:08.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:08 smithi029 ceph-mon[41500]: pgmap v733: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 330 op/s; 34031/327846 objects degraded (10.380%); 28/327846 objects misplaced (0.009%) 2024-03-24T12:43:08.336 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:08 smithi190 ceph-mon[33748]: pgmap v733: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 740 B/s rd, 0 B/s wr, 330 op/s; 34031/327846 objects degraded (10.380%); 28/327846 objects misplaced (0.009%) 2024-03-24T12:43:08.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:08.450 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:08.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:08.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:08.590 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:08.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:08.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:08.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:43:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:43:08.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:08.730 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:08.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:08.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:08.880 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:08.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:08.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:09.021 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:09.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:09.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:09.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:09.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:09.263 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:09.295 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:09.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:09.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:09.397 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:09.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:09.439 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:09.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:09.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:09.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:09.585 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:09.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:09.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:09.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:09.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:09.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:09.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:09.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:09.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:09.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:09.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:10.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:10.012 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:10.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:10.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:10.166 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:10.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:10.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:10.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:10.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:10.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:10.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:10 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:10.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:10 smithi029 ceph-mon[41500]: pgmap v734: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 462 op/s; 33612/322773 objects degraded (10.414%); 28/322773 objects misplaced (0.009%) 2024-03-24T12:43:10.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:10 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:10.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:10 smithi029 ceph-mon[29919]: pgmap v734: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 462 op/s; 33612/322773 objects degraded (10.414%); 28/322773 objects misplaced (0.009%) 2024-03-24T12:43:10.417 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:10 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:10.417 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:10 smithi190 ceph-mon[33748]: pgmap v734: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.6 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 462 op/s; 33612/322773 objects degraded (10.414%); 28/322773 objects misplaced (0.009%) 2024-03-24T12:43:10.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:10.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:10.453 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:10.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:10.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:10.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:10.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:10.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:10.662 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:43:10.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:10.725 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:10.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:10.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:10.827 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:10.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:10.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:10.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:10.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:10.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:11.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:11.126 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:11.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:11.194 INFO:teuthology.orchestra.run.smithi029.stderr:set pool 10 pg_num to 26 2024-03-24T12:43:11.198 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:43:11.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:11.253 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:11.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:11.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:11.362 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 33612/322773 objects degraded (10.414%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:11.362 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[41500]: osdmap e324: 8 total, 7 up, 8 in 2024-03-24T12:43:11.362 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/585066251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:43:11.362 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:43:11.362 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1937527082' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:43:11.362 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:43:11.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 33612/322773 objects degraded (10.414%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:11.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[29919]: osdmap e324: 8 total, 7 up, 8 in 2024-03-24T12:43:11.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/585066251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:43:11.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:43:11.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1937527082' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:43:11.363 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:11 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:43:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:11.397 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:11.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:11.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:11 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 33612/322773 objects degraded (10.414%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:11.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:11 smithi190 ceph-mon[33748]: osdmap e324: 8 total, 7 up, 8 in 2024-03-24T12:43:11.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:11 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/585066251' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:43:11.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:11 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:43:11.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:11 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1937527082' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:43:11.405 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:11 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-24T12:43:11.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:11.538 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:11.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:11.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:11.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:11.676 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:11.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:11.778 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:11.812 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:11.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:11.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:11.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:11.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:11.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:12.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:12.081 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:12.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:12.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:12.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:12.208 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:12.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:12.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:12.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:12.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:12.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:12.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[29919]: pgmap v736: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 452 op/s; 33374/321231 objects degraded (10.389%); 28/321231 objects misplaced (0.009%) 2024-03-24T12:43:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[29919]: 10.f deep-scrub ok 2024-03-24T12:43:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:43:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:43:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:43:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[29919]: osdmap e325: 8 total, 7 up, 8 in 2024-03-24T12:43:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:43:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[41500]: pgmap v736: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 452 op/s; 33374/321231 objects degraded (10.389%); 28/321231 objects misplaced (0.009%) 2024-03-24T12:43:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[41500]: 10.f deep-scrub ok 2024-03-24T12:43:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:43:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:43:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:43:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[41500]: osdmap e325: 8 total, 7 up, 8 in 2024-03-24T12:43:12.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:12 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:43:12.441 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:12.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:12.473 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:12.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:12.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:12 smithi190 ceph-mon[33748]: pgmap v736: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 96 active+clean, 21 active+undersized, 5 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.5 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 452 op/s; 33374/321231 objects degraded (10.389%); 28/321231 objects misplaced (0.009%) 2024-03-24T12:43:12.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:12 smithi190 ceph-mon[33748]: 10.f deep-scrub ok 2024-03-24T12:43:12.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:12 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:43:12.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:12 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:43:12.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:12 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-24T12:43:12.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:12 smithi190 ceph-mon[33748]: osdmap e325: 8 total, 7 up, 8 in 2024-03-24T12:43:12.519 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:12 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:43:12.575 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:12.612 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:12.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:12.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:12.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:12.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:12.737 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:12.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:12.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:12.873 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:12.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:12.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:12.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:12.989 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:43:12.989 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:43:13.009 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:13.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:13.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:13.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:13.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:13.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:13.309 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:13.310 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:13 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:43:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:13.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:13.411 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:13.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:13 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:43:13.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:13 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-24T12:43:13.444 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:13.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:13.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:13.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:13.570 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:13.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:13.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T12:43:13.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:13.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:13.695 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:13.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:13.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:13.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:13.821 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:13.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:13.882 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:43:13.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:13.947 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:13.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:14.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:14.072 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:14.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:14.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:14.218 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:14.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:14.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:14.287 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:14 smithi190 ceph-mon[33748]: pgmap v738: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 98 active+clean, 21 active+undersized, 3 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 769 B/s rd, 0 B/s wr, 472 op/s; 33181/318432 objects degraded (10.420%); 28/318432 objects misplaced (0.009%) 2024-03-24T12:43:14.287 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:14 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:43:14.287 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:14 smithi190 ceph-mon[33748]: osdmap e326: 8 total, 7 up, 8 in 2024-03-24T12:43:14.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:14.357 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:14.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:14.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:14.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:14 smithi029 ceph-mon[29919]: pgmap v738: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 98 active+clean, 21 active+undersized, 3 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 769 B/s rd, 0 B/s wr, 472 op/s; 33181/318432 objects degraded (10.420%); 28/318432 objects misplaced (0.009%) 2024-03-24T12:43:14.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:14 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:43:14.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:14 smithi029 ceph-mon[29919]: osdmap e326: 8 total, 7 up, 8 in 2024-03-24T12:43:14.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:14 smithi029 ceph-mon[41500]: pgmap v738: 158 pgs: 1 active+clean+scrubbing, 22 active+clean+remapped, 98 active+clean, 21 active+undersized, 3 active+clean+scrubbing+deep, 13 active+undersized+degraded; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 769 B/s rd, 0 B/s wr, 472 op/s; 33181/318432 objects degraded (10.420%); 28/318432 objects misplaced (0.009%) 2024-03-24T12:43:14.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:14 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-24T12:43:14.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:14 smithi029 ceph-mon[41500]: osdmap e326: 8 total, 7 up, 8 in 2024-03-24T12:43:14.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:14.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:14.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:14.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:14.633 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:14.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:14.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:14.772 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:14.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:14.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:14.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:14.911 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:14.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:15.012 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:15.038 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:15.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:15.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:15.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:15.164 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:15.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:15.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:15.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:15.290 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:15.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:15.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:15 smithi190 ceph-mon[33748]: osdmap e327: 8 total, 7 up, 8 in 2024-03-24T12:43:15.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:15 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:43:15.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:15 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 33181/318432 objects degraded (10.420%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:15.377 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:15 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:43:15.378 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:15 smithi190 ceph-mon[33748]: osdmap e328: 8 total, 7 up, 8 in 2024-03-24T12:43:15.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:15.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:15.415 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:15.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:15.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:15.542 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:15.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:15.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:15.644 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:15.650 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:43:15.650 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:43:15.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:15 smithi029 ceph-mon[29919]: osdmap e327: 8 total, 7 up, 8 in 2024-03-24T12:43:15.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:15 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:43:15.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:15 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 33181/318432 objects degraded (10.420%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:15.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:15 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:43:15.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:15 smithi029 ceph-mon[29919]: osdmap e328: 8 total, 7 up, 8 in 2024-03-24T12:43:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:15 smithi029 ceph-mon[41500]: osdmap e327: 8 total, 7 up, 8 in 2024-03-24T12:43:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:15 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-24T12:43:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:15 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 33181/318432 objects degraded (10.420%), 13 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:15 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-24T12:43:15.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:15 smithi029 ceph-mon[41500]: osdmap e328: 8 total, 7 up, 8 in 2024-03-24T12:43:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:15.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:15.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:15.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:15.795 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:15.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:15.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:15.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:15.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:16.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:16.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:16.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:16.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:16.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:16.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:16.183 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:16.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:16.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:16 smithi190 ceph-mon[33748]: pgmap v741: 168 pgs: 2 peering, 7 unknown, 1 active+clean+scrubbing, 22 active+clean+remapped, 101 active+clean, 21 active+undersized, 14 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 651 op/s; 25924/312243 objects degraded (8.303%); 28/312243 objects misplaced (0.009%) 2024-03-24T12:43:16.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:16 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:43:16.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:16 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:43:16.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:16 smithi190 ceph-mon[33748]: osdmap e329: 8 total, 7 up, 8 in 2024-03-24T12:43:16.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T12:43:16.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:16.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:16.314 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:16.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:16.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:16.443 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:16.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:16.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:16.524 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:43:16.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:16.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:16.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:16.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:16 smithi029 ceph-mon[29919]: pgmap v741: 168 pgs: 2 peering, 7 unknown, 1 active+clean+scrubbing, 22 active+clean+remapped, 101 active+clean, 21 active+undersized, 14 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 651 op/s; 25924/312243 objects degraded (8.303%); 28/312243 objects misplaced (0.009%) 2024-03-24T12:43:16.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:16 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:43:16.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:16 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:43:16.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:16 smithi029 ceph-mon[29919]: osdmap e329: 8 total, 7 up, 8 in 2024-03-24T12:43:16.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:16 smithi029 ceph-mon[41500]: pgmap v741: 168 pgs: 2 peering, 7 unknown, 1 active+clean+scrubbing, 22 active+clean+remapped, 101 active+clean, 21 active+undersized, 14 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 651 op/s; 25924/312243 objects degraded (8.303%); 28/312243 objects misplaced (0.009%) 2024-03-24T12:43:16.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:16 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:43:16.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:16 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:43:16.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:16 smithi029 ceph-mon[41500]: osdmap e329: 8 total, 7 up, 8 in 2024-03-24T12:43:16.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:16.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:16.704 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:16.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:16.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:16.832 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:16.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:16.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:16.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:16.962 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:16.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:16.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:17.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:17.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:17.089 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:17.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:17.190 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:17.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:17.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:17.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:17.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:17.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:17.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:17.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:17.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:17.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:17.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:17.627 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:17.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:17.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:17.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:17.764 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:17.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:17.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:17.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:17.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:17.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:43:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:43:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:43:18.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:18.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:18.026 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:18.026 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:18.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:18.153 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:18.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:18.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:18.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:18.292 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:18.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:18.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:18.351 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: pgmap v744: 168 pgs: 2 peering, 6 unknown, 1 active+clean+scrubbing, 22 active+clean+remapped, 102 active+clean, 21 active+undersized, 14 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 573 op/s; 25924/311562 objects degraded (8.321%); 28/311562 objects misplaced (0.009%) 2024-03-24T12:43:18.351 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 327'9581 2024-03-24T12:43:18.351 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 327'12901 2024-03-24T12:43:18.351 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: osdmap e330: 8 total, 7 up, 8 in 2024-03-24T12:43:18.351 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 327'12047 2024-03-24T12:43:18.351 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 327'9461 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9323 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 327'9673 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9384 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9307 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9450 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 327'12901 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 327'12047 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 327'9461 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 327'9307 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 327'9323 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 327'9384 2024-03-24T12:43:18.352 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 327'9307 2024-03-24T12:43:18.353 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:18.394 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:18.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:18.422 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:18.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:18.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:18.557 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:18.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:18.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:18.629 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:43:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:43:18.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:18.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: pgmap v744: 168 pgs: 2 peering, 6 unknown, 1 active+clean+scrubbing, 22 active+clean+remapped, 102 active+clean, 21 active+undersized, 14 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 573 op/s; 25924/311562 objects degraded (8.321%); 28/311562 objects misplaced (0.009%) 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 327'9581 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 327'12901 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: osdmap e330: 8 total, 7 up, 8 in 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 327'12047 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 327'9461 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9323 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 327'9673 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9384 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9307 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9450 2024-03-24T12:43:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 327'12901 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 327'12047 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 327'9461 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 327'9307 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 327'9323 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 327'9384 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 327'9307 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: pgmap v744: 168 pgs: 2 peering, 6 unknown, 1 active+clean+scrubbing, 22 active+clean+remapped, 102 active+clean, 21 active+undersized, 14 active+undersized+degraded; 6.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 573 op/s; 25924/311562 objects degraded (8.321%); 28/311562 objects misplaced (0.009%) 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 327'9581 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.18 starting backfill to osd.1 from (0'0,0'0] MAX to 327'12901 2024-03-24T12:43:18.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: osdmap e330: 8 total, 7 up, 8 in 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.14 starting backfill to osd.1 from (0'0,0'0] MAX to 327'12047 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.12 starting backfill to osd.2 from (0'0,0'0] MAX to 327'9461 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.13 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9323 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.17 starting backfill to osd.4 from (0'0,0'0] MAX to 327'9673 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9384 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.19 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9307 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.15 starting backfill to osd.1 from (0'0,0'0] MAX to 327'9450 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.18 starting backfill to osd.3 from (0'0,0'0] MAX to 327'12901 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.14 starting backfill to osd.2 from (0'0,0'0] MAX to 327'12047 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.12 starting backfill to osd.6 from (0'0,0'0] MAX to 327'9461 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.19 starting backfill to osd.3 from (0'0,0'0] MAX to 327'9307 2024-03-24T12:43:18.671 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.13 starting backfill to osd.7 from (0'0,0'0] MAX to 327'9323 2024-03-24T12:43:18.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.16 starting backfill to osd.6 from (0'0,0'0] MAX to 327'9384 2024-03-24T12:43:18.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: 10.19 starting backfill to osd.4 from (0'0,0'0] MAX to 327'9307 2024-03-24T12:43:18.672 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:18.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:18.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:18.730 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:43:18.731 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:43:18.801 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:18.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:18.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:18.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:18.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:18.963 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:18.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:19.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:19.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:19.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:19.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:19.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:19.245 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:19.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:19.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:19.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:19.386 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:19.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:19.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:19.521 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:19.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:19.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:19.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:19.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:19 smithi190 ceph-mon[33748]: pgmap v746: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 2 activating+remapped, 1 active+remapped+backfilling, 1 unknown, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 15 active+undersized+degraded; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 31805/297177 objects degraded (10.702%); 34893/297177 objects misplaced (11.741%); 3.2 MiB/s, 50 objects/s recovering 2024-03-24T12:43:19.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:19 smithi190 ceph-mon[33748]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:43:19.658 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:19.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:19.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:19.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:19.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:19 smithi029 ceph-mon[29919]: pgmap v746: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 2 activating+remapped, 1 active+remapped+backfilling, 1 unknown, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 15 active+undersized+degraded; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 31805/297177 objects degraded (10.702%); 34893/297177 objects misplaced (11.741%); 3.2 MiB/s, 50 objects/s recovering 2024-03-24T12:43:19.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:19 smithi029 ceph-mon[29919]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:43:19.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:19.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:19 smithi029 ceph-mon[41500]: pgmap v746: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 2 activating+remapped, 1 active+remapped+backfilling, 1 unknown, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 15 active+undersized+degraded; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 31805/297177 objects degraded (10.702%); 34893/297177 objects misplaced (11.741%); 3.2 MiB/s, 50 objects/s recovering 2024-03-24T12:43:19.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:19 smithi029 ceph-mon[41500]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:43:19.699 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T12:43:19.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:19.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:19.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:19.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:19.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:19.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:19.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:19.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:20.045 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:20.069 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:20.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:20.092 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:43:20.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:20.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:20.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:20.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:20.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:20.338 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:20.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:20.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:20.463 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:20.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:20.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:20 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 31805/297177 objects degraded (10.702%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:20.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:20.604 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:20.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:20.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:20.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:20 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 31805/297177 objects degraded (10.702%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:20.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:20 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 31805/297177 objects degraded (10.702%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:20.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:20.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:20.730 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:20.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:20.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:20.856 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:20.857 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:20.958 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:20.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:20.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:20.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:21.094 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:21.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:21.129 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:21.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:21.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:21.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:21.336 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:21 smithi190 ceph-mon[33748]: pgmap v747: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 2 activating+remapped, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 412 op/s; 31460/304836 objects degraded (10.320%); 50478/304836 objects misplaced (16.559%); 2.4 MiB/s, 39 objects/s recovering 2024-03-24T12:43:21.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:21.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:21.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:21.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:21.514 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:21.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:21.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:21.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:21.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:21 smithi029 ceph-mon[29919]: pgmap v747: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 2 activating+remapped, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 412 op/s; 31460/304836 objects degraded (10.320%); 50478/304836 objects misplaced (16.559%); 2.4 MiB/s, 39 objects/s recovering 2024-03-24T12:43:21.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:21 smithi029 ceph-mon[41500]: pgmap v747: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 activating+undersized+degraded+remapped, 2 active+remapped+backfill_toofull, 2 activating+remapped, 1 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 6.2 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 412 op/s; 31460/304836 objects degraded (10.320%); 50478/304836 objects misplaced (16.559%); 2.4 MiB/s, 39 objects/s recovering 2024-03-24T12:43:21.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:21.676 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:21.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:21.777 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:21.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:21.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:21.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:21.940 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:21.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:22.041 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:22.047 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:43:22.048 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:43:22.082 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:22.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:22.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:22.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:22.210 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:22.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:22.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:22.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:22.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:22.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:22.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:22.494 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:22.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:22.596 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:22.625 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:22.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:22.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:22.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:22.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T12:43:22.753 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:22.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:22.856 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:22.894 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:22.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:22.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:22.996 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:23.034 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:23.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:23.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:23.129 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:43:23.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:23.174 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:23.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:23.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:23.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:23.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:23.419 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:23.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:23.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:23.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:23.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:23.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:23.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:23.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:23.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:23.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:23.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:23.855 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:23.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:23.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:23 smithi190 ceph-mon[33748]: pgmap v748: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 672 B/s rd, 0 B/s wr, 509 op/s; 30824/299928 objects degraded (10.277%); 45786/299928 objects misplaced (15.266%); 26 MiB/s, 410 objects/s recovering 2024-03-24T12:43:23.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:23.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:23.998 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:23.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:24.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:24.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:24.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:24.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:24.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:23 smithi029 ceph-mon[29919]: pgmap v748: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 672 B/s rd, 0 B/s wr, 509 op/s; 30824/299928 objects degraded (10.277%); 45786/299928 objects misplaced (15.266%); 26 MiB/s, 410 objects/s recovering 2024-03-24T12:43:24.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:23 smithi029 ceph-mon[41500]: pgmap v748: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 6.1 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 672 B/s rd, 0 B/s wr, 509 op/s; 30824/299928 objects degraded (10.277%); 45786/299928 objects misplaced (15.266%); 26 MiB/s, 410 objects/s recovering 2024-03-24T12:43:24.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:24.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:24.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:24.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:24.405 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:24.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:24.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:24.552 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:24.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:24.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:24.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:24.679 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:24.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:24.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:24.811 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:24.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:24.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:24.913 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:24.945 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:24.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:24.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:25.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:25.087 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:25.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:25.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:25.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:25.212 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:25.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:25.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:25.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:25.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:25.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:25.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:25.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:25.615 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:25.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:25.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:25.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:25.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:25.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:25.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:25.868 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:25.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:25.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:25 smithi190 ceph-mon[33748]: pgmap v749: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 782 op/s; 28886/291387 objects degraded (9.913%); 44855/291387 objects misplaced (15.394%); 29 MiB/s, 461 objects/s recovering 2024-03-24T12:43:25.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:25 smithi190 ceph-mon[33748]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:43:25.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:25 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 28886/291387 objects degraded (9.913%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:25.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:25 smithi190 ceph-mon[33748]: osdmap e331: 8 total, 7 up, 8 in 2024-03-24T12:43:25.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:26.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:26.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:26.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:26.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:26.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:26.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:26.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:25 smithi029 ceph-mon[41500]: pgmap v749: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 782 op/s; 28886/291387 objects degraded (9.913%); 44855/291387 objects misplaced (15.394%); 29 MiB/s, 461 objects/s recovering 2024-03-24T12:43:26.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:25 smithi029 ceph-mon[41500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:43:26.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:25 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 28886/291387 objects degraded (9.913%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:26.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:25 smithi029 ceph-mon[41500]: osdmap e331: 8 total, 7 up, 8 in 2024-03-24T12:43:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:25 smithi029 ceph-mon[29919]: pgmap v749: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 782 op/s; 28886/291387 objects degraded (9.913%); 44855/291387 objects misplaced (15.394%); 29 MiB/s, 461 objects/s recovering 2024-03-24T12:43:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:25 smithi029 ceph-mon[29919]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:43:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:25 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 28886/291387 objects degraded (9.913%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:25 smithi029 ceph-mon[29919]: osdmap e331: 8 total, 7 up, 8 in 2024-03-24T12:43:26.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:26.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:26.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:26.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:26.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:26.385 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:26.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:26.486 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:26.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:26.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:26.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:26.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:26.662 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:26.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:26.764 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:26.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:26.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:26.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:26.905 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:26.947 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:26.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:26.982 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:43:26.982 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:43:27.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:27.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:27.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:27.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:27.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:27.200 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:27.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:27.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:27.343 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:27.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:27.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:27 smithi190 ceph-mon[33748]: osdmap e332: 8 total, 7 up, 8 in 2024-03-24T12:43:27.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:27.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:27.475 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:27.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:27.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:27.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:27.599 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:27.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:27.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:27 smithi029 ceph-mon[41500]: osdmap e332: 8 total, 7 up, 8 in 2024-03-24T12:43:27.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:27 smithi029 ceph-mon[29919]: osdmap e332: 8 total, 7 up, 8 in 2024-03-24T12:43:27.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:27.743 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:27.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:27.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:27.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:27.876 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:27.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:27.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:43:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:43:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:43:27.920 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T12:43:27.978 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:28.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:28.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:28.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:28.142 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:28.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:28.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:28.244 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:28.247 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:43:28.289 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:28.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:28.391 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:28.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:28.416 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:28.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:28.483 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:28 smithi190 ceph-mon[33748]: pgmap v752: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 1.24k op/s; 28886/291018 objects degraded (9.926%); 44855/291018 objects misplaced (15.413%); 29 MiB/s, 467 objects/s recovering 2024-03-24T12:43:28.483 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:43:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:43:28.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:28.543 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:28.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:28.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:28.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:28 smithi029 ceph-mon[29919]: pgmap v752: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 1.24k op/s; 28886/291018 objects degraded (9.926%); 44855/291018 objects misplaced (15.413%); 29 MiB/s, 467 objects/s recovering 2024-03-24T12:43:28.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:28 smithi029 ceph-mon[41500]: pgmap v752: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 1.24k op/s; 28886/291018 objects degraded (9.926%); 44855/291018 objects misplaced (15.413%); 29 MiB/s, 467 objects/s recovering 2024-03-24T12:43:28.684 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:28.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:28.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:28.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:28.830 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:28.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:28.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:28.955 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:28.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:28.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:29.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:29.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:29.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:29.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:29.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:29.330 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:29.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:29.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:29.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:29.494 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:29.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:29.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:29.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:29.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:29.639 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:29.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:29.740 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:29.776 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:29.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:29.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:29.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:29.908 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:29.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:30.010 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:30.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:30.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:30.185 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:30.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:30.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:30.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:30 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:30.274 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:30 smithi190 ceph-mon[33748]: pgmap v753: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 867 op/s; 27246/284022 objects degraded (9.593%); 41346/284022 objects misplaced (14.557%); 51 MiB/s, 822 objects/s recovering 2024-03-24T12:43:30.275 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:30 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 27246/284022 objects degraded (9.593%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:30.287 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:30.297 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:43:30.297 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:43:30.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:30.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:30.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:30.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:30.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:30.438 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:30.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:30.487 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:30 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:30.488 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:30 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:30.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:30.570 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:30.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:30.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:30.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:30.712 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:30.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:30.713 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:30.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:30.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:30.854 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:30.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:30.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:30 smithi029 ceph-mon[41500]: pgmap v753: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 867 op/s; 27246/284022 objects degraded (9.593%); 41346/284022 objects misplaced (14.557%); 51 MiB/s, 822 objects/s recovering 2024-03-24T12:43:30.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:30 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 27246/284022 objects degraded (9.593%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:30.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:30 smithi029 ceph-mon[29919]: pgmap v753: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 867 op/s; 27246/284022 objects degraded (9.593%); 41346/284022 objects misplaced (14.557%); 51 MiB/s, 822 objects/s recovering 2024-03-24T12:43:30.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:30 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 27246/284022 objects degraded (9.593%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:30.955 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:30.989 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:30.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:31.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:43:31.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:31.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:31.127 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:31.127 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:31.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:31.267 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:31.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:31.368 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:43:31.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:31.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:31.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:31.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:31.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:31.513 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:31 smithi190 ceph-mon[33748]: pgmap v754: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s; 26879/282009 objects degraded (9.531%); 40895/282009 objects misplaced (14.501%); 29 MiB/s, 461 objects/s recovering 2024-03-24T12:43:31.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:31.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:31.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:31.659 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:31.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:31.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:31 smithi029 ceph-mon[29919]: pgmap v754: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s; 26879/282009 objects degraded (9.531%); 40895/282009 objects misplaced (14.501%); 29 MiB/s, 461 objects/s recovering 2024-03-24T12:43:31.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:31 smithi029 ceph-mon[41500]: pgmap v754: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 747 op/s; 26879/282009 objects degraded (9.531%); 40895/282009 objects misplaced (14.501%); 29 MiB/s, 461 objects/s recovering 2024-03-24T12:43:31.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:31.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:31.785 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:31.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:31.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:31.929 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:31.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:31.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:32.031 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:32.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:32.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:32.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:32.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:32.192 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:32.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:32.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:32.321 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:32.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:32.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:32.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:32.468 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:32.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:32.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:32.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:32.606 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:32.606 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:32.707 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:32.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:32.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:32.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:32.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:32.879 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:32.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:32.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:32.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:33.008 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:33.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:33.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:33.149 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:33.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:33.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:33.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:33.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:33.408 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:33.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:33.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:33.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:33.548 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:33.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:33.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:33.615 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:43:33.615 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:43:33.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:33.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:33.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:33.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:33.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:33.815 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:33.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:33.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:33.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:33.964 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:33.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:34.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:34.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:34.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:34.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:34.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:34.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:34.225 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:34.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:34.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:34.367 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:34.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:34.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:34.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:34 smithi029 ceph-mon[29919]: pgmap v755: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 565 op/s; 26222/277812 objects degraded (9.439%); 38606/277812 objects misplaced (13.896%); 30 MiB/s, 473 objects/s recovering 2024-03-24T12:43:34.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:34.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:34 smithi029 ceph-mon[41500]: pgmap v755: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 565 op/s; 26222/277812 objects degraded (9.439%); 38606/277812 objects misplaced (13.896%); 30 MiB/s, 473 objects/s recovering 2024-03-24T12:43:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:34.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:34.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:34.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:34.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:34 smithi190 ceph-mon[33748]: pgmap v755: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 565 op/s; 26222/277812 objects degraded (9.439%); 38606/277812 objects misplaced (13.896%); 30 MiB/s, 473 objects/s recovering 2024-03-24T12:43:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:34.637 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:34.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:34.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:34.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:34.763 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:34.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:34.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:34.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:34.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:34.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:34.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:34.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:35.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:35.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:35.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:35.152 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:35.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:35.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:35.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:35.277 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:35.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:35.278 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:35.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:35.424 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:35.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:35.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:35.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:35.551 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:35.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:35.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:43:35.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:35.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:35.676 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:35.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:35.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:35.816 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:35.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:35.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:35.888 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:43:35.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:35.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:35.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:36.045 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:36.074 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:36.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:36.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:36.150 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:36 smithi029 ceph-mon[29919]: pgmap v756: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 728 op/s; 24630/270012 objects degraded (9.122%); 37730/270012 objects misplaced (13.973%); 29 MiB/s, 459 objects/s recovering 2024-03-24T12:43:36.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:36 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 24630/270012 objects degraded (9.122%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:36.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:36 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:43:36.151 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:36 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:43:36.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:36.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:36.213 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:36.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:36.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:36.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:36 smithi190 ceph-mon[33748]: pgmap v756: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 728 op/s; 24630/270012 objects degraded (9.122%); 37730/270012 objects misplaced (13.973%); 29 MiB/s, 459 objects/s recovering 2024-03-24T12:43:36.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:36 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 24630/270012 objects degraded (9.122%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:36.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:36 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:43:36.326 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:36 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:43:36.339 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:36.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:36.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:36.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:36 smithi029 ceph-mon[41500]: pgmap v756: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 728 op/s; 24630/270012 objects degraded (9.122%); 37730/270012 objects misplaced (13.973%); 29 MiB/s, 459 objects/s recovering 2024-03-24T12:43:36.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:36 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 24630/270012 objects degraded (9.122%), 17 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:36.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:36 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:43:36.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:36 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:43:36.441 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:36.473 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:36.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:36.577 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:36.615 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:36.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:36.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:36.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:36.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:36.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:36.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:36.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:36.875 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:36.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:36.876 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:36.977 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:37.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:37.002 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:37.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:37.103 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:37.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:37.130 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:37.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:37.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:37.256 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:37.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:37.357 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:37.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:37 smithi190 ceph-mon[33748]: osdmap e333: 8 total, 7 up, 8 in 2024-03-24T12:43:37.498 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:37 smithi029 ceph-mon[29919]: osdmap e333: 8 total, 7 up, 8 in 2024-03-24T12:43:37.499 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:37 smithi029 ceph-mon[41500]: osdmap e333: 8 total, 7 up, 8 in 2024-03-24T12:43:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:37.500 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:37.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:37.601 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:37.642 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:37.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:37.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:43:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:43:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:43:37.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:37.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:37.777 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:37.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:37.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:37.909 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:37.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:38.011 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:38.042 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:38.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:38.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:38.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:38.168 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:38.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:38.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:38.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:38.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:38.301 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:43:38.301 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:43:38.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:38 smithi029 ceph-mon[29919]: pgmap v758: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 713 op/s; 24630/269634 objects degraded (9.135%); 37730/269634 objects misplaced (13.993%); 28 MiB/s, 442 objects/s recovering 2024-03-24T12:43:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:38 smithi029 ceph-mon[29919]: osdmap e334: 8 total, 7 up, 8 in 2024-03-24T12:43:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:38 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:43:38.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:38 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:43:38.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:38 smithi029 ceph-mon[41500]: pgmap v758: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 713 op/s; 24630/269634 objects degraded (9.135%); 37730/269634 objects misplaced (13.993%); 28 MiB/s, 442 objects/s recovering 2024-03-24T12:43:38.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:38 smithi029 ceph-mon[41500]: osdmap e334: 8 total, 7 up, 8 in 2024-03-24T12:43:38.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:38 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:43:38.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:38 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:43:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:38.434 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:38.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:38.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:38.581 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:38.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:38.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:38.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:43:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:43:38.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:38 smithi190 ceph-mon[33748]: pgmap v758: 168 pgs: 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 103 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 713 op/s; 24630/269634 objects degraded (9.135%); 37730/269634 objects misplaced (13.993%); 28 MiB/s, 442 objects/s recovering 2024-03-24T12:43:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:38 smithi190 ceph-mon[33748]: osdmap e334: 8 total, 7 up, 8 in 2024-03-24T12:43:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:38 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:43:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:38 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:43:38.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:38.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:38.708 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:38.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:38.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:38.848 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:38.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:38.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:38.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:38.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:38.987 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:38.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:39.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:43:39.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:39.113 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:39.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:39.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:39.241 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:39.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:39.293 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:43:39.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:39.370 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:39.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:39.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:39.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:43:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:39.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:39.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:39.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:39.634 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:39.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:39 smithi029 ceph-mon[41500]: osdmap e335: 8 total, 7 up, 8 in 2024-03-24T12:43:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:39 smithi029 ceph-mon[41500]: pgmap v761: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 912 op/s; 23136/261414 objects degraded (8.850%); 35099/261414 objects misplaced (13.427%); 24 MiB/s, 378 objects/s recovering 2024-03-24T12:43:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:39 smithi029 ceph-mon[29919]: osdmap e335: 8 total, 7 up, 8 in 2024-03-24T12:43:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:39 smithi029 ceph-mon[29919]: pgmap v761: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 912 op/s; 23136/261414 objects degraded (8.850%); 35099/261414 objects misplaced (13.427%); 24 MiB/s, 378 objects/s recovering 2024-03-24T12:43:39.735 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:39.775 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:39.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:39.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:39.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:39 smithi190 ceph-mon[33748]: osdmap e335: 8 total, 7 up, 8 in 2024-03-24T12:43:39.784 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:39 smithi190 ceph-mon[33748]: pgmap v761: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 912 op/s; 23136/261414 objects degraded (8.850%); 35099/261414 objects misplaced (13.427%); 24 MiB/s, 378 objects/s recovering 2024-03-24T12:43:39.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:39.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:39.991 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:43:40.045 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:40.047 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:43:40.047 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:43:40.048 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-03-24T12:43:40.048 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-24T12:43:40.048 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-24T12:43:40.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-03-24T12:43:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:40.094 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:40.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:40.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:40.229 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:40.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:40.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:40.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:40.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:40.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:40.512 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:40.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:40.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:40.546 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:40 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 23136/261414 objects degraded (8.850%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:40.547 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:40 smithi190 ceph-mon[33748]: osdmap e336: 8 total, 7 up, 8 in 2024-03-24T12:43:40.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:40.651 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:40.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:40.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:40.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:40.785 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:40.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:40.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:40.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:40 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 23136/261414 objects degraded (8.850%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:40.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:40 smithi029 ceph-mon[41500]: osdmap e336: 8 total, 7 up, 8 in 2024-03-24T12:43:40.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:40 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 23136/261414 objects degraded (8.850%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:40.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:40 smithi029 ceph-mon[29919]: osdmap e336: 8 total, 7 up, 8 in 2024-03-24T12:43:40.925 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:40.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:41.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:41.066 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:41.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:41.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:41.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:41.191 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:41.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:41.199 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:43:41.199 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:43:41.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:41.320 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:41.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:41.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:41.462 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:41.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:41.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:41.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:41.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:41.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:41.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:41.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:41.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:41.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:41.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:41.855 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:41.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:41.886 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:41 smithi190 ceph-mon[33748]: pgmap v763: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 761 op/s; 22688/259068 objects degraded (8.758%); 34526/259068 objects misplaced (13.327%); 22 MiB/s, 353 objects/s recovering 2024-03-24T12:43:41.886 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:41 smithi029 ceph-mon[29919]: pgmap v763: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 761 op/s; 22688/259068 objects degraded (8.758%); 34526/259068 objects misplaced (13.327%); 22 MiB/s, 353 objects/s recovering 2024-03-24T12:43:41.887 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:41 smithi029 ceph-mon[41500]: pgmap v763: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 761 op/s; 22688/259068 objects degraded (8.758%); 34526/259068 objects misplaced (13.327%); 22 MiB/s, 353 objects/s recovering 2024-03-24T12:43:41.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:43:41.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:42.018 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:42.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:42.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:42.172 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:42.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:42.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:42.298 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:42.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:42.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:42.430 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:42.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:42.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:42.556 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:42.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:42.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:42.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:42.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:42.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:42.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:42.819 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:42.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:42.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:42.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:42.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:42.966 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:42.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:43.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:43.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:43.223 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:43.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:43.257 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:43.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:43.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:43.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:43.382 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:43.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:43.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:43.535 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:43.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:43.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:43.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:43.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:43.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:43.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:43.797 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:43.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:43.899 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:43.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:43.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:44.025 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:44.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:44.068 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:44.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:44.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:43 smithi190 ceph-mon[33748]: pgmap v764: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 862 op/s; 22141/254124 objects degraded (8.713%); 32076/254124 objects misplaced (12.622%); 28 MiB/s, 443 objects/s recovering 2024-03-24T12:43:44.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:43 smithi029 ceph-mon[41500]: pgmap v764: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 862 op/s; 22141/254124 objects degraded (8.713%); 32076/254124 objects misplaced (12.622%); 28 MiB/s, 443 objects/s recovering 2024-03-24T12:43:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:43 smithi029 ceph-mon[29919]: pgmap v764: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 862 op/s; 22141/254124 objects degraded (8.713%); 32076/254124 objects misplaced (12.622%); 28 MiB/s, 443 objects/s recovering 2024-03-24T12:43:44.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:44.197 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:44.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:44.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:44.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:44.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:44.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:44.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:44.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:44.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:44.463 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:44.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:44.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:44.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:44.590 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:44.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:44.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:44.719 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:44.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:44.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:44.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:44.864 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:44.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:44.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:44.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:44.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:44.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:45.073 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:44 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1251924216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:43:45.073 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:44 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1251924216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:43:45.091 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:45.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:45.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:45.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:44 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1251924216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:43:45.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:45.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:45.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:45.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:45.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:45.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:45.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:45.549 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:45.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:45.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:45.651 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:45.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:45.685 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:45.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:45.786 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:45.815 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:45.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:45.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:45.817 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:43:45.817 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:43:45.817 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:43:45.817 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:43:45.817 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:43:45.817 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:43:45.817 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:43:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:43:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:43:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:43:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:43:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:43:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:43:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:43:45.818 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:43:45.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:45.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:46.056 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:46.086 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:46.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:46 smithi029 ceph-mon[29919]: pgmap v765: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.04k op/s; 21146/246033 objects degraded (8.595%); 31135/246033 objects misplaced (12.655%); 22 MiB/s, 350 objects/s recovering 2024-03-24T12:43:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:46 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 21146/246033 objects degraded (8.595%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:46.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:46 smithi029 ceph-mon[41500]: pgmap v765: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.04k op/s; 21146/246033 objects degraded (8.595%); 31135/246033 objects misplaced (12.655%); 22 MiB/s, 350 objects/s recovering 2024-03-24T12:43:46.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:46 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 21146/246033 objects degraded (8.595%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:46.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:46.216 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:46.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:46.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:46.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:46.359 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:46.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:46.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:46 smithi190 ceph-mon[33748]: pgmap v765: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.04k op/s; 21146/246033 objects degraded (8.595%); 31135/246033 objects misplaced (12.655%); 22 MiB/s, 350 objects/s recovering 2024-03-24T12:43:46.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:46 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 21146/246033 objects degraded (8.595%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:46.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:46.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:46.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:46.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:46.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:46.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:46.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:46.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:46.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:46.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:46.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:46.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:46.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:46.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:46.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:46.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:47.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:47.041 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:47.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:47.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:47.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:47.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:47.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:47.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:47.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:43:47.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:43:47.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:43:47.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:47.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:47.314 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:43:47.314 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:43:47.314 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:43:47.314 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:47.314 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:47.342 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:47.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:43:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:47 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:43:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:47 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:43:47.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:47.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:47.444 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:47.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:47.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:47.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:47.633 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:47.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:47.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:47.776 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:47.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:47.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:47.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:47.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:47.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:47.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:43:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:43:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:43:48.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:48.047 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:48.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:48.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:48.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:48.196 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:48.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:48.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:48.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:48.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:48.338 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:48.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:48 smithi190 ceph-mon[33748]: pgmap v766: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 610 B/s rd, 0 B/s wr, 629 op/s; 21146/245607 objects degraded (8.610%); 31135/245607 objects misplaced (12.677%); 7.6 MiB/s, 121 objects/s recovering 2024-03-24T12:43:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:48 smithi190 ceph-mon[33748]: osdmap e337: 8 total, 7 up, 8 in 2024-03-24T12:43:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:48 smithi029 ceph-mon[41500]: pgmap v766: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 610 B/s rd, 0 B/s wr, 629 op/s; 21146/245607 objects degraded (8.610%); 31135/245607 objects misplaced (12.677%); 7.6 MiB/s, 121 objects/s recovering 2024-03-24T12:43:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:48 smithi029 ceph-mon[41500]: osdmap e337: 8 total, 7 up, 8 in 2024-03-24T12:43:48.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:48 smithi029 ceph-mon[29919]: pgmap v766: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 610 B/s rd, 0 B/s wr, 629 op/s; 21146/245607 objects degraded (8.610%); 31135/245607 objects misplaced (12.677%); 7.6 MiB/s, 121 objects/s recovering 2024-03-24T12:43:48.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:48 smithi029 ceph-mon[29919]: osdmap e337: 8 total, 7 up, 8 in 2024-03-24T12:43:48.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:48.478 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:48.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:48.580 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:43:48.601 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:43:48.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:48.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:48.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:48.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:43:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:43:48.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:48.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:48.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:48.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:48.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:48.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:48.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:48.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:49.024 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:49.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:49.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:49.126 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:49.137 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:43:49.148 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:43:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:49.222 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:49.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:49.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:49.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:49.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:49.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:49.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:49.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:49.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:49.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:49.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:49.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:43:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:49.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:49.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:49.531 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:49.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:49.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:49.666 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:49.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:49.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:49.769 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:49.807 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:49.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:49.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:49.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:49.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:50.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:50.087 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:50.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:50.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:50.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:50.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:50.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:50 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:50.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:50 smithi029 ceph-mon[41500]: pgmap v768: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 989 op/s; 20017/233517 objects degraded (8.572%); 28323/233517 objects misplaced (12.129%); 15 MiB/s, 243 objects/s recovering 2024-03-24T12:43:50.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:50 smithi029 ceph-mon[41500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:43:50.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:50 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:50.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:50 smithi029 ceph-mon[29919]: pgmap v768: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 989 op/s; 20017/233517 objects degraded (8.572%); 28323/233517 objects misplaced (12.129%); 15 MiB/s, 243 objects/s recovering 2024-03-24T12:43:50.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:50 smithi029 ceph-mon[29919]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:43:50.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:50.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:50 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:50.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:50 smithi190 ceph-mon[33748]: pgmap v768: 168 pgs: 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 2 active+undersized+degraded+remapped+backfill_toofull, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 989 op/s; 20017/233517 objects degraded (8.572%); 28323/233517 objects misplaced (12.129%); 15 MiB/s, 243 objects/s recovering 2024-03-24T12:43:50.355 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:50 smithi190 ceph-mon[33748]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:43:50.383 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:50.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:50.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:50.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:50.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:50.518 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:50.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:50.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:43:50.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:50.681 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:50.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:50.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:50.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:50.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:50.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:50.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:50.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:51.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:51.084 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:51.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:51.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:51.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:51.221 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:51.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:51.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:51.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:51.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:51.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:51 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 20017/233517 objects degraded (8.572%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:51.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:51 smithi190 ceph-mon[33748]: osdmap e338: 8 total, 7 up, 8 in 2024-03-24T12:43:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:51 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 20017/233517 objects degraded (8.572%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:51 smithi029 ceph-mon[29919]: osdmap e338: 8 total, 7 up, 8 in 2024-03-24T12:43:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:51 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 20017/233517 objects degraded (8.572%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:51 smithi029 ceph-mon[41500]: osdmap e338: 8 total, 7 up, 8 in 2024-03-24T12:43:51.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:51.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:51.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:51.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:51.651 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:51.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:51.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:51.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:51.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:51.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:51.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:52.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:52.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:52.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:52.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:52.234 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:52.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:52.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:52.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:52.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:52.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:52 smithi029 ceph-mon[41500]: pgmap v770: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 19221/230061 objects degraded (8.355%); 27264/230061 objects misplaced (11.851%); 9.7 MiB/s, 154 objects/s recovering 2024-03-24T12:43:52.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:52 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:43:52.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:52 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:43:52.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:52 smithi029 ceph-mon[41500]: osdmap e339: 8 total, 7 up, 8 in 2024-03-24T12:43:52.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:52 smithi029 ceph-mon[29919]: pgmap v770: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 19221/230061 objects degraded (8.355%); 27264/230061 objects misplaced (11.851%); 9.7 MiB/s, 154 objects/s recovering 2024-03-24T12:43:52.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:52 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:43:52.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:52 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:43:52.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:52 smithi029 ceph-mon[29919]: osdmap e339: 8 total, 7 up, 8 in 2024-03-24T12:43:52.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:52 smithi190 ceph-mon[33748]: pgmap v770: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 3 active+remapped+backfilling, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.00k op/s; 19221/230061 objects degraded (8.355%); 27264/230061 objects misplaced (11.851%); 9.7 MiB/s, 154 objects/s recovering 2024-03-24T12:43:52.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:52 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:43:52.429 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:52 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:43:52.430 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:52 smithi190 ceph-mon[33748]: osdmap e339: 8 total, 7 up, 8 in 2024-03-24T12:43:52.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:52.508 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:52.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:52.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:52.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:52.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:52.634 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:52.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:52.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:52.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:52.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:52.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:52.866 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:52.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:52.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:53.003 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:53.041 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:53.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:53.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:53.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:53.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:53.206 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:43:53.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:53.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:53.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:53.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:53.453 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:53.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:53.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:53.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:53.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:53.578 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:53.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:53 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3054216961' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:43:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:53 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:43:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:53 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:53 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:53 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3054216961' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:43:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:53 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:43:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:53 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:53 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:53.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:53 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3054216961' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:43:53.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:53 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:43:53.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:53 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:53.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:53 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:53.679 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:53.719 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:53.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:53.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:53.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:53.862 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:53.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:53.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:53.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:54.020 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:54.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:54.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:54.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:54.165 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:54.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:54.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:54.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:54.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[29919]: pgmap v772: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.20k op/s; 19063/224010 objects degraded (8.510%); 25567/224010 objects misplaced (11.413%); 19 MiB/s, 307 objects/s recovering 2024-03-24T12:43:54.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:43:54.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:43:54.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[29919]: osdmap e340: 8 total, 7 up, 8 in 2024-03-24T12:43:54.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:43:54.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:43:54.310 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:54.310 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[41500]: pgmap v772: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.20k op/s; 19063/224010 objects degraded (8.510%); 25567/224010 objects misplaced (11.413%); 19 MiB/s, 307 objects/s recovering 2024-03-24T12:43:54.311 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:43:54.311 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:43:54.311 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[41500]: osdmap e340: 8 total, 7 up, 8 in 2024-03-24T12:43:54.311 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:43:54.311 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:43:54.311 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:54 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:54.314 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:54.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:54.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:54.415 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:54.440 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:54.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:54.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:54.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:54.588 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:54.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:54 smithi190 ceph-mon[33748]: pgmap v772: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 23 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.5 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.20k op/s; 19063/224010 objects degraded (8.510%); 25567/224010 objects misplaced (11.413%); 19 MiB/s, 307 objects/s recovering 2024-03-24T12:43:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:54 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:43:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:54 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:43:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:54 smithi190 ceph-mon[33748]: osdmap e340: 8 total, 7 up, 8 in 2024-03-24T12:43:54.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:54 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:43:54.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:54 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:43:54.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:54 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:43:54.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:54.725 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:54.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:54.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:43:54.857 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:54.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:54.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:43:54.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:54.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:54.996 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:54.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:55.097 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:55.137 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:55.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:55.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:55.262 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:55.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:55.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:55.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:55.402 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:55.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:55.505 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:55.537 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:55.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:55 smithi190 ceph-mon[33748]: pgmap v774: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 24 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 993 op/s; 17104/215637 objects degraded (7.932%); 22437/215637 objects misplaced (10.405%); 28 MiB/s, 453 objects/s recovering 2024-03-24T12:43:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:55 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 19063/224010 objects degraded (8.510%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:55.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:43:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:55.663 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:55.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:43:55.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:55 smithi029 ceph-mon[41500]: pgmap v774: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 24 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 993 op/s; 17104/215637 objects degraded (7.932%); 22437/215637 objects misplaced (10.405%); 28 MiB/s, 453 objects/s recovering 2024-03-24T12:43:55.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:55 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 19063/224010 objects degraded (8.510%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:55.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:55 smithi029 ceph-mon[29919]: pgmap v774: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 24 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 993 op/s; 17104/215637 objects degraded (7.932%); 22437/215637 objects misplaced (10.405%); 28 MiB/s, 453 objects/s recovering 2024-03-24T12:43:55.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:55 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 19063/224010 objects degraded (8.510%), 16 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:43:55.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:55.789 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:55.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:55.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:55.933 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:55.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:56.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:56.055 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:43:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-24T12:43:56.056 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:43:56.066 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:56.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:56.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:56.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:56.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:56.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:56.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:56.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:56.319 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:56.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:56.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:56 smithi029 ceph-mon[29919]: 10.17 scrub starts 2024-03-24T12:43:56.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:56 smithi029 ceph-mon[29919]: 10.17 scrub ok 2024-03-24T12:43:56.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:56 smithi029 ceph-mon[41500]: 10.17 scrub starts 2024-03-24T12:43:56.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:56 smithi029 ceph-mon[41500]: 10.17 scrub ok 2024-03-24T12:43:56.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:56.449 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:56.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:56.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:56.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:56.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:56.585 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:56.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:56 smithi190 ceph-mon[33748]: 10.17 scrub starts 2024-03-24T12:43:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:56 smithi190 ceph-mon[33748]: 10.17 scrub ok 2024-03-24T12:43:56.687 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:56.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:56.711 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:56.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:56.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:56.842 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:56.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:56.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:56.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:56.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:56.972 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:56.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:57.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:57.099 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:57.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:57.201 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:57.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:57.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:57.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:57.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:57.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:57.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:57.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:57.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:57.487 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:57.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:57.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:57.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[41500]: osdmap e341: 8 total, 7 up, 8 in 2024-03-24T12:43:57.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[41500]: 10.10 scrub starts 2024-03-24T12:43:57.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[41500]: 10.16 scrub starts 2024-03-24T12:43:57.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[41500]: 10.10 scrub ok 2024-03-24T12:43:57.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[41500]: pgmap v776: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 24 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 823 op/s; 17104/215244 objects degraded (7.946%); 22437/215244 objects misplaced (10.424%); 27 MiB/s, 425 objects/s recovering 2024-03-24T12:43:57.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[41500]: 10.8 scrub starts 2024-03-24T12:43:57.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[29919]: osdmap e341: 8 total, 7 up, 8 in 2024-03-24T12:43:57.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[29919]: 10.10 scrub starts 2024-03-24T12:43:57.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[29919]: 10.16 scrub starts 2024-03-24T12:43:57.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[29919]: 10.10 scrub ok 2024-03-24T12:43:57.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[29919]: pgmap v776: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 24 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 823 op/s; 17104/215244 objects degraded (7.946%); 22437/215244 objects misplaced (10.424%); 27 MiB/s, 425 objects/s recovering 2024-03-24T12:43:57.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-mon[29919]: 10.8 scrub starts 2024-03-24T12:43:57.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:57.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:57 smithi190 ceph-mon[33748]: osdmap e341: 8 total, 7 up, 8 in 2024-03-24T12:43:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:57 smithi190 ceph-mon[33748]: 10.10 scrub starts 2024-03-24T12:43:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:57 smithi190 ceph-mon[33748]: 10.16 scrub starts 2024-03-24T12:43:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:57 smithi190 ceph-mon[33748]: 10.10 scrub ok 2024-03-24T12:43:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:57 smithi190 ceph-mon[33748]: pgmap v776: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfilling, 24 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 823 op/s; 17104/215244 objects degraded (7.946%); 22437/215244 objects misplaced (10.424%); 27 MiB/s, 425 objects/s recovering 2024-03-24T12:43:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:57 smithi190 ceph-mon[33748]: 10.8 scrub starts 2024-03-24T12:43:57.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:57.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:57.764 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:57.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:57.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:57.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:57.891 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:57.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:57.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:43:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:43:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:43:57.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:58.033 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:58.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:58.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:58.135 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:58.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:58.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:58.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:58.273 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:58.304 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:58.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:58.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:58.430 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:58.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:58.531 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:43:58.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:58.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:43:58.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:43:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:43:58.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:58 smithi190 ceph-mon[33748]: 10.16 scrub ok 2024-03-24T12:43:58.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:58 smithi190 ceph-mon[33748]: osdmap e342: 8 total, 7 up, 8 in 2024-03-24T12:43:58.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:58 smithi190 ceph-mon[33748]: 10.8 scrub ok 2024-03-24T12:43:58.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:58 smithi190 ceph-mon[33748]: 10.1 scrub starts 2024-03-24T12:43:58.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:58 smithi190 ceph-mon[33748]: 10.12 scrub starts 2024-03-24T12:43:58.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:58 smithi190 ceph-mon[33748]: 10.4 scrub starts 2024-03-24T12:43:58.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:58 smithi190 ceph-mon[33748]: 10.1 scrub ok 2024-03-24T12:43:58.658 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:58.673 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[41500]: 10.16 scrub ok 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[41500]: osdmap e342: 8 total, 7 up, 8 in 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[41500]: 10.8 scrub ok 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[41500]: 10.1 scrub starts 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[41500]: 10.12 scrub starts 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[41500]: 10.4 scrub starts 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[41500]: 10.1 scrub ok 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[29919]: 10.16 scrub ok 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[29919]: osdmap e342: 8 total, 7 up, 8 in 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[29919]: 10.8 scrub ok 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[29919]: 10.1 scrub starts 2024-03-24T12:43:58.674 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[29919]: 10.12 scrub starts 2024-03-24T12:43:58.675 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[29919]: 10.4 scrub starts 2024-03-24T12:43:58.675 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:58 smithi029 ceph-mon[29919]: 10.1 scrub ok 2024-03-24T12:43:58.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:58.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:58.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:43:58.822 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:58.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:58.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:43:58.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:58.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:58.954 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:58.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:59.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:43:59.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:59.105 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:59.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:43:59.206 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:59.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:59.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:59.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:59.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:59.377 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:59.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:59.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:59.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:59.510 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:59.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:59.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:43:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:59 smithi190 ceph-mon[33748]: 10.12 scrub ok 2024-03-24T12:43:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:59 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:43:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:59 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:43:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:59 smithi190 ceph-mon[33748]: pgmap v778: 168 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 25 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 16647/206043 objects degraded (8.079%); 20291/206043 objects misplaced (9.848%) 2024-03-24T12:43:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:59 smithi190 ceph-mon[33748]: 3.1b scrub starts 2024-03-24T12:43:59.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:43:59 smithi190 ceph-mon[33748]: 3.1b scrub ok 2024-03-24T12:43:59.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:43:59.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:59.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:43:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[29919]: 10.12 scrub ok 2024-03-24T12:43:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:43:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:43:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[29919]: pgmap v778: 168 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 25 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 16647/206043 objects degraded (8.079%); 20291/206043 objects misplaced (9.848%) 2024-03-24T12:43:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[29919]: 3.1b scrub starts 2024-03-24T12:43:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[29919]: 3.1b scrub ok 2024-03-24T12:43:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[41500]: 10.12 scrub ok 2024-03-24T12:43:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:43:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:43:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:43:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[41500]: pgmap v778: 168 pgs: 1 active+remapped+backfilling, 2 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 25 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.2 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 16647/206043 objects degraded (8.079%); 20291/206043 objects misplaced (9.848%) 2024-03-24T12:43:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[41500]: 3.1b scrub starts 2024-03-24T12:43:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:43:59 smithi029 ceph-mon[41500]: 3.1b scrub ok 2024-03-24T12:43:59.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:59.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:59.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:59.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:43:59.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:43:59.896 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:43:59.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:43:59.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:00.028 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:00.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:00.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:00.156 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:00.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:00.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:00.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:00.303 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:00.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:00.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:00.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:00.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[41500]: 10.15 scrub starts 2024-03-24T12:44:00.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[29919]: 10.15 scrub starts 2024-03-24T12:44:00.428 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:00.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:00.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:00.530 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:00.566 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:00.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:00.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:00 smithi190 ceph-mon[33748]: 10.15 scrub starts 2024-03-24T12:44:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:00 smithi190 ceph-mon[33748]: 10.4 scrub ok 2024-03-24T12:44:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:00 smithi190 ceph-mon[33748]: osdmap e343: 8 total, 7 up, 8 in 2024-03-24T12:44:00.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:00 smithi190 ceph-mon[33748]: 10.0 scrub starts 2024-03-24T12:44:00.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:00 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 16647/206043 objects degraded (8.079%), 15 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:00.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:00 smithi190 ceph-mon[33748]: osdmap e344: 8 total, 7 up, 8 in 2024-03-24T12:44:00.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:00.718 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:00.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:00.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:00.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:00.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:00.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[29919]: 10.4 scrub ok 2024-03-24T12:44:00.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[29919]: osdmap e343: 8 total, 7 up, 8 in 2024-03-24T12:44:00.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[29919]: 10.0 scrub starts 2024-03-24T12:44:00.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 16647/206043 objects degraded (8.079%), 15 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:00.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[29919]: osdmap e344: 8 total, 7 up, 8 in 2024-03-24T12:44:00.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[41500]: 10.4 scrub ok 2024-03-24T12:44:00.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[41500]: osdmap e343: 8 total, 7 up, 8 in 2024-03-24T12:44:00.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[41500]: 10.0 scrub starts 2024-03-24T12:44:00.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 16647/206043 objects degraded (8.079%), 15 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:00.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:00 smithi029 ceph-mon[41500]: osdmap e344: 8 total, 7 up, 8 in 2024-03-24T12:44:00.946 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:00.971 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:00.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:01.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:01.099 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:01.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:01.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:01.245 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:01.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:01.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:01.388 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:01.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:01.490 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:01.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:01.536 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:01.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:01 smithi190 ceph-mon[33748]: 10.0 scrub ok 2024-03-24T12:44:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:01 smithi190 ceph-mon[33748]: pgmap v781: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 25 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 919 B/s rd, 0 op/s; 15593/203643 objects degraded (7.657%); 19147/203643 objects misplaced (9.402%) 2024-03-24T12:44:01.637 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:01.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:01.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:01.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:01.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:01 smithi029 ceph-mon[29919]: 10.0 scrub ok 2024-03-24T12:44:01.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:01 smithi029 ceph-mon[29919]: pgmap v781: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 25 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 919 B/s rd, 0 op/s; 15593/203643 objects degraded (7.657%); 19147/203643 objects misplaced (9.402%) 2024-03-24T12:44:01.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:01 smithi029 ceph-mon[41500]: 10.0 scrub ok 2024-03-24T12:44:01.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:01 smithi029 ceph-mon[41500]: pgmap v781: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 25 active+clean+remapped, 102 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.1 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 919 B/s rd, 0 op/s; 15593/203643 objects degraded (7.657%); 19147/203643 objects misplaced (9.402%) 2024-03-24T12:44:01.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:01.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:01.792 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:01.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:01.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:01.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:02.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:02.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:02.078 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:02.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:02.179 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:02.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:02.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:02.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:02.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:02.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:02.337 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:02.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:02.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:02.472 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:02.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:02.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:02.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:02.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:02.598 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:02.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:02.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:02 smithi029 ceph-mon[41500]: 10.15 scrub starts 2024-03-24T12:44:02.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:02 smithi029 ceph-mon[41500]: 10.11 scrub starts 2024-03-24T12:44:02.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:02 smithi029 ceph-mon[41500]: 10.d scrub starts 2024-03-24T12:44:02.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:02 smithi029 ceph-mon[41500]: 10.11 scrub ok 2024-03-24T12:44:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:02 smithi029 ceph-mon[29919]: 10.15 scrub starts 2024-03-24T12:44:02.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:02 smithi029 ceph-mon[29919]: 10.11 scrub starts 2024-03-24T12:44:02.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:02 smithi029 ceph-mon[29919]: 10.d scrub starts 2024-03-24T12:44:02.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:02 smithi029 ceph-mon[29919]: 10.11 scrub ok 2024-03-24T12:44:02.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:02.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:02.726 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:02.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:02.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:02.850 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:02.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:02.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:02.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:02 smithi190 ceph-mon[33748]: 10.15 scrub starts 2024-03-24T12:44:02.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:02 smithi190 ceph-mon[33748]: 10.11 scrub starts 2024-03-24T12:44:02.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:02 smithi190 ceph-mon[33748]: 10.d scrub starts 2024-03-24T12:44:02.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:02 smithi190 ceph-mon[33748]: 10.11 scrub ok 2024-03-24T12:44:02.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:02.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:02.979 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:02.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:03.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:03.104 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:03.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:03.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:03.206 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:03.233 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:03.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:03.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:03.363 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:03.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:03.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:03.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:03.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:03.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:03.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:03.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:03.630 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:03.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:03 smithi029 ceph-mon[29919]: 10.15 scrub ok 2024-03-24T12:44:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:03 smithi029 ceph-mon[29919]: 10.d scrub ok 2024-03-24T12:44:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:03 smithi029 ceph-mon[29919]: pgmap v782: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 25 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 15472/199491 objects degraded (7.756%); 18913/199491 objects misplaced (9.481%) 2024-03-24T12:44:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:03 smithi029 ceph-mon[41500]: 10.15 scrub ok 2024-03-24T12:44:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:03 smithi029 ceph-mon[41500]: 10.d scrub ok 2024-03-24T12:44:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:03 smithi029 ceph-mon[41500]: pgmap v782: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 25 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 15472/199491 objects degraded (7.756%); 18913/199491 objects misplaced (9.481%) 2024-03-24T12:44:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:03.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:03.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:03.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:03.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:03.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:03.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:03 smithi190 ceph-mon[33748]: 10.15 scrub ok 2024-03-24T12:44:03.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:03 smithi190 ceph-mon[33748]: 10.d scrub ok 2024-03-24T12:44:03.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:03 smithi190 ceph-mon[33748]: pgmap v782: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 1 active+remapped+backfilling, 2 active+remapped+backfill_toofull, 25 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 4.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 15472/199491 objects degraded (7.756%); 18913/199491 objects misplaced (9.481%) 2024-03-24T12:44:03.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:03.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:03.883 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:03.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:03.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:04.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:04.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:04.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:04.157 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:04.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:04.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:04.259 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:04.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:04.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:04.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:04.395 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:04.428 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:04.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:04.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:04.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:04.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:04.574 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:04.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:04 smithi029 ceph-mon[29919]: 6.18 scrub starts 2024-03-24T12:44:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:04 smithi029 ceph-mon[29919]: 6.18 scrub ok 2024-03-24T12:44:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:04 smithi029 ceph-mon[41500]: 6.18 scrub starts 2024-03-24T12:44:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:04 smithi029 ceph-mon[41500]: 6.18 scrub ok 2024-03-24T12:44:04.676 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:04.707 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:04.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:04.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:04.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:04.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:04.839 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:04.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:04.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:04 smithi190 ceph-mon[33748]: 6.18 scrub starts 2024-03-24T12:44:04.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:04 smithi190 ceph-mon[33748]: 6.18 scrub ok 2024-03-24T12:44:04.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:04.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:04.982 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:04.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:05.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:05.116 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:05.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:05.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:05.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:05.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:05.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:05.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:05.371 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:05.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:05.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:05.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:05.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:05.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:05.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:05.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:05.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:05.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:05 smithi029 ceph-mon[29919]: pgmap v783: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 14362/193800 objects degraded (7.411%); 17451/193800 objects misplaced (9.005%) 2024-03-24T12:44:05.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:05 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 15472/199491 objects degraded (7.756%), 15 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:05.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:05 smithi029 ceph-mon[41500]: pgmap v783: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 14362/193800 objects degraded (7.411%); 17451/193800 objects misplaced (9.005%) 2024-03-24T12:44:05.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:05 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 15472/199491 objects degraded (7.756%), 15 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:05.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:05.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:05.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:05.877 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:05.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:05 smithi190 ceph-mon[33748]: pgmap v783: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 14362/193800 objects degraded (7.411%); 17451/193800 objects misplaced (9.005%) 2024-03-24T12:44:05.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:05 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 15472/199491 objects degraded (7.756%), 15 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:05.918 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:05.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:05.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:06.020 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:06.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:06.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:06.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:06.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:06.197 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:06.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:06.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:06.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:06.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:06.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:06.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:06.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:06.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:06.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:06.588 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:06.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:06.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:06 smithi029 ceph-mon[29919]: 3.b scrub starts 2024-03-24T12:44:06.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:06 smithi029 ceph-mon[29919]: 3.b scrub ok 2024-03-24T12:44:06.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:06 smithi029 ceph-mon[29919]: 10.c scrub starts 2024-03-24T12:44:06.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:06 smithi029 ceph-mon[41500]: 3.b scrub starts 2024-03-24T12:44:06.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:06 smithi029 ceph-mon[41500]: 3.b scrub ok 2024-03-24T12:44:06.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:06 smithi029 ceph-mon[41500]: 10.c scrub starts 2024-03-24T12:44:06.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:06.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:06.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:06.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:06.861 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:06.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:06.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:06.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:06 smithi190 ceph-mon[33748]: 3.b scrub starts 2024-03-24T12:44:06.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:06 smithi190 ceph-mon[33748]: 3.b scrub ok 2024-03-24T12:44:06.885 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:06 smithi190 ceph-mon[33748]: 10.c scrub starts 2024-03-24T12:44:06.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:06.996 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:06.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:06.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:07.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:07.124 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:07.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:07.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:07.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:07.249 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:07.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:07.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:07.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:07.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:07.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:07.505 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:07.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:07.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:07.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:07.639 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:07.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:07.668 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:44:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:44:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:44:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:44:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[29919]: osdmap e345: 8 total, 7 up, 8 in 2024-03-24T12:44:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[29919]: 3.9 scrub starts 2024-03-24T12:44:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[29919]: 3.9 scrub ok 2024-03-24T12:44:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:44:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[29919]: 3.13 scrub ok 2024-03-24T12:44:07.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[29919]: pgmap v785: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 688 B/s rd, 0 op/s; 14362/193509 objects degraded (7.422%); 17451/193509 objects misplaced (9.018%) 2024-03-24T12:44:07.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:44:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:44:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[41500]: osdmap e345: 8 total, 7 up, 8 in 2024-03-24T12:44:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[41500]: 3.9 scrub starts 2024-03-24T12:44:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[41500]: 3.9 scrub ok 2024-03-24T12:44:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:44:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[41500]: 3.13 scrub ok 2024-03-24T12:44:07.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:07 smithi029 ceph-mon[41500]: pgmap v785: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 688 B/s rd, 0 op/s; 14362/193509 objects degraded (7.422%); 17451/193509 objects misplaced (9.018%) 2024-03-24T12:44:07.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:07.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:07.777 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:07.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:07.879 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:07.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:07 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:44:07.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:07 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:44:07.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:07 smithi190 ceph-mon[33748]: osdmap e345: 8 total, 7 up, 8 in 2024-03-24T12:44:07.891 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:07 smithi190 ceph-mon[33748]: 3.9 scrub starts 2024-03-24T12:44:07.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:07 smithi190 ceph-mon[33748]: 3.9 scrub ok 2024-03-24T12:44:07.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:07 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:44:07.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:07 smithi190 ceph-mon[33748]: 3.13 scrub ok 2024-03-24T12:44:07.892 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:07 smithi190 ceph-mon[33748]: pgmap v785: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 688 B/s rd, 0 op/s; 14362/193509 objects degraded (7.422%); 17451/193509 objects misplaced (9.018%) 2024-03-24T12:44:07.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:07.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:07.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:08.016 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:08.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:08.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:08.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:08.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:08.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:08.175 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:08.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:08.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:08.310 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:08.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:08.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:08.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:08.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:08.446 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:08.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:08.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:08.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:08.572 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:08.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:44:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:44:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[41500]: 10.c scrub ok 2024-03-24T12:44:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[41500]: osdmap e346: 8 total, 7 up, 8 in 2024-03-24T12:44:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[41500]: 1.7 scrub starts 2024-03-24T12:44:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[41500]: 1.7 scrub ok 2024-03-24T12:44:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[41500]: 10.13 scrub starts 2024-03-24T12:44:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:44:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:44:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[29919]: 10.c scrub ok 2024-03-24T12:44:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[29919]: osdmap e346: 8 total, 7 up, 8 in 2024-03-24T12:44:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[29919]: 1.7 scrub starts 2024-03-24T12:44:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[29919]: 1.7 scrub ok 2024-03-24T12:44:08.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:08 smithi029 ceph-mon[29919]: 10.13 scrub starts 2024-03-24T12:44:08.673 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:08.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:08.709 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:08.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:08.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:08.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:08.844 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:08.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:08.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:44:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:44:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:08 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:44:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:08 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:44:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:08 smithi190 ceph-mon[33748]: 10.c scrub ok 2024-03-24T12:44:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:08 smithi190 ceph-mon[33748]: osdmap e346: 8 total, 7 up, 8 in 2024-03-24T12:44:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:08 smithi190 ceph-mon[33748]: 1.7 scrub starts 2024-03-24T12:44:08.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:08 smithi190 ceph-mon[33748]: 1.7 scrub ok 2024-03-24T12:44:08.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:08 smithi190 ceph-mon[33748]: 10.13 scrub starts 2024-03-24T12:44:08.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:08.984 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:08.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:08.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:09.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:09.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:09.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:09.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:09.274 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:09.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:09.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:09.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:09.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:09.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:09.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:09.550 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:09.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:09.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:09.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:09.692 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:09.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:09.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:09.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:09.838 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:09.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:09.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:09.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:09 smithi190 ceph-mon[33748]: 3.7 scrub starts 2024-03-24T12:44:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:09 smithi190 ceph-mon[33748]: 3.7 scrub ok 2024-03-24T12:44:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:09 smithi190 ceph-mon[33748]: 10.13 scrub ok 2024-03-24T12:44:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:09 smithi190 ceph-mon[33748]: 6.3 scrub starts 2024-03-24T12:44:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:09 smithi190 ceph-mon[33748]: pgmap v787: 168 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13599/182421 objects degraded (7.455%); 16586/182421 objects misplaced (9.092%) 2024-03-24T12:44:09.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:09 smithi190 ceph-mon[33748]: 6.3 scrub ok 2024-03-24T12:44:09.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[29919]: 3.7 scrub starts 2024-03-24T12:44:09.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[29919]: 3.7 scrub ok 2024-03-24T12:44:09.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[29919]: 10.13 scrub ok 2024-03-24T12:44:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[29919]: 6.3 scrub starts 2024-03-24T12:44:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[29919]: pgmap v787: 168 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13599/182421 objects degraded (7.455%); 16586/182421 objects misplaced (9.092%) 2024-03-24T12:44:09.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[29919]: 6.3 scrub ok 2024-03-24T12:44:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[41500]: 3.7 scrub starts 2024-03-24T12:44:09.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[41500]: 3.7 scrub ok 2024-03-24T12:44:09.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[41500]: 10.13 scrub ok 2024-03-24T12:44:09.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:09.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[41500]: 6.3 scrub starts 2024-03-24T12:44:09.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[41500]: pgmap v787: 168 pgs: 1 active+clean+scrubbing, 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13599/182421 objects degraded (7.455%); 16586/182421 objects misplaced (9.092%) 2024-03-24T12:44:09.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:09 smithi029 ceph-mon[41500]: 6.3 scrub ok 2024-03-24T12:44:09.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:09.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:09.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:09.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:10.074 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:10.120 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:10.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:10.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:10.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:10.246 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:10.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:10.351 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:10.381 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:10.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:10.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:10.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:10.522 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:10.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:10.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:10.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:10.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:10.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:10.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:10.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:10.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:10.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:10 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:44:10.843 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:10 smithi029 ceph-mon[29919]: 4.4 scrub ok 2024-03-24T12:44:10.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:10 smithi029 ceph-mon[29919]: 1.9 scrub starts 2024-03-24T12:44:10.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:10 smithi029 ceph-mon[29919]: 1.9 scrub ok 2024-03-24T12:44:10.844 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:10 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 13599/182421 objects degraded (7.455%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:10.844 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:10 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:44:10.844 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:10 smithi029 ceph-mon[41500]: 4.4 scrub ok 2024-03-24T12:44:10.845 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:10 smithi029 ceph-mon[41500]: 1.9 scrub starts 2024-03-24T12:44:10.845 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:10 smithi029 ceph-mon[41500]: 1.9 scrub ok 2024-03-24T12:44:10.845 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:10 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 13599/182421 objects degraded (7.455%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:10 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:44:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:10 smithi190 ceph-mon[33748]: 4.4 scrub ok 2024-03-24T12:44:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:10 smithi190 ceph-mon[33748]: 1.9 scrub starts 2024-03-24T12:44:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:10 smithi190 ceph-mon[33748]: 1.9 scrub ok 2024-03-24T12:44:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:10 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 13599/182421 objects degraded (7.455%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:10.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:10.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:10.927 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:10.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:11.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:11.069 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:11.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:11.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:11.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:11.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:11.205 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:11.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:11.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:11.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:11.349 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:11.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:11.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:11.475 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:11.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:11.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:11.577 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:11.617 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:11.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:11.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:11.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:11.751 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:11.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:11.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:11 smithi190 ceph-mon[33748]: 10.5 scrub starts 2024-03-24T12:44:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:11 smithi190 ceph-mon[33748]: 6.b scrub starts 2024-03-24T12:44:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:11 smithi190 ceph-mon[33748]: 6.b scrub ok 2024-03-24T12:44:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:11 smithi190 ceph-mon[33748]: 1.1 scrub starts 2024-03-24T12:44:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:11 smithi190 ceph-mon[33748]: 1.1 scrub ok 2024-03-24T12:44:11.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:11 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:44:11.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:11 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:44:11.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:11 smithi190 ceph-mon[33748]: pgmap v788: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 13141/179634 objects degraded (7.315%); 15944/179634 objects misplaced (8.876%) 2024-03-24T12:44:11.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:11.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:11.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[41500]: 10.5 scrub starts 2024-03-24T12:44:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[41500]: 6.b scrub starts 2024-03-24T12:44:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[41500]: 6.b scrub ok 2024-03-24T12:44:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[41500]: 1.1 scrub starts 2024-03-24T12:44:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[41500]: 1.1 scrub ok 2024-03-24T12:44:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:44:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:44:11.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[41500]: pgmap v788: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 13141/179634 objects degraded (7.315%); 15944/179634 objects misplaced (8.876%) 2024-03-24T12:44:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[29919]: 10.5 scrub starts 2024-03-24T12:44:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[29919]: 6.b scrub starts 2024-03-24T12:44:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[29919]: 6.b scrub ok 2024-03-24T12:44:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[29919]: 1.1 scrub starts 2024-03-24T12:44:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[29919]: 1.1 scrub ok 2024-03-24T12:44:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:44:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:44:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:11 smithi029 ceph-mon[29919]: pgmap v788: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 104 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.6 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 13141/179634 objects degraded (7.315%); 15944/179634 objects misplaced (8.876%) 2024-03-24T12:44:11.990 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:12.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:12.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:12.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:12.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:12.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:12.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:12.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:12.297 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:12.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:12.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:12.425 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:12.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:12.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:12.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:12.565 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:12.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:12.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:12.707 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:12.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:12.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:12.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:12.846 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:12.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:12.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:12 smithi190 ceph-mon[33748]: 10.5 scrub ok 2024-03-24T12:44:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:12 smithi190 ceph-mon[33748]: 10.2 scrub starts 2024-03-24T12:44:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:12 smithi190 ceph-mon[33748]: 10.2 scrub ok 2024-03-24T12:44:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:12 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:44:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:12 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:44:12.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:12 smithi190 ceph-mon[33748]: 5.1 scrub starts 2024-03-24T12:44:12.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:12 smithi190 ceph-mon[33748]: 5.1 scrub ok 2024-03-24T12:44:12.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:12 smithi190 ceph-mon[33748]: 1.6 scrub starts 2024-03-24T12:44:12.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:12 smithi190 ceph-mon[33748]: 1.6 scrub ok 2024-03-24T12:44:12.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[41500]: 10.5 scrub ok 2024-03-24T12:44:12.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[41500]: 10.2 scrub starts 2024-03-24T12:44:12.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[41500]: 10.2 scrub ok 2024-03-24T12:44:12.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:44:12.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:44:12.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[41500]: 5.1 scrub starts 2024-03-24T12:44:12.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[41500]: 5.1 scrub ok 2024-03-24T12:44:12.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[41500]: 1.6 scrub starts 2024-03-24T12:44:12.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[41500]: 1.6 scrub ok 2024-03-24T12:44:12.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[29919]: 10.5 scrub ok 2024-03-24T12:44:12.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[29919]: 10.2 scrub starts 2024-03-24T12:44:12.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[29919]: 10.2 scrub ok 2024-03-24T12:44:12.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:44:12.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:44:12.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[29919]: 5.1 scrub starts 2024-03-24T12:44:12.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[29919]: 5.1 scrub ok 2024-03-24T12:44:12.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[29919]: 1.6 scrub starts 2024-03-24T12:44:12.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:12 smithi029 ceph-mon[29919]: 1.6 scrub ok 2024-03-24T12:44:12.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:12.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:12.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:12.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:13.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:13.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:13.116 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:13.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:13.219 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:13.244 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:13.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:13.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:13.381 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:13.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:13.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:13.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:13.509 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:13.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:13.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:13.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:13.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:13.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:13.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:13.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:13.779 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:13.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:13.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:13 smithi190 ceph-mon[33748]: 3.2 scrub starts 2024-03-24T12:44:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:13 smithi190 ceph-mon[33748]: 3.2 scrub ok 2024-03-24T12:44:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:13 smithi190 ceph-mon[33748]: pgmap v789: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 126 op/s; 12955/172740 objects degraded (7.500%); 15643/172740 objects misplaced (9.056%) 2024-03-24T12:44:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:13.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:13.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:13.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:13 smithi029 ceph-mon[29919]: 3.2 scrub starts 2024-03-24T12:44:13.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:13 smithi029 ceph-mon[29919]: 3.2 scrub ok 2024-03-24T12:44:13.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:13 smithi029 ceph-mon[29919]: pgmap v789: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 126 op/s; 12955/172740 objects degraded (7.500%); 15643/172740 objects misplaced (9.056%) 2024-03-24T12:44:13.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:13 smithi029 ceph-mon[41500]: 3.2 scrub starts 2024-03-24T12:44:13.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:13 smithi029 ceph-mon[41500]: 3.2 scrub ok 2024-03-24T12:44:13.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:13 smithi029 ceph-mon[41500]: pgmap v789: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 126 op/s; 12955/172740 objects degraded (7.500%); 15643/172740 objects misplaced (9.056%) 2024-03-24T12:44:14.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:14.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:14.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:14.145 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:14.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:14.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:14.280 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:14.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:14.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:14.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:14.407 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:14.431 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:14.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:14.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:14.569 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:14.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:14.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:14.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:14.701 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:14.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:14.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:14.803 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:14.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:14.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:14.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:14 smithi190 ceph-mon[33748]: 1.8 scrub starts 2024-03-24T12:44:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:14 smithi190 ceph-mon[33748]: 1.8 scrub ok 2024-03-24T12:44:14.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:14 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:44:14.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:14 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:44:14.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:14 smithi029 ceph-mon[29919]: 1.8 scrub starts 2024-03-24T12:44:14.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:14 smithi029 ceph-mon[29919]: 1.8 scrub ok 2024-03-24T12:44:14.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:14 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:44:14.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:14 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:44:14.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:14 smithi029 ceph-mon[41500]: 1.8 scrub starts 2024-03-24T12:44:14.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:14 smithi029 ceph-mon[41500]: 1.8 scrub ok 2024-03-24T12:44:14.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:14 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:44:14.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:14 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:44:14.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:14.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:14.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:14.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:15.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:15.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:15.111 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:15.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:15.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:15.236 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:15.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:15.338 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:15.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:15.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:15.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:15.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:15.504 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:15.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:15.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:15.633 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:15.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:15.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:15.735 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:15.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:15.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:15.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:15.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:15 smithi190 ceph-mon[33748]: 4.6 scrub starts 2024-03-24T12:44:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:15 smithi190 ceph-mon[33748]: 4.6 scrub ok 2024-03-24T12:44:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:15 smithi190 ceph-mon[33748]: pgmap v790: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11980/164007 objects degraded (7.305%); 14540/164007 objects misplaced (8.865%) 2024-03-24T12:44:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:15 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 12955/172740 objects degraded (7.500%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:15.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:15.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:15.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:15 smithi029 ceph-mon[29919]: 4.6 scrub starts 2024-03-24T12:44:15.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:15 smithi029 ceph-mon[29919]: 4.6 scrub ok 2024-03-24T12:44:15.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:15 smithi029 ceph-mon[29919]: pgmap v790: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11980/164007 objects degraded (7.305%); 14540/164007 objects misplaced (8.865%) 2024-03-24T12:44:15.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:15 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 12955/172740 objects degraded (7.500%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:15.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:15 smithi029 ceph-mon[41500]: 4.6 scrub starts 2024-03-24T12:44:15.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:15 smithi029 ceph-mon[41500]: 4.6 scrub ok 2024-03-24T12:44:15.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:15 smithi029 ceph-mon[41500]: pgmap v790: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 11980/164007 objects degraded (7.305%); 14540/164007 objects misplaced (8.865%) 2024-03-24T12:44:15.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:15 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 12955/172740 objects degraded (7.500%), 14 pgs degraded, 30 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:16.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:16.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:16.055 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:16.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:16.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:16.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:16.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:16.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:16.331 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:16.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:16.432 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:16.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:16.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:16.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:16.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:16.597 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:16.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:16.702 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:16.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:16.738 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:16.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:16.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:16.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:16.879 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:16.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:16 smithi190 ceph-mon[33748]: 6.c scrub starts 2024-03-24T12:44:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:16 smithi190 ceph-mon[33748]: 6.c scrub ok 2024-03-24T12:44:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:16 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:44:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:16 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:44:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:16 smithi190 ceph-mon[33748]: 1.3 scrub starts 2024-03-24T12:44:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:16 smithi190 ceph-mon[33748]: 1.3 scrub ok 2024-03-24T12:44:16.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[29919]: 6.c scrub starts 2024-03-24T12:44:16.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[29919]: 6.c scrub ok 2024-03-24T12:44:16.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:44:16.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:44:16.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[29919]: 1.3 scrub starts 2024-03-24T12:44:16.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[29919]: 1.3 scrub ok 2024-03-24T12:44:16.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[41500]: 6.c scrub starts 2024-03-24T12:44:16.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[41500]: 6.c scrub ok 2024-03-24T12:44:16.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:44:16.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:44:16.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[41500]: 1.3 scrub starts 2024-03-24T12:44:16.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:16 smithi029 ceph-mon[41500]: 1.3 scrub ok 2024-03-24T12:44:16.980 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:17.018 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:17.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:17.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:17.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:17.149 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:17.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:17.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:17.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:17.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:17.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:17.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:17.422 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:17.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:17.523 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:17.559 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:17.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:17.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:17.687 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:17.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:17.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:17.831 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:17.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:17 smithi190 ceph-mon[33748]: 6.1f scrub starts 2024-03-24T12:44:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:17 smithi190 ceph-mon[33748]: 6.1f scrub ok 2024-03-24T12:44:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:17 smithi190 ceph-mon[33748]: 3.d scrub starts 2024-03-24T12:44:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:17 smithi190 ceph-mon[33748]: 3.d scrub ok 2024-03-24T12:44:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:17 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:44:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:17 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:44:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:17 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:44:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:17 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:44:17.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:17 smithi190 ceph-mon[33748]: pgmap v791: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 11980/163650 objects degraded (7.321%); 14540/163650 objects misplaced (8.885%) 2024-03-24T12:44:17.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:44:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:44:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[29919]: 6.1f scrub starts 2024-03-24T12:44:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[29919]: 6.1f scrub ok 2024-03-24T12:44:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[29919]: 3.d scrub starts 2024-03-24T12:44:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[29919]: 3.d scrub ok 2024-03-24T12:44:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:44:17.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:44:17.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:44:17.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:44:17.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[29919]: pgmap v791: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 11980/163650 objects degraded (7.321%); 14540/163650 objects misplaced (8.885%) 2024-03-24T12:44:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[41500]: 6.1f scrub starts 2024-03-24T12:44:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[41500]: 6.1f scrub ok 2024-03-24T12:44:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[41500]: 3.d scrub starts 2024-03-24T12:44:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[41500]: 3.d scrub ok 2024-03-24T12:44:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:44:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:44:17.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:44:17.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:44:17.919 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:17 smithi029 ceph-mon[41500]: pgmap v791: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.3 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 11980/163650 objects degraded (7.321%); 14540/163650 objects misplaced (8.885%) 2024-03-24T12:44:17.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:17.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:17.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:18.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:18.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:18.083 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:18.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:18.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:18.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:18.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:18.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:18.349 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:18.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:18.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:18.477 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:18.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:18.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:18.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:18.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:18.620 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:18.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:18.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:18.759 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:18.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:18.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:18.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:18.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:44:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:44:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:18 smithi190 ceph-mon[33748]: 4.b scrub starts 2024-03-24T12:44:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:18 smithi190 ceph-mon[33748]: 4.b scrub ok 2024-03-24T12:44:18.884 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:18.885 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:18.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:18.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:18.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:18 smithi029 ceph-mon[29919]: 4.b scrub starts 2024-03-24T12:44:18.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:18 smithi029 ceph-mon[29919]: 4.b scrub ok 2024-03-24T12:44:18.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:18.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:18 smithi029 ceph-mon[41500]: 4.b scrub starts 2024-03-24T12:44:18.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:18 smithi029 ceph-mon[41500]: 4.b scrub ok 2024-03-24T12:44:18.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:18.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:19.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:19.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:19.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:19.121 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:19.155 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:19.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:19.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:19.297 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:19.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:19.398 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:19.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:19.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:19.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:19.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:19.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:19.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:19.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:19.700 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:19.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:19.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:19.802 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:19.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:19.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:19 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:44:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:19 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:44:19.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:19 smithi190 ceph-mon[33748]: pgmap v792: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 11298/150441 objects degraded (7.510%); 13747/150441 objects misplaced (9.138%) 2024-03-24T12:44:19.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:19.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:19 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:44:19.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:19 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:44:19.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:19 smithi029 ceph-mon[29919]: pgmap v792: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 11298/150441 objects degraded (7.510%); 13747/150441 objects misplaced (9.138%) 2024-03-24T12:44:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:19 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:44:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:19 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:44:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:19 smithi029 ceph-mon[41500]: pgmap v792: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 11298/150441 objects degraded (7.510%); 13747/150441 objects misplaced (9.138%) 2024-03-24T12:44:19.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:19.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:19.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:19.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:20.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:20.118 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:20.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:20.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:20.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:20.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:20.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:20.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:20.349 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:20.379 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:20.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:20.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:20.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:20.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:20.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:20.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:20.633 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:20 smithi029 ceph-mon[29919]: 5.1e scrub starts 2024-03-24T12:44:20.665 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:20.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:20.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:20.808 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:20.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:20.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:20 smithi190 ceph-mon[33748]: 5.1e scrub starts 2024-03-24T12:44:20.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:20 smithi190 ceph-mon[33748]: 5.1e scrub ok 2024-03-24T12:44:20.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:20 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 11298/150441 objects degraded (7.510%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:20.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:20.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:20 smithi029 ceph-mon[29919]: 5.1e scrub ok 2024-03-24T12:44:20.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:20 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 11298/150441 objects degraded (7.510%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:20.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:20 smithi029 ceph-mon[41500]: 5.1e scrub starts 2024-03-24T12:44:20.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:20 smithi029 ceph-mon[41500]: 5.1e scrub ok 2024-03-24T12:44:20.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:20 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 11298/150441 objects degraded (7.510%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:20.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:20.949 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:20.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:21.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:21.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:21.083 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:21.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:21.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:21.217 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:21.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:21.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:21.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:21.355 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:21.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:21.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:21.484 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:21.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:21.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:21.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:21.626 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:21.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:21.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:21.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:21.868 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:21.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:21.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:21.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:21 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:44:21.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:21 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:44:21.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:21 smithi029 ceph-mon[41500]: pgmap v793: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 10756/147279 objects degraded (7.303%); 12707/147279 objects misplaced (8.628%) 2024-03-24T12:44:21.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:21 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:44:21.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:21 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:44:21.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:21 smithi029 ceph-mon[29919]: pgmap v793: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 10756/147279 objects degraded (7.303%); 12707/147279 objects misplaced (8.628%) 2024-03-24T12:44:22.004 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:22.034 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:22.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:21 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:44:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:21 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:44:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:21 smithi190 ceph-mon[33748]: pgmap v793: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 3.0 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 10756/147279 objects degraded (7.303%); 12707/147279 objects misplaced (8.628%) 2024-03-24T12:44:22.137 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:22.179 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:22.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:22.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:22.310 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:22.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:22.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:22.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:22.453 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:22.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:22.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:22.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:22.608 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:22.608 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:22.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:22.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:22.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:22.859 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:22.890 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:22.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:22.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:44:22.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:44:22.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[41500]: 1.5 scrub starts 2024-03-24T12:44:22.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[41500]: 1.5 scrub ok 2024-03-24T12:44:22.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[41500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:44:22.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[41500]: osdmap e347: 8 total, 7 up, 8 in 2024-03-24T12:44:22.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:44:22.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:44:22.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[29919]: 1.5 scrub starts 2024-03-24T12:44:22.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[29919]: 1.5 scrub ok 2024-03-24T12:44:22.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[29919]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:44:22.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:22 smithi029 ceph-mon[29919]: osdmap e347: 8 total, 7 up, 8 in 2024-03-24T12:44:22.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:23.031 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:23.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:23.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:22 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:44:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:22 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:44:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:22 smithi190 ceph-mon[33748]: 1.5 scrub starts 2024-03-24T12:44:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:22 smithi190 ceph-mon[33748]: 1.5 scrub ok 2024-03-24T12:44:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:22 smithi190 ceph-mon[33748]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:44:23.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:22 smithi190 ceph-mon[33748]: osdmap e347: 8 total, 7 up, 8 in 2024-03-24T12:44:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:23.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:23.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:23.272 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:44:23.275 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:23.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:23.319 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:23.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:23.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:23.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:23.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:23.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:23.556 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:23.587 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:23.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:23.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:23.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:23.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:23.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:23.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:23.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:23.874 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:23.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:23.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:23 smithi029 ceph-mon[41500]: 3.e scrub starts 2024-03-24T12:44:23.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:23 smithi029 ceph-mon[41500]: 3.e scrub ok 2024-03-24T12:44:23.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:23 smithi029 ceph-mon[41500]: pgmap v795: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 10595/140976 objects degraded (7.515%); 12106/140976 objects misplaced (8.587%); 2.2 MiB/s, 35 objects/s recovering 2024-03-24T12:44:23.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:23 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:44:23.888 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:23 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:44:23.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:23 smithi029 ceph-mon[29919]: 3.e scrub starts 2024-03-24T12:44:23.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:23 smithi029 ceph-mon[29919]: 3.e scrub ok 2024-03-24T12:44:23.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:23 smithi029 ceph-mon[29919]: pgmap v795: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 10595/140976 objects degraded (7.515%); 12106/140976 objects misplaced (8.587%); 2.2 MiB/s, 35 objects/s recovering 2024-03-24T12:44:23.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:23 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:44:23.889 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:23 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:44:23.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:24.019 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:24.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:24.113 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:44:24.113 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:44:24.113 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.113 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:211488 10.a e348/278 10:50a3bae9:::benchmark_data_smithi029_160664_object84514:head v 348'13352, mlcod=348'13348)", 2024-03-24T12:44:24.114 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:24.099372+0000", 2024-03-24T12:44:24.114 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.013343908, 2024-03-24T12:44:24.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.013391604, 2024-03-24T12:44:24.114 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:44:24.114 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:44:24.114 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:44:24.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.114 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:44:24.114 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099372+0000", 2024-03-24T12:44:24.114 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099372+0000", 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099372+0000", 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2900000000000004e-07 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099379+0000", 2024-03-24T12:44:24.115 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.99e-06 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099392+0000", 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3132999999999999e-05 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099399+0000", 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8179999999999998e-06 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.116 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:44:24.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.101490+0000", 2024-03-24T12:44:24.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020904809999999999 2024-03-24T12:44:24.117 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.117 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.117 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:44:24.117 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.101510+0000", 2024-03-24T12:44:24.117 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0567000000000001e-05 2024-03-24T12:44:24.117 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.117 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:44:24.118 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.118 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:211483 10.5 e348/326 10:a19bfd8e:::benchmark_data_smithi029_160664_object84509:head v 348'11302, mlcod=348'11300)", 2024-03-24T12:44:24.118 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:24.099452+0000", 2024-03-24T12:44:24.118 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.013263659000000001, 2024-03-24T12:44:24.118 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.013389046999999999, 2024-03-24T12:44:24.118 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:44:24.118 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099452+0000", 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099452+0000", 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:44:24.119 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099452+0000", 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6899999999999998e-07 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099456+0000", 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9589999999999997e-06 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099464+0000", 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2429999999999999e-06 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099469+0000", 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.592e-06 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.101074+0000", 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0016044970000000001 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.101093+0000", 2024-03-24T12:44:24.121 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9578000000000001e-05 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "MOSDPGPushReply(10.14 348/329 [PushReplyOp(10:2ec4531b:::benchmark_data_smithi029_160664_object118281:head)])", 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:24.099498+0000", 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.013217318, 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.013406428999999999, 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:44:24.122 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099498+0000", 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099498+0000", 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099498+0000", 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.05e-07 2024-03-24T12:44:24.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099501+0000", 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.526e-06 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099506+0000", 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3360000000000002e-06 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099512+0000", 2024-03-24T12:44:24.124 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4539999999999998e-06 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.112521+0000", 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.013009484 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:211489 10.4 10:2057e19c:::benchmark_data_smithi029_160664_object84515:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:24.099542+0000", 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.013173127999999999, 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.013443800000000001, 2024-03-24T12:44:24.125 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 211489 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099542+0000", 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.126 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099542+0000", 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099544+0000", 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.128e-06 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099546+0000", 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.818e-06 2024-03-24T12:44:24.127 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099554+0000", 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3980000000000002e-06 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.099559+0000", 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6609999999999997e-06 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:44:24.128 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.101672+0000", 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0021127329999999999 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.101816+0000", 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014407499999999999 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.102213+0000", 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00039673600000000002 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.129 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.108242+0000", 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060292710000000001 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.112299+0000", 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0040566170000000002 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.130 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:211491 10.14 10:2d4c50cb:::benchmark_data_smithi029_160664_object84517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:24.100350+0000", 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.012365364, 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012730136, 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 211491 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.131 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.100350+0000", 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.100350+0000", 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.100351+0000", 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2729999999999999e-06 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.100356+0000", 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3649999999999997e-06 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.100367+0000", 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0733999999999999e-05 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.100375+0000", 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7569999999999993e-06 2024-03-24T12:44:24.133 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.101158+0000", 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00078297000000000002 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.101284+0000", 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012530199999999999 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2,3,4", 2024-03-24T12:44:24.134 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.101606+0000", 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000322491 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.112132+0000", 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010526077999999999 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.112220+0000", 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8311000000000002e-05 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.135 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.112257+0000", 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6798000000000002e-05 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.113017+0000", 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00075989000000000004 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:211495 10.b e348/278 10:d0f47909:::benchmark_data_smithi029_160664_object84521:head v 348'13149, mlcod=348'13147)", 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:24.103587+0000", 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0091289479999999996, 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0095946509999999992, 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.103587+0000", 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.137 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.103587+0000", 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.103587+0000", 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0900000000000002e-07 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.103592+0000", 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6260000000000003e-06 2024-03-24T12:44:24.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.103603+0000", 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1175999999999999e-05 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.103609+0000", 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4250000000000003e-06 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.103691+0000", 2024-03-24T12:44:24.139 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1312999999999999e-05 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.103710+0000", 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9575000000000001e-05 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:211497 10.6 10:638716e4:::benchmark_data_smithi029_160664_object84523:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e348)", 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:24.104380+0000", 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0083360020000000003, 2024-03-24T12:44:24.140 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.008865029, 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 211497 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.104380+0000", 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.141 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.104380+0000", 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.104381+0000", 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.209e-06 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.142 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.104385+0000", 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.262e-06 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.104397+0000", 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1790000000000001e-05 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:44:24.143 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.104406+0000", 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9209999999999998e-06 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.104463+0000", 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7027999999999997e-05 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.106186+0000", 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001723397 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4", 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.106602+0000", 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00041540300000000001 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.110450+0000", 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0038482009999999999 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.145 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:211500 10.d e348/278 10:b2e84c0f:::benchmark_data_smithi029_160664_object84526:head v 348'18924, mlcod=348'18921)", 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:24.105964+0000", 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0067511769999999997, 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.007363952, 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.105964+0000", 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.146 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.105964+0000", 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.105965+0000", 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.75e-07 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.105971+0000", 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4990000000000002e-06 2024-03-24T12:44:24.147 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.105979+0000", 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6659999999999997e-06 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.105986+0000", 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6440000000000003e-06 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.106047+0000", 2024-03-24T12:44:24.148 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0940000000000003e-05 2024-03-24T12:44:24.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.149 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.149 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:44:24.149 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.106065+0000", 2024-03-24T12:44:24.149 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8116000000000001e-05 2024-03-24T12:44:24.149 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.149 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:44:24.149 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.149 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.149 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.150 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.15279.0:211503 10.2 e348/326 10:47e2a7ce:::benchmark_data_smithi029_160664_object84529:head v 348'11270, mlcod=348'11267)", 2024-03-24T12:44:24.150 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:24.110001+0000", 2024-03-24T12:44:24.150 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0027141410000000002, 2024-03-24T12:44:24.150 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0033880109999999998, 2024-03-24T12:44:24.150 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:44:24.150 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:44:24.150 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:44:24.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.151 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:44:24.151 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.110001+0000", 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.110001+0000", 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.110002+0000", 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.75e-07 2024-03-24T12:44:24.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.110006+0000", 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4610000000000001e-06 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.110014+0000", 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5190000000000003e-06 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.110020+0000", 2024-03-24T12:44:24.153 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9889999999999997e-06 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.110269+0000", 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024876100000000001 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:24.110290+0000", 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0798e-05 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:44:24.154 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.155 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:44:24.155 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:44:24.155 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 9 2024-03-24T12:44:24.155 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:44:24.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:24.156 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:23 smithi190 ceph-mon[33748]: 3.e scrub starts 2024-03-24T12:44:24.156 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:23 smithi190 ceph-mon[33748]: 3.e scrub ok 2024-03-24T12:44:24.156 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:23 smithi190 ceph-mon[33748]: pgmap v795: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.9 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 10595/140976 objects degraded (7.515%); 12106/140976 objects misplaced (8.587%); 2.2 MiB/s, 35 objects/s recovering 2024-03-24T12:44:24.156 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:23 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:44:24.156 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:23 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:44:24.200 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:24.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:24.302 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:24.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:24.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:24.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:24.437 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:24.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:24.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:24.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:24.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:24.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:24.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:24.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:24.727 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:24.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:24.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:44:24.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:24.874 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:24.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:24 smithi029 ceph-mon[29919]: 6.1e scrub starts 2024-03-24T12:44:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:24 smithi029 ceph-mon[29919]: 6.1e scrub ok 2024-03-24T12:44:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:24 smithi029 ceph-mon[29919]: osdmap e348: 8 total, 7 up, 8 in 2024-03-24T12:44:24.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:24 smithi029 ceph-mon[41500]: 6.1e scrub starts 2024-03-24T12:44:24.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:24 smithi029 ceph-mon[41500]: 6.1e scrub ok 2024-03-24T12:44:24.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:24 smithi029 ceph-mon[41500]: osdmap e348: 8 total, 7 up, 8 in 2024-03-24T12:44:24.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:24.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:24.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:25.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:25.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:25.131 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:25.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:25.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:24 smithi190 ceph-mon[33748]: 6.1e scrub starts 2024-03-24T12:44:25.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:24 smithi190 ceph-mon[33748]: 6.1e scrub ok 2024-03-24T12:44:25.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:24 smithi190 ceph-mon[33748]: osdmap e348: 8 total, 7 up, 8 in 2024-03-24T12:44:25.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:25.289 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:25.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:25.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:25.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:25.419 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:25.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:25.520 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:25.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:25.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:25.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:25.656 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:25.690 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:25.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:25.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:25.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:25.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:25.832 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:25.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:25.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:25 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:44:25.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:25 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:44:25.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:25 smithi029 ceph-mon[29919]: pgmap v797: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 9519/131448 objects degraded (7.242%); 9805/131448 objects misplaced (7.459%); 7.6 MiB/s, 120 objects/s recovering 2024-03-24T12:44:25.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:25 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 10595/140976 objects degraded (7.515%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:25.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:25 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:44:25.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:25 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:44:25.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:25 smithi029 ceph-mon[41500]: pgmap v797: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 9519/131448 objects degraded (7.242%); 9805/131448 objects misplaced (7.459%); 7.6 MiB/s, 120 objects/s recovering 2024-03-24T12:44:25.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:25 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 10595/140976 objects degraded (7.515%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:25.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:25.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:26.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:26.106 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:26.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:26.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:26.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:25 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:44:26.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:25 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:44:26.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:25 smithi190 ceph-mon[33748]: pgmap v797: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s; 9519/131448 objects degraded (7.242%); 9805/131448 objects misplaced (7.459%); 7.6 MiB/s, 120 objects/s recovering 2024-03-24T12:44:26.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:25 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 10595/140976 objects degraded (7.515%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:26.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:26.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:26.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:26.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:26.379 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:26.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:26.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:26.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:26.506 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:26.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:26.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:26.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:26.643 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:26.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:26.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:26.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:26.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:26.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:26.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:26.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:27.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:27.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:27.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:27.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:27.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:27.180 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:27.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:27.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:27.310 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:27.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:27.411 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:27.435 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:27.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:27.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:27.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:27.565 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:27.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:27.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:27.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:27.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:27.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:27.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:27.824 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:27.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:27.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-mon[41500]: 3.14 scrub starts 2024-03-24T12:44:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-mon[29919]: 3.14 scrub starts 2024-03-24T12:44:27.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:44:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:44:27.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:27.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:27.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:28.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:28.083 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:28.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:28.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:28.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:27 smithi190 ceph-mon[33748]: 3.14 scrub starts 2024-03-24T12:44:28.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:27 smithi190 ceph-mon[33748]: 3.14 scrub ok 2024-03-24T12:44:28.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:27 smithi190 ceph-mon[33748]: 10.14 scrub starts 2024-03-24T12:44:28.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:27 smithi190 ceph-mon[33748]: osdmap e349: 8 total, 7 up, 8 in 2024-03-24T12:44:28.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:27 smithi190 ceph-mon[33748]: pgmap v799: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 908 op/s; 9519/130935 objects degraded (7.270%); 9805/130935 objects misplaced (7.488%); 8.1 MiB/s, 128 objects/s recovering 2024-03-24T12:44:28.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:28.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:28.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:28.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-mon[29919]: 3.14 scrub ok 2024-03-24T12:44:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-mon[29919]: 10.14 scrub starts 2024-03-24T12:44:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-mon[29919]: osdmap e349: 8 total, 7 up, 8 in 2024-03-24T12:44:28.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-mon[29919]: pgmap v799: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 908 op/s; 9519/130935 objects degraded (7.270%); 9805/130935 objects misplaced (7.488%); 8.1 MiB/s, 128 objects/s recovering 2024-03-24T12:44:28.245 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-mon[41500]: 3.14 scrub ok 2024-03-24T12:44:28.246 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-mon[41500]: 10.14 scrub starts 2024-03-24T12:44:28.246 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-mon[41500]: osdmap e349: 8 total, 7 up, 8 in 2024-03-24T12:44:28.246 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:27 smithi029 ceph-mon[41500]: pgmap v799: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.7 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 908 op/s; 9519/130935 objects degraded (7.270%); 9805/130935 objects misplaced (7.488%); 8.1 MiB/s, 128 objects/s recovering 2024-03-24T12:44:28.263 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 7, 6] out_osds: [] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:44:28.263 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:44:28.264 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 4, 5, 7, 6] 2024-03-24T12:44:28.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd out 5 2024-03-24T12:44:28.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:28.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:28.339 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:28.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:28.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:28.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:28.470 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:28.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:28.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:28.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:28.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:28.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:28.666 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:44:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:44:28.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:28.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:28.786 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:28.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:28.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:28.912 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:28.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:28.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:28 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:44:28.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:28 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:44:28.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:28 smithi029 ceph-mon[29919]: osdmap e350: 8 total, 7 up, 8 in 2024-03-24T12:44:28.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:28 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:44:28.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:28 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:44:28.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:28 smithi029 ceph-mon[41500]: osdmap e350: 8 total, 7 up, 8 in 2024-03-24T12:44:29.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:29.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:29.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:28 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:44:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:28 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:44:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:28 smithi190 ceph-mon[33748]: osdmap e350: 8 total, 7 up, 8 in 2024-03-24T12:44:29.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:29.168 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:29.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:29.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:29.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:29.295 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:29.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:29.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:29.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:29.421 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:29.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:29.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:29.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:29.547 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:29.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:29.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:29.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:29.673 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:29.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:29.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:29.801 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:29.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:29.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:29.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:29.930 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:29.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:30.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:30.059 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:30.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:30.110 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:29 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:44:30.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:29 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:44:30.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:44:30.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:30.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 4.d scrub starts 2024-03-24T12:44:30.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 4.d scrub ok 2024-03-24T12:44:30.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 10.18 scrub starts 2024-03-24T12:44:30.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: osdmap e351: 8 total, 7 up, 8 in 2024-03-24T12:44:30.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: pgmap v802: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 990 B/s rd, 0 B/s wr, 878 op/s; 8986/120462 objects degraded (7.460%); 9271/120462 objects misplaced (7.696%); 7.4 MiB/s, 117 objects/s recovering 2024-03-24T12:44:30.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:29 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:44:30.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:29 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:44:30.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:29 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:30.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:29 smithi190 ceph-mon[33748]: 4.d scrub starts 2024-03-24T12:44:30.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:29 smithi190 ceph-mon[33748]: 4.d scrub ok 2024-03-24T12:44:30.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:29 smithi190 ceph-mon[33748]: 10.18 scrub starts 2024-03-24T12:44:30.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:29 smithi190 ceph-mon[33748]: osdmap e351: 8 total, 7 up, 8 in 2024-03-24T12:44:30.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:29 smithi190 ceph-mon[33748]: pgmap v802: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 990 B/s rd, 0 B/s wr, 878 op/s; 8986/120462 objects degraded (7.460%); 9271/120462 objects misplaced (7.696%); 7.4 MiB/s, 117 objects/s recovering 2024-03-24T12:44:30.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:30.203 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:30.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:30.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:30.344 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:30.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:30.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:44:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 4.d scrub starts 2024-03-24T12:44:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 4.d scrub ok 2024-03-24T12:44:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 10.18 scrub starts 2024-03-24T12:44:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: osdmap e351: 8 total, 7 up, 8 in 2024-03-24T12:44:30.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: pgmap v802: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 2 active+remapped+backfilling, 26 active+clean+remapped, 105 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 990 B/s rd, 0 B/s wr, 878 op/s; 8986/120462 objects degraded (7.460%); 9271/120462 objects misplaced (7.696%); 7.4 MiB/s, 117 objects/s recovering 2024-03-24T12:44:30.446 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:30.487 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:30.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:30.588 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:30.618 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:30.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:30.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:30.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:30.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:30.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:30.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:30.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:30.882 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:30.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:30.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:30.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:31.031 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:31.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:31.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: 4.12 scrub starts 2024-03-24T12:44:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: 4.12 scrub ok 2024-03-24T12:44:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: 10.14 scrub starts 2024-03-24T12:44:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: 3.c scrub starts 2024-03-24T12:44:31.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: 3.c scrub ok 2024-03-24T12:44:31.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: 10.14 scrub ok 2024-03-24T12:44:31.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: 4.1a scrub starts 2024-03-24T12:44:31.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: 4.1a scrub ok 2024-03-24T12:44:31.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: osdmap e352: 8 total, 7 up, 8 in 2024-03-24T12:44:31.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:44:31.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:30 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 8986/120462 objects degraded (7.460%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:31.134 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:31.166 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 4.12 scrub starts 2024-03-24T12:44:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 4.12 scrub ok 2024-03-24T12:44:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 10.14 scrub starts 2024-03-24T12:44:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 3.c scrub starts 2024-03-24T12:44:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 3.c scrub ok 2024-03-24T12:44:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 10.14 scrub ok 2024-03-24T12:44:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 4.1a scrub starts 2024-03-24T12:44:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: 4.1a scrub ok 2024-03-24T12:44:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: osdmap e352: 8 total, 7 up, 8 in 2024-03-24T12:44:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:44:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 8986/120462 objects degraded (7.460%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 4.12 scrub starts 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 4.12 scrub ok 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 10.14 scrub starts 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 3.c scrub starts 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 3.c scrub ok 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 10.14 scrub ok 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 4.1a scrub starts 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: 4.1a scrub ok 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: osdmap e352: 8 total, 7 up, 8 in 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:44:31.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:30 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 8986/120462 objects degraded (7.460%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:31.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:31.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:31.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:31.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:31.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:31.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:31.454 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:31.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:31.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:31.589 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:31.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:31.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:31.721 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:31.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:31.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:31.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:31.847 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:31.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:31.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:31.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:31.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:32.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:32.115 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:32.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:32.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:31 smithi190 ceph-mon[33748]: 10.18 scrub starts 2024-03-24T12:44:32.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:31 smithi190 ceph-mon[33748]: pgmap v804: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 26 active+clean+remapped, 106 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 8687/118233 objects degraded (7.347%); 7367/118233 objects misplaced (6.231%) 2024-03-24T12:44:32.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:31 smithi190 ceph-mon[33748]: 10.18 scrub ok 2024-03-24T12:44:32.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:31 smithi029 ceph-mon[29919]: 10.18 scrub starts 2024-03-24T12:44:32.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:31 smithi029 ceph-mon[29919]: pgmap v804: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 26 active+clean+remapped, 106 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 8687/118233 objects degraded (7.347%); 7367/118233 objects misplaced (6.231%) 2024-03-24T12:44:32.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:31 smithi029 ceph-mon[29919]: 10.18 scrub ok 2024-03-24T12:44:32.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:31 smithi029 ceph-mon[41500]: 10.18 scrub starts 2024-03-24T12:44:32.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:31 smithi029 ceph-mon[41500]: pgmap v804: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 26 active+clean+remapped, 106 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 8687/118233 objects degraded (7.347%); 7367/118233 objects misplaced (6.231%) 2024-03-24T12:44:32.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:31 smithi029 ceph-mon[41500]: 10.18 scrub ok 2024-03-24T12:44:32.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:32.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:32.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:32.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:32.383 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:32.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:32.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:32.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:32.512 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:32.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:32.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:32.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:32.637 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:32.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:32.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:32.780 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:32.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:32.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:32.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:32.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:33.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:33.044 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:33.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:33.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:32 smithi190 ceph-mon[33748]: 3.1 scrub starts 2024-03-24T12:44:33.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:32 smithi190 ceph-mon[33748]: 3.1 scrub ok 2024-03-24T12:44:33.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:32 smithi190 ceph-mon[33748]: 6.a scrub starts 2024-03-24T12:44:33.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:32 smithi190 ceph-mon[33748]: 6.a scrub ok 2024-03-24T12:44:33.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:32 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:44:33.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:32 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:44:33.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:32 smithi190 ceph-mon[33748]: 4.10 scrub starts 2024-03-24T12:44:33.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:32 smithi190 ceph-mon[33748]: 4.10 scrub ok 2024-03-24T12:44:33.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[41500]: 3.1 scrub starts 2024-03-24T12:44:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[41500]: 3.1 scrub ok 2024-03-24T12:44:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[41500]: 6.a scrub starts 2024-03-24T12:44:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[41500]: 6.a scrub ok 2024-03-24T12:44:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:44:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:44:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[41500]: 4.10 scrub starts 2024-03-24T12:44:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[41500]: 4.10 scrub ok 2024-03-24T12:44:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[29919]: 3.1 scrub starts 2024-03-24T12:44:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[29919]: 3.1 scrub ok 2024-03-24T12:44:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[29919]: 6.a scrub starts 2024-03-24T12:44:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[29919]: 6.a scrub ok 2024-03-24T12:44:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:44:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:44:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[29919]: 4.10 scrub starts 2024-03-24T12:44:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:32 smithi029 ceph-mon[29919]: 4.10 scrub ok 2024-03-24T12:44:33.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:33.190 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:33.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:33.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:33.324 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:33.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:33.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:33.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:33.454 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:33.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:33.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:33.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:33.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:33.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:33.728 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:33.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:33.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:33.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:33.853 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:33.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:33.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:33.997 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:33.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:34.098 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:34.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:34.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:34.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:34.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:33 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:44:34.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:33 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:44:34.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:33 smithi190 ceph-mon[33748]: pgmap v805: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 26 active+clean+remapped, 106 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 8593/114468 objects degraded (7.507%); 7259/114468 objects misplaced (6.342%) 2024-03-24T12:44:34.135 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:33 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:44:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:33 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:44:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:33 smithi029 ceph-mon[41500]: pgmap v805: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 26 active+clean+remapped, 106 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 8593/114468 objects degraded (7.507%); 7259/114468 objects misplaced (6.342%) 2024-03-24T12:44:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:34.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:33 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:44:34.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:33 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:44:34.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:33 smithi029 ceph-mon[29919]: pgmap v805: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 26 active+clean+remapped, 106 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 8593/114468 objects degraded (7.507%); 7259/114468 objects misplaced (6.342%) 2024-03-24T12:44:34.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:34.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:34.273 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:34.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:34.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:34.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:34.414 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:34.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:34.515 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:34.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:34.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:34.660 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:34.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:34.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:34.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:34.829 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:34.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:34.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:34.932 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:34.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T12:44:34.964 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:34.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:34.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:34 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:44:34.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:34 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:44:34.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:34 smithi029 ceph-mon[41500]: 3.10 scrub starts 2024-03-24T12:44:34.994 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:34 smithi029 ceph-mon[41500]: 3.10 scrub ok 2024-03-24T12:44:34.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:34 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:44:34.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:34 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:44:34.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:34 smithi029 ceph-mon[29919]: 3.10 scrub starts 2024-03-24T12:44:34.995 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:34 smithi029 ceph-mon[29919]: 3.10 scrub ok 2024-03-24T12:44:35.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:35.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:35.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:35.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:35.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:34 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:44:35.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:34 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:44:35.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:34 smithi190 ceph-mon[33748]: 3.10 scrub starts 2024-03-24T12:44:35.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:34 smithi190 ceph-mon[33748]: 3.10 scrub ok 2024-03-24T12:44:35.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:35.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:35.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:35.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:35.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:35.376 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:35.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:35.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:35.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:35.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:35.517 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:35.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:35.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:35.658 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:35.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:35.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:35.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:35.785 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:35.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:35.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:35.921 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:35.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:36.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:36.063 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:36.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:36.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:35 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:44:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:35 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:44:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:35 smithi190 ceph-mon[33748]: 4.1 scrub starts 2024-03-24T12:44:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:35 smithi190 ceph-mon[33748]: 4.1 scrub ok 2024-03-24T12:44:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:35 smithi190 ceph-mon[33748]: 6.15 scrub starts 2024-03-24T12:44:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:35 smithi190 ceph-mon[33748]: 1.a scrub starts 2024-03-24T12:44:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:35 smithi190 ceph-mon[33748]: 1.a scrub ok 2024-03-24T12:44:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:35 smithi190 ceph-mon[33748]: 6.15 scrub ok 2024-03-24T12:44:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:35 smithi190 ceph-mon[33748]: pgmap v806: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 7757/107358 objects degraded (7.225%); 6613/107358 objects misplaced (6.160%) 2024-03-24T12:44:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:35 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 8593/114468 objects degraded (7.507%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:36.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:44:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:44:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[29919]: 4.1 scrub starts 2024-03-24T12:44:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[29919]: 4.1 scrub ok 2024-03-24T12:44:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[29919]: 6.15 scrub starts 2024-03-24T12:44:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[29919]: 1.a scrub starts 2024-03-24T12:44:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[29919]: 1.a scrub ok 2024-03-24T12:44:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[29919]: 6.15 scrub ok 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[29919]: pgmap v806: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 7757/107358 objects degraded (7.225%); 6613/107358 objects misplaced (6.160%) 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 8593/114468 objects degraded (7.507%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[41500]: 4.1 scrub starts 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[41500]: 4.1 scrub ok 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[41500]: 6.15 scrub starts 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[41500]: 1.a scrub starts 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[41500]: 1.a scrub ok 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[41500]: 6.15 scrub ok 2024-03-24T12:44:36.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[41500]: pgmap v806: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 7757/107358 objects degraded (7.225%); 6613/107358 objects misplaced (6.160%) 2024-03-24T12:44:36.169 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:35 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 8593/114468 objects degraded (7.507%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:36.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:36.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:36.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:36.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:36.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:36.325 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:36.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:36.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:36.509 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:44:36.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:36.511 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:36.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:44:36.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:36.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:36.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:36.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:36.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:36.825 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:36.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:36.926 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:36.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:36.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:37.052 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:37.083 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:37.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:36 smithi190 ceph-mon[33748]: 3.1d scrub starts 2024-03-24T12:44:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:36 smithi190 ceph-mon[33748]: 3.1d scrub ok 2024-03-24T12:44:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:36 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:44:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:36 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:44:37.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:36 smithi029 ceph-mon[41500]: 3.1d scrub starts 2024-03-24T12:44:37.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:36 smithi029 ceph-mon[41500]: 3.1d scrub ok 2024-03-24T12:44:37.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:36 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:44:37.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:36 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:44:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:36 smithi029 ceph-mon[29919]: 3.1d scrub starts 2024-03-24T12:44:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:36 smithi029 ceph-mon[29919]: 3.1d scrub ok 2024-03-24T12:44:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:36 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:44:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:36 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:44:37.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:37.224 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:37.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:37.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:37.362 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:37.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:37.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:37.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:37.501 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:37.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:37.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:37.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:37.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:37.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:37.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:37.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:37.784 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:37.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:37.885 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:37.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:44:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:44:37.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:44:37.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:44:37.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:44:37.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:44:37.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:44:37.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:44:37.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[41500]: pgmap v807: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 628 B/s rd, 0 op/s; 7757/106302 objects degraded (7.297%); 6613/106302 objects misplaced (6.221%) 2024-03-24T12:44:37.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:44:37.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:44:37.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:44:37.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:44:37.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:44:37.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:44:37.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:37 smithi029 ceph-mon[29919]: pgmap v807: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 628 B/s rd, 0 op/s; 7757/106302 objects degraded (7.297%); 6613/106302 objects misplaced (6.221%) 2024-03-24T12:44:37.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:37.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:37.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:38.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:38.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:38.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:38.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:38.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:37 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:44:38.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:37 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:44:38.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:37 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:44:38.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:37 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:44:38.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:37 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:44:38.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:37 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:44:38.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:37 smithi190 ceph-mon[33748]: pgmap v807: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 2.2 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 628 B/s rd, 0 op/s; 7757/106302 objects degraded (7.297%); 6613/106302 objects misplaced (6.221%) 2024-03-24T12:44:38.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:38.205 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:38.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:38.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:38.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:38.345 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:38.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:38.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:38.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:38.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:38.493 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:38.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:38.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:38.637 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:38.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:38.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:38.746 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:44:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:44:38.781 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:38.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:38.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:38.862 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:38 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:44:38.862 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:38 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:44:38.862 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:38 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:44:38.863 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:38 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:44:38.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:38.920 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:38.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:39.022 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:39.072 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:39.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:39.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:39.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:38 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:44:39.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:38 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:44:39.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:39.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:39.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:39.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:39.345 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:39.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:39.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:39.447 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:39.472 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:39.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:39.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:39.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:39.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:39.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:39.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:39.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:39.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:39.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:39.846 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:39.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:39.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:39.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:39.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:40.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:40.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:40.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:40.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:39 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:44:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:39 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:44:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:39 smithi190 ceph-mon[33748]: pgmap v808: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 6941/92136 objects degraded (7.533%); 5906/92136 objects misplaced (6.410%) 2024-03-24T12:44:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:39 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2223488406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:44:40.148 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:40.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:39 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:44:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:39 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:44:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:39 smithi029 ceph-mon[29919]: pgmap v808: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 6941/92136 objects degraded (7.533%); 5906/92136 objects misplaced (6.410%) 2024-03-24T12:44:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:39 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2223488406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:44:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:39 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:44:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:39 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:44:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:39 smithi029 ceph-mon[41500]: pgmap v808: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.9 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 6941/92136 objects degraded (7.533%); 5906/92136 objects misplaced (6.410%) 2024-03-24T12:44:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:39 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2223488406' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:44:40.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:40.290 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:40.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:40.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:40.431 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:40.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:40.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:40.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:40.546 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:44:40.546 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:44:40.546 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:44:40.546 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:44:40.546 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:44:40.546 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:44:40.547 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:44:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:44:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:44:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:44:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:44:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:44:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:44:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:44:40.547 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:44:40.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:40.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:40.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:40.665 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:40.708 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:40.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:40.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:40.840 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:40.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:40.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:40.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[29919]: 5.1b scrub starts 2024-03-24T12:44:40.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[29919]: 5.1b scrub ok 2024-03-24T12:44:40.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[29919]: 5.1c scrub starts 2024-03-24T12:44:40.919 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[29919]: 5.1c scrub ok 2024-03-24T12:44:40.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[29919]: 4.8 scrub starts 2024-03-24T12:44:40.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[29919]: 4.8 scrub ok 2024-03-24T12:44:40.920 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 6941/92136 objects degraded (7.533%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:40.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[41500]: 5.1b scrub starts 2024-03-24T12:44:40.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[41500]: 5.1b scrub ok 2024-03-24T12:44:40.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[41500]: 5.1c scrub starts 2024-03-24T12:44:40.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[41500]: 5.1c scrub ok 2024-03-24T12:44:40.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[41500]: 4.8 scrub starts 2024-03-24T12:44:40.920 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[41500]: 4.8 scrub ok 2024-03-24T12:44:40.921 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:40 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 6941/92136 objects degraded (7.533%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:40.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:40.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:40.978 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:40.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:41.079 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:41.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:41.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:40 smithi190 ceph-mon[33748]: 5.1b scrub starts 2024-03-24T12:44:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:40 smithi190 ceph-mon[33748]: 5.1b scrub ok 2024-03-24T12:44:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:40 smithi190 ceph-mon[33748]: 5.1c scrub starts 2024-03-24T12:44:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:40 smithi190 ceph-mon[33748]: 5.1c scrub ok 2024-03-24T12:44:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:40 smithi190 ceph-mon[33748]: 4.8 scrub starts 2024-03-24T12:44:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:40 smithi190 ceph-mon[33748]: 4.8 scrub ok 2024-03-24T12:44:41.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:40 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 6941/92136 objects degraded (7.533%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:41.217 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:41.252 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:41.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:41.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:41.353 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:41.403 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:41.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:41.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:41.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:44:41.532 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:44:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:41.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:41.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:41.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:41.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:41.697 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:41.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:41.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:41.831 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:41.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:41.933 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:41.968 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:41.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:41.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:42.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:42.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:42.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:42.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:41 smithi190 ceph-mon[33748]: 1.2 scrub starts 2024-03-24T12:44:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:41 smithi190 ceph-mon[33748]: 1.2 scrub ok 2024-03-24T12:44:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:41 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:44:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:41 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:44:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:41 smithi190 ceph-mon[33748]: pgmap v809: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 918 B/s rd, 0 op/s; 6337/87996 objects degraded (7.201%); 5482/87996 objects misplaced (6.230%) 2024-03-24T12:44:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:41 smithi029 ceph-mon[29919]: 1.2 scrub starts 2024-03-24T12:44:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:41 smithi029 ceph-mon[29919]: 1.2 scrub ok 2024-03-24T12:44:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:41 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:44:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:41 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:44:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:41 smithi029 ceph-mon[29919]: pgmap v809: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 918 B/s rd, 0 op/s; 6337/87996 objects degraded (7.201%); 5482/87996 objects misplaced (6.230%) 2024-03-24T12:44:42.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:41 smithi029 ceph-mon[41500]: 1.2 scrub starts 2024-03-24T12:44:42.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:41 smithi029 ceph-mon[41500]: 1.2 scrub ok 2024-03-24T12:44:42.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:41 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:44:42.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:41 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:44:42.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:41 smithi029 ceph-mon[41500]: pgmap v809: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 918 B/s rd, 0 op/s; 6337/87996 objects degraded (7.201%); 5482/87996 objects misplaced (6.230%) 2024-03-24T12:44:42.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:42.228 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:42.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:42.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:42.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:42.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:42.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:42.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:42.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:42.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:42.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:42.644 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:42.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:42.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:42.782 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:42.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:42.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:42.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:42.926 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:42.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:43.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:43.066 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:43.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:43.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:43.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:42 smithi190 ceph-mon[33748]: 3.1e scrub starts 2024-03-24T12:44:43.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:42 smithi190 ceph-mon[33748]: 3.1e scrub ok 2024-03-24T12:44:43.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:42 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:44:43.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:42 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:44:43.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:42 smithi029 ceph-mon[41500]: 3.1e scrub starts 2024-03-24T12:44:43.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:42 smithi029 ceph-mon[41500]: 3.1e scrub ok 2024-03-24T12:44:43.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:42 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:44:43.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:42 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:44:43.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:42 smithi029 ceph-mon[29919]: 3.1e scrub starts 2024-03-24T12:44:43.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:42 smithi029 ceph-mon[29919]: 3.1e scrub ok 2024-03-24T12:44:43.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:42 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:44:43.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:42 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:44:43.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:43.201 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:43.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:43.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:43.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:43.347 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:43.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:43.448 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:43.476 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:43.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:43.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:43.512 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:44:43.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:44:43.582 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:43.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:43.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:43.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:43.717 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:43.747 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:43.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:43.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:43.850 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:43.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:43.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:43.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:43.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:44.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:44.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:44.113 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:43 smithi190 ceph-mon[33748]: 6.9 scrub starts 2024-03-24T12:44:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:43 smithi190 ceph-mon[33748]: 6.9 scrub ok 2024-03-24T12:44:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:43 smithi190 ceph-mon[33748]: pgmap v810: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 484 op/s; 6134/80700 objects degraded (7.601%); 5276/80700 objects misplaced (6.538%) 2024-03-24T12:44:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:43 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:44:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:43 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:44:44.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:44.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:43 smithi029 ceph-mon[29919]: 6.9 scrub starts 2024-03-24T12:44:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:43 smithi029 ceph-mon[29919]: 6.9 scrub ok 2024-03-24T12:44:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:43 smithi029 ceph-mon[29919]: pgmap v810: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 484 op/s; 6134/80700 objects degraded (7.601%); 5276/80700 objects misplaced (6.538%) 2024-03-24T12:44:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:43 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:44:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:43 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:44:44.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:43 smithi029 ceph-mon[41500]: 6.9 scrub starts 2024-03-24T12:44:44.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:43 smithi029 ceph-mon[41500]: 6.9 scrub ok 2024-03-24T12:44:44.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:43 smithi029 ceph-mon[41500]: pgmap v810: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 484 op/s; 6134/80700 objects degraded (7.601%); 5276/80700 objects misplaced (6.538%) 2024-03-24T12:44:44.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:43 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:44:44.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:43 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:44:44.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:44.275 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:44.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:44.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:44.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:44.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:44.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:44.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:44.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:44.650 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:44.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:44.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:44.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:44.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:44.813 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:44.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:44.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:44.945 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:44.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:44.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:45.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:45.087 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:45.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:44 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:44:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:44 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:44:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:44 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:44:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:44 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:44:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:44 smithi190 ceph-mon[33748]: 5.2 scrub starts 2024-03-24T12:44:45.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:44 smithi190 ceph-mon[33748]: 5.2 scrub ok 2024-03-24T12:44:45.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:44:45.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:44:45.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:44:45.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:44:45.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[29919]: 5.2 scrub starts 2024-03-24T12:44:45.170 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[29919]: 5.2 scrub ok 2024-03-24T12:44:45.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:44:45.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:44:45.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:44:45.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:44:45.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[41500]: 5.2 scrub starts 2024-03-24T12:44:45.170 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:44 smithi029 ceph-mon[41500]: 5.2 scrub ok 2024-03-24T12:44:45.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:45.240 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:45.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:45.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:45.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:45.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:45.388 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:45.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:45.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:45.532 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:45.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:45.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:45.666 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:45.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:45.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:45.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:45.804 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:45.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:45.804 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:45 smithi029 ceph-mon[29919]: 5.0 scrub starts 2024-03-24T12:44:45.805 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:45 smithi029 ceph-mon[29919]: 5.0 scrub ok 2024-03-24T12:44:45.805 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:45 smithi029 ceph-mon[29919]: pgmap v811: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 692 op/s; 4894/69429 objects degraded (7.049%); 4273/69429 objects misplaced (6.154%) 2024-03-24T12:44:45.805 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:45 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 6134/80700 objects degraded (7.601%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:45.905 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:45.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:45.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:46.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:46.063 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:45 smithi029 ceph-mon[41500]: 5.0 scrub starts 2024-03-24T12:44:46.063 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:45 smithi029 ceph-mon[41500]: 5.0 scrub ok 2024-03-24T12:44:46.063 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:45 smithi029 ceph-mon[41500]: pgmap v811: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 692 op/s; 4894/69429 objects degraded (7.049%); 4273/69429 objects misplaced (6.154%) 2024-03-24T12:44:46.063 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:45 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 6134/80700 objects degraded (7.601%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:46.093 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:46.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:45 smithi190 ceph-mon[33748]: 5.0 scrub starts 2024-03-24T12:44:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:45 smithi190 ceph-mon[33748]: 5.0 scrub ok 2024-03-24T12:44:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:45 smithi190 ceph-mon[33748]: pgmap v811: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.4 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 692 op/s; 4894/69429 objects degraded (7.049%); 4273/69429 objects misplaced (6.154%) 2024-03-24T12:44:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:45 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 6134/80700 objects degraded (7.601%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:46.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:46.230 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:46.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:46.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:46.368 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:46.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:46.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:46.470 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:46.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:46.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:46.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:46.605 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:46.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:46.637 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:46.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:46.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:46.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:46.878 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:46.903 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:46 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:44:46.903 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:46 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:44:46.903 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:46 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:44:46.904 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:46 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:44:46.913 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:46.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:46.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:47.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:47.046 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:47.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:47.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:47.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:46 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:44:47.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:46 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:44:47.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:47.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:47.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:47.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:47.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:47.318 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:47.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:47.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:47.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:47.460 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:47.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:47.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:47.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:47.585 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a[29896]: 2024-03-24T12:44:47.220+0000 7f0eeb338700 -1 mon.a@0(leader).osd e352 definitely_dead 0 2024-03-24T12:44:47.600 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:47.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:47.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:47.732 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:47.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:47.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:47.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:47.837 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:44:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:44:47.867 INFO:teuthology.orchestra.run.smithi029.stderr:marked out osd.5. 2024-03-24T12:44:47.868 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:47.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:47.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:47.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:47.997 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:47.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:48.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:44:48.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:44:48.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[41500]: 5.18 scrub starts 2024-03-24T12:44:48.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[41500]: 5.18 scrub ok 2024-03-24T12:44:48.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:44:48.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:44:48.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[41500]: pgmap v812: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 4894/68196 objects degraded (7.176%); 4273/68196 objects misplaced (6.266%) 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/770189390' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[41500]: Client client.admin marked osd.5 out, after it was down for 529 seconds 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[29919]: 5.18 scrub starts 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[29919]: 5.18 scrub ok 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[29919]: pgmap v812: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 4894/68196 objects degraded (7.176%); 4273/68196 objects misplaced (6.266%) 2024-03-24T12:44:48.095 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/770189390' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-24T12:44:48.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-24T12:44:48.096 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:47 smithi029 ceph-mon[29919]: Client client.admin marked osd.5 out, after it was down for 529 seconds 2024-03-24T12:44:48.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:48.126 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:48.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:48.128 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:47 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:44:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:47 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:44:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:47 smithi190 ceph-mon[33748]: 5.18 scrub starts 2024-03-24T12:44:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:47 smithi190 ceph-mon[33748]: 5.18 scrub ok 2024-03-24T12:44:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:47 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:44:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:47 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:44:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:47 smithi190 ceph-mon[33748]: pgmap v812: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 26 active+clean+remapped, 107 active+clean, 21 active+undersized, 13 active+undersized+degraded; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 4894/68196 objects degraded (7.176%); 4273/68196 objects misplaced (6.266%) 2024-03-24T12:44:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:47 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/770189390' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-24T12:44:48.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:47 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-24T12:44:48.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:47 smithi190 ceph-mon[33748]: Client client.admin marked osd.5 out, after it was down for 529 seconds 2024-03-24T12:44:48.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:48.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:48.259 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:48.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:48.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:48.390 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:48.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:48.417 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:44:48.247+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:44:48.492 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:48.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:48.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:48.516 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:48.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:48.621 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:44:48.622 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:44:48.641 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:48.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:48.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:48.766 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:48.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:48.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:48.834 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:44:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:44:48.868 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:48.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:48.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:49.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:49.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:49.131 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-mon[41500]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T12:44:49.131 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-24T12:44:49.131 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-mon[41500]: osdmap e353: 8 total, 7 up, 7 in 2024-03-24T12:44:49.132 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:44:49.132 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-mon[29919]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T12:44:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-24T12:44:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-mon[29919]: osdmap e353: 8 total, 7 up, 7 in 2024-03-24T12:44:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:44:49.132 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:48 smithi190 ceph-mon[33748]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T12:44:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:48 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-24T12:44:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:48 smithi190 ceph-mon[33748]: osdmap e353: 8 total, 7 up, 7 in 2024-03-24T12:44:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:44:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:44:49.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:49.188 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:49.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:49.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:49.318 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:49.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:49.422 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:49.452 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:49.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:49.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:44:49.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:49.588 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:49.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:49.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:49.716 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:49.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:49.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:49.843 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:49.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:49.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:49.857 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:44:49.922 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:49.925 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[29919]: pgmap v814: 168 pgs: 1 remapped, 14 peering, 5 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 12 active+clean+remapped, 107 active+clean, 17 active+undersized, 11 active+undersized+degraded; 1.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 2588/54396 objects degraded (4.758%); 1930/54396 objects misplaced (3.548%) 2024-03-24T12:44:49.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[29919]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 352'12055 2024-03-24T12:44:49.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[29919]: 10.12 starting backfill to osd.7 from (0'0,0'0] MAX to 352'12109 2024-03-24T12:44:49.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[29919]: 10.15 starting backfill to osd.2 from (0'0,0'0] MAX to 352'12096 2024-03-24T12:44:49.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[29919]: osdmap e354: 8 total, 7 up, 7 in 2024-03-24T12:44:49.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[29919]: 10.16 starting backfill to osd.4 from (295'4804,337'10275] MIN to 352'12013 2024-03-24T12:44:49.927 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3466094081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:44:49.928 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:49.928 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[41500]: pgmap v814: 168 pgs: 1 remapped, 14 peering, 5 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 12 active+clean+remapped, 107 active+clean, 17 active+undersized, 11 active+undersized+degraded; 1.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 2588/54396 objects degraded (4.758%); 1930/54396 objects misplaced (3.548%) 2024-03-24T12:44:49.928 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[41500]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 352'12055 2024-03-24T12:44:49.928 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[41500]: 10.12 starting backfill to osd.7 from (0'0,0'0] MAX to 352'12109 2024-03-24T12:44:49.928 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[41500]: 10.15 starting backfill to osd.2 from (0'0,0'0] MAX to 352'12096 2024-03-24T12:44:49.928 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[41500]: osdmap e354: 8 total, 7 up, 7 in 2024-03-24T12:44:49.928 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[41500]: 10.16 starting backfill to osd.4 from (295'4804,337'10275] MIN to 352'12013 2024-03-24T12:44:49.928 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:49 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3466094081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:44:49.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:49.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:49.981 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:49.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:50.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:50.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:50.111 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:50.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:44:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:49 smithi190 ceph-mon[33748]: pgmap v814: 168 pgs: 1 remapped, 14 peering, 5 remapped+peering, 1 active+undersized+degraded+remapped+backfill_toofull, 12 active+clean+remapped, 107 active+clean, 17 active+undersized, 11 active+undersized+degraded; 1.1 GiB data, 8.0 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 2588/54396 objects degraded (4.758%); 1930/54396 objects misplaced (3.548%) 2024-03-24T12:44:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:49 smithi190 ceph-mon[33748]: 10.11 starting backfill to osd.6 from (0'0,0'0] MAX to 352'12055 2024-03-24T12:44:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:49 smithi190 ceph-mon[33748]: 10.12 starting backfill to osd.7 from (0'0,0'0] MAX to 352'12109 2024-03-24T12:44:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:49 smithi190 ceph-mon[33748]: 10.15 starting backfill to osd.2 from (0'0,0'0] MAX to 352'12096 2024-03-24T12:44:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:49 smithi190 ceph-mon[33748]: osdmap e354: 8 total, 7 up, 7 in 2024-03-24T12:44:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:49 smithi190 ceph-mon[33748]: 10.16 starting backfill to osd.4 from (295'4804,337'10275] MIN to 352'12013 2024-03-24T12:44:50.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:49 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3466094081' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:44:50.212 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:50.252 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:50.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:50.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:50.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:50.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:50.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:50.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:50.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:50.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:50.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:50.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:50.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:50.672 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:50.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:50.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:50.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:50.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:50.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:50.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:51.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:51.082 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:51.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:51.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:51.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:44:51.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3466094081' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 352'11963 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 352'12034 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: osdmap e355: 8 total, 7 up, 7 in 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 352'14966 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 352'12312 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 352'11995 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: 6.10 scrub starts 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: 6.10 scrub ok 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: 6.11 scrub starts 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 2588/54396 objects degraded (4.758%), 12 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:51.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:50 smithi190 ceph-mon[33748]: osdmap e356: 8 total, 7 up, 7 in 2024-03-24T12:44:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:44:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3466094081' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:44:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 352'11963 2024-03-24T12:44:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 352'12034 2024-03-24T12:44:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: osdmap e355: 8 total, 7 up, 7 in 2024-03-24T12:44:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 352'14966 2024-03-24T12:44:51.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 352'12312 2024-03-24T12:44:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 352'11995 2024-03-24T12:44:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: 6.10 scrub starts 2024-03-24T12:44:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: 6.10 scrub ok 2024-03-24T12:44:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: 6.11 scrub starts 2024-03-24T12:44:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 2588/54396 objects degraded (4.758%), 12 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:51.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[41500]: osdmap e356: 8 total, 7 up, 7 in 2024-03-24T12:44:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:44:51.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3466094081' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: 10.9 starting backfill to osd.4 from (0'0,0'0] MAX to 352'11963 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 352'12034 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: osdmap e355: 8 total, 7 up, 7 in 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: 10.a starting backfill to osd.2 from (0'0,0'0] MAX to 352'14966 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 352'12312 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 352'11995 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: 6.10 scrub starts 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: 6.10 scrub ok 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: 6.11 scrub starts 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 2588/54396 objects degraded (4.758%), 12 pgs degraded, 29 pgs undersized (PG_DEGRADED) 2024-03-24T12:44:51.169 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:50 smithi029 ceph-mon[29919]: osdmap e356: 8 total, 7 up, 7 in 2024-03-24T12:44:51.184 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:51.219 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:51.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:51.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:51.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:51.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:51.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:51.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:51.498 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:51.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:51.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:51.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:51.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:51.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:51.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:51.779 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:51.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:51.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:51.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:51.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:51.912 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:51.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:52.013 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:52.058 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:52.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:52.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:52.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:51 smithi190 ceph-mon[33748]: 4.7 deep-scrub starts 2024-03-24T12:44:52.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:51 smithi190 ceph-mon[33748]: 4.7 deep-scrub ok 2024-03-24T12:44:52.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:51 smithi190 ceph-mon[33748]: 6.11 scrub ok 2024-03-24T12:44:52.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:51 smithi190 ceph-mon[33748]: 3.16 scrub starts 2024-03-24T12:44:52.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:51 smithi190 ceph-mon[33748]: 3.16 scrub ok 2024-03-24T12:44:52.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:51 smithi190 ceph-mon[33748]: pgmap v818: 168 pgs: 1 active+recovery_wait+degraded, 1 remapped, 15 peering, 8 remapped+peering, 9 active+clean+remapped, 119 active+clean, 9 active+undersized, 6 active+undersized+degraded; 1.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 558/51462 objects degraded (1.084%); 514/51462 objects misplaced (0.999%); 8.4 MiB/s, 136 objects/s recovering 2024-03-24T12:44:52.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:51 smithi190 ceph-mon[33748]: osdmap e357: 8 total, 7 up, 7 in 2024-03-24T12:44:52.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:51 smithi190 ceph-mon[33748]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:44:52.149 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[41500]: 4.7 deep-scrub starts 2024-03-24T12:44:52.149 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[41500]: 4.7 deep-scrub ok 2024-03-24T12:44:52.149 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[41500]: 6.11 scrub ok 2024-03-24T12:44:52.149 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[41500]: 3.16 scrub starts 2024-03-24T12:44:52.149 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[41500]: 3.16 scrub ok 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[41500]: pgmap v818: 168 pgs: 1 active+recovery_wait+degraded, 1 remapped, 15 peering, 8 remapped+peering, 9 active+clean+remapped, 119 active+clean, 9 active+undersized, 6 active+undersized+degraded; 1.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 558/51462 objects degraded (1.084%); 514/51462 objects misplaced (0.999%); 8.4 MiB/s, 136 objects/s recovering 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[41500]: osdmap e357: 8 total, 7 up, 7 in 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[41500]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[29919]: 4.7 deep-scrub starts 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[29919]: 4.7 deep-scrub ok 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[29919]: 6.11 scrub ok 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[29919]: 3.16 scrub starts 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[29919]: 3.16 scrub ok 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[29919]: pgmap v818: 168 pgs: 1 active+recovery_wait+degraded, 1 remapped, 15 peering, 8 remapped+peering, 9 active+clean+remapped, 119 active+clean, 9 active+undersized, 6 active+undersized+degraded; 1.0 GiB data, 7.9 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 558/51462 objects degraded (1.084%); 514/51462 objects misplaced (0.999%); 8.4 MiB/s, 136 objects/s recovering 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[29919]: osdmap e357: 8 total, 7 up, 7 in 2024-03-24T12:44:52.150 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:51 smithi029 ceph-mon[29919]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-24T12:44:52.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:52.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:52.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:52.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:52.299 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:52.339 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:52.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:52.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:52.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:52.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:52.474 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:52.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:52.575 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:52.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:52.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:52.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:52.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:52.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:52.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:52.848 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:52.894 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:52.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:52.996 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:53.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:53.037 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:53.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:53.121 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T12:44:53.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:52 smithi190 ceph-mon[33748]: 10.19 scrub starts 2024-03-24T12:44:53.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:52 smithi190 ceph-mon[33748]: 10.19 scrub ok 2024-03-24T12:44:53.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:52 smithi190 ceph-mon[33748]: 6.1c scrub starts 2024-03-24T12:44:53.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:52 smithi190 ceph-mon[33748]: 6.1c scrub ok 2024-03-24T12:44:53.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:52 smithi190 ceph-mon[33748]: 3.5 scrub starts 2024-03-24T12:44:53.134 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:52 smithi190 ceph-mon[33748]: 3.5 scrub ok 2024-03-24T12:44:53.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:53.165 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[41500]: 10.19 scrub starts 2024-03-24T12:44:53.165 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[41500]: 10.19 scrub ok 2024-03-24T12:44:53.165 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[41500]: 6.1c scrub starts 2024-03-24T12:44:53.165 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[41500]: 6.1c scrub ok 2024-03-24T12:44:53.165 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[41500]: 3.5 scrub starts 2024-03-24T12:44:53.165 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[41500]: 3.5 scrub ok 2024-03-24T12:44:53.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[29919]: 10.19 scrub starts 2024-03-24T12:44:53.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[29919]: 10.19 scrub ok 2024-03-24T12:44:53.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[29919]: 6.1c scrub starts 2024-03-24T12:44:53.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[29919]: 6.1c scrub ok 2024-03-24T12:44:53.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[29919]: 3.5 scrub starts 2024-03-24T12:44:53.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:52 smithi029 ceph-mon[29919]: 3.5 scrub ok 2024-03-24T12:44:53.210 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:53.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:53.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:53.362 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:53.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:53.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:53.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:53.513 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:53.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:53.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:53.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:53.655 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:53.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:53.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:53.799 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:53.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:53.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:53.868 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:44:53.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:53.991 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:53.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:53.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:54.040 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[41500]: 4.2 scrub starts 2024-03-24T12:44:54.041 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[41500]: 4.2 scrub ok 2024-03-24T12:44:54.041 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[41500]: 5.4 deep-scrub starts 2024-03-24T12:44:54.041 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[41500]: 5.4 deep-scrub ok 2024-03-24T12:44:54.041 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[41500]: pgmap v820: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+degraded, 1 remapped, 15 peering, 8 remapped+peering, 142 active+clean; 976 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 584 op/s; 433/47394 objects degraded (0.914%); 16 MiB/s, 264 objects/s recovering 2024-03-24T12:44:54.041 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[41500]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:44:54.041 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[41500]: 4.3 scrub starts 2024-03-24T12:44:54.041 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[41500]: 4.3 scrub ok 2024-03-24T12:44:54.041 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:44:54.041 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[29919]: 4.2 scrub starts 2024-03-24T12:44:54.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[29919]: 4.2 scrub ok 2024-03-24T12:44:54.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[29919]: 5.4 deep-scrub starts 2024-03-24T12:44:54.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[29919]: 5.4 deep-scrub ok 2024-03-24T12:44:54.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[29919]: pgmap v820: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+degraded, 1 remapped, 15 peering, 8 remapped+peering, 142 active+clean; 976 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 584 op/s; 433/47394 objects degraded (0.914%); 16 MiB/s, 264 objects/s recovering 2024-03-24T12:44:54.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[29919]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:44:54.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[29919]: 4.3 scrub starts 2024-03-24T12:44:54.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[29919]: 4.3 scrub ok 2024-03-24T12:44:54.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:53 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:44:54.060 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7, 6] out_osds: [5] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:44:54.060 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:44:54.060 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 3, 4, 7, 6] 2024-03-24T12:44:54.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd out 6 2024-03-24T12:44:54.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:54.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:54.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:54.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:54.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:53 smithi190 ceph-mon[33748]: 4.2 scrub starts 2024-03-24T12:44:54.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:53 smithi190 ceph-mon[33748]: 4.2 scrub ok 2024-03-24T12:44:54.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:53 smithi190 ceph-mon[33748]: 5.4 deep-scrub starts 2024-03-24T12:44:54.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:53 smithi190 ceph-mon[33748]: 5.4 deep-scrub ok 2024-03-24T12:44:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:53 smithi190 ceph-mon[33748]: pgmap v820: 168 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+degraded, 1 remapped, 15 peering, 8 remapped+peering, 142 active+clean; 976 MiB data, 7.6 GiB used, 618 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 584 op/s; 433/47394 objects degraded (0.914%); 16 MiB/s, 264 objects/s recovering 2024-03-24T12:44:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:53 smithi190 ceph-mon[33748]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:44:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:53 smithi190 ceph-mon[33748]: 4.3 scrub starts 2024-03-24T12:44:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:53 smithi190 ceph-mon[33748]: 4.3 scrub ok 2024-03-24T12:44:54.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:53 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:44:54.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:54.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:54.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:54.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:54.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:54.420 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:54.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:54.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:54.567 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:54.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:54.590 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:44:54.604 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:44:54.615 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:44:54.634 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:44:54.643 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:44:54.667 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:44:54.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:54.687 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:44:54.694 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:54.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:54.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:54.709 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:44:54.732 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:44:54.756 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:44:54.756 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:44:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:44:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:44:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:44:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:44:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:44:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:44:54.757 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:44:54.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:54.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:54.821 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:54.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:54.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:54.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:55.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:55.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:55.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:55.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:54 smithi029 ceph-mon[29919]: 5.19 scrub starts 2024-03-24T12:44:55.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:54 smithi029 ceph-mon[29919]: 5.19 scrub ok 2024-03-24T12:44:55.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:54 smithi029 ceph-mon[29919]: osdmap e358: 8 total, 7 up, 7 in 2024-03-24T12:44:55.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:54 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2892317587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:44:55.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:54 smithi029 ceph-mon[41500]: 5.19 scrub starts 2024-03-24T12:44:55.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:54 smithi029 ceph-mon[41500]: 5.19 scrub ok 2024-03-24T12:44:55.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:54 smithi029 ceph-mon[41500]: osdmap e358: 8 total, 7 up, 7 in 2024-03-24T12:44:55.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:54 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2892317587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:44:55.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:55.244 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:55.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:55.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:55.346 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:55.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:55.378 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:55.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:55.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:54 smithi190 ceph-mon[33748]: 5.19 scrub starts 2024-03-24T12:44:55.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:54 smithi190 ceph-mon[33748]: 5.19 scrub ok 2024-03-24T12:44:55.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:54 smithi190 ceph-mon[33748]: osdmap e358: 8 total, 7 up, 7 in 2024-03-24T12:44:55.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:54 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2892317587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:44:55.479 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:55.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:55.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:55.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:55.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:55.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:55.629 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:55.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:55.730 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:55.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:55.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:55.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:55.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:55.882 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:55.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:55.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:56.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:56.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:56.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:56.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:56.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[29919]: 6.19 scrub starts 2024-03-24T12:44:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[29919]: 6.19 scrub ok 2024-03-24T12:44:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[29919]: 6.1d scrub starts 2024-03-24T12:44:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[29919]: 6.1d scrub ok 2024-03-24T12:44:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[29919]: 4.15 deep-scrub starts 2024-03-24T12:44:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[29919]: 4.15 deep-scrub ok 2024-03-24T12:44:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[29919]: pgmap v822: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+degraded, 2 peering, 159 active+clean; 823 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 820 B/s rd, 0 B/s wr, 817 op/s; 1663/40077 objects degraded (4.150%); 799/40077 objects misplaced (1.994%); 24 MiB/s, 384 objects/s recovering 2024-03-24T12:44:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[29919]: osdmap e359: 8 total, 7 up, 7 in 2024-03-24T12:44:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 1663/40077 objects degraded (4.150%), 5 pgs degraded (PG_DEGRADED) 2024-03-24T12:44:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[29919]: osdmap e360: 8 total, 7 up, 7 in 2024-03-24T12:44:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[41500]: 6.19 scrub starts 2024-03-24T12:44:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[41500]: 6.19 scrub ok 2024-03-24T12:44:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[41500]: 6.1d scrub starts 2024-03-24T12:44:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[41500]: 6.1d scrub ok 2024-03-24T12:44:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[41500]: 4.15 deep-scrub starts 2024-03-24T12:44:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[41500]: 4.15 deep-scrub ok 2024-03-24T12:44:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[41500]: pgmap v822: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+degraded, 2 peering, 159 active+clean; 823 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 820 B/s rd, 0 B/s wr, 817 op/s; 1663/40077 objects degraded (4.150%); 799/40077 objects misplaced (1.994%); 24 MiB/s, 384 objects/s recovering 2024-03-24T12:44:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[41500]: osdmap e359: 8 total, 7 up, 7 in 2024-03-24T12:44:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 1663/40077 objects degraded (4.150%), 5 pgs degraded (PG_DEGRADED) 2024-03-24T12:44:56.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:55 smithi029 ceph-mon[41500]: osdmap e360: 8 total, 7 up, 7 in 2024-03-24T12:44:56.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:56.263 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:56.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:56.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:56.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:55 smithi190 ceph-mon[33748]: 6.19 scrub starts 2024-03-24T12:44:56.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:55 smithi190 ceph-mon[33748]: 6.19 scrub ok 2024-03-24T12:44:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:55 smithi190 ceph-mon[33748]: 6.1d scrub starts 2024-03-24T12:44:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:55 smithi190 ceph-mon[33748]: 6.1d scrub ok 2024-03-24T12:44:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:55 smithi190 ceph-mon[33748]: 4.15 deep-scrub starts 2024-03-24T12:44:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:55 smithi190 ceph-mon[33748]: 4.15 deep-scrub ok 2024-03-24T12:44:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:55 smithi190 ceph-mon[33748]: pgmap v822: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 1 active+recovery_wait+degraded, 2 peering, 159 active+clean; 823 MiB data, 7.4 GiB used, 618 GiB / 626 GiB avail; 820 B/s rd, 0 B/s wr, 817 op/s; 1663/40077 objects degraded (4.150%); 799/40077 objects misplaced (1.994%); 24 MiB/s, 384 objects/s recovering 2024-03-24T12:44:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:55 smithi190 ceph-mon[33748]: osdmap e359: 8 total, 7 up, 7 in 2024-03-24T12:44:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:55 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 1663/40077 objects degraded (4.150%), 5 pgs degraded (PG_DEGRADED) 2024-03-24T12:44:56.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:55 smithi190 ceph-mon[33748]: osdmap e360: 8 total, 7 up, 7 in 2024-03-24T12:44:56.389 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:56.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:56.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:56.455 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:44:56.455 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:44:56.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:56.515 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:56.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:56.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:56.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:56.645 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:56.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:56.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:56.747 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:56.778 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:56.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:56.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:56.880 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:44:56.912 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:56.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:56.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:44:56.946 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[29919]: 3.1c scrub starts 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[29919]: 3.1c scrub ok 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[29919]: 10.3 scrub starts 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[29919]: 10.3 scrub ok 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[29919]: 5.12 scrub starts 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[29919]: 5.12 scrub ok 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[29919]: osdmap e361: 8 total, 7 up, 7 in 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[41500]: 3.1c scrub starts 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[41500]: 3.1c scrub ok 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[41500]: 10.3 scrub starts 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[41500]: 10.3 scrub ok 2024-03-24T12:44:56.947 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[41500]: 5.12 scrub starts 2024-03-24T12:44:56.948 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[41500]: 5.12 scrub ok 2024-03-24T12:44:56.948 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:56 smithi029 ceph-mon[41500]: osdmap e361: 8 total, 7 up, 7 in 2024-03-24T12:44:57.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:57.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:57.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:57.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:57.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:57.188 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:57.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:57.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:57.322 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:57.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:57.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:57.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:56 smithi190 ceph-mon[33748]: 3.1c scrub starts 2024-03-24T12:44:57.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:56 smithi190 ceph-mon[33748]: 3.1c scrub ok 2024-03-24T12:44:57.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:56 smithi190 ceph-mon[33748]: 10.3 scrub starts 2024-03-24T12:44:57.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:56 smithi190 ceph-mon[33748]: 10.3 scrub ok 2024-03-24T12:44:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:56 smithi190 ceph-mon[33748]: 5.12 scrub starts 2024-03-24T12:44:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:56 smithi190 ceph-mon[33748]: 5.12 scrub ok 2024-03-24T12:44:57.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:56 smithi190 ceph-mon[33748]: osdmap e361: 8 total, 7 up, 7 in 2024-03-24T12:44:57.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:57.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:57.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:57.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:57.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:57.623 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:57.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:57.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:57.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:44:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:44:57.725 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:57.758 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:57.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:57.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:57.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:57.897 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:57.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:57.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:58.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:58.041 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:58.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:58.143 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:58.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[41500]: 5.e scrub starts 2024-03-24T12:44:58.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[41500]: 5.e scrub ok 2024-03-24T12:44:58.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[41500]: 3.11 scrub starts 2024-03-24T12:44:58.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[41500]: 3.11 scrub ok 2024-03-24T12:44:58.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[41500]: pgmap v826: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 161 active+clean; 790 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1659/38484 objects degraded (4.311%); 799/38484 objects misplaced (2.076%); 13 MiB/s, 216 objects/s recovering 2024-03-24T12:44:58.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[41500]: 5.16 scrub starts 2024-03-24T12:44:58.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[41500]: 5.16 scrub ok 2024-03-24T12:44:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[29919]: 5.e scrub starts 2024-03-24T12:44:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[29919]: 5.e scrub ok 2024-03-24T12:44:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[29919]: 3.11 scrub starts 2024-03-24T12:44:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[29919]: 3.11 scrub ok 2024-03-24T12:44:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[29919]: pgmap v826: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 161 active+clean; 790 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1659/38484 objects degraded (4.311%); 799/38484 objects misplaced (2.076%); 13 MiB/s, 216 objects/s recovering 2024-03-24T12:44:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[29919]: 5.16 scrub starts 2024-03-24T12:44:58.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:57 smithi029 ceph-mon[29919]: 5.16 scrub ok 2024-03-24T12:44:58.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:58.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:58.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:58.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:57 smithi190 ceph-mon[33748]: 5.e scrub starts 2024-03-24T12:44:58.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:57 smithi190 ceph-mon[33748]: 5.e scrub ok 2024-03-24T12:44:58.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:57 smithi190 ceph-mon[33748]: 3.11 scrub starts 2024-03-24T12:44:58.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:57 smithi190 ceph-mon[33748]: 3.11 scrub ok 2024-03-24T12:44:58.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:57 smithi190 ceph-mon[33748]: pgmap v826: 168 pgs: 2 active+remapped+backfill_toofull, 1 active+undersized+degraded+remapped+backfilling, 3 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 161 active+clean; 790 MiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1659/38484 objects degraded (4.311%); 799/38484 objects misplaced (2.076%); 13 MiB/s, 216 objects/s recovering 2024-03-24T12:44:58.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:57 smithi190 ceph-mon[33748]: 5.16 scrub starts 2024-03-24T12:44:58.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:57 smithi190 ceph-mon[33748]: 5.16 scrub ok 2024-03-24T12:44:58.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:58.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:58.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:58.444 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:58.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:58.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:58.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:44:58.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:58.586 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:58.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:58.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:44:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:44:58.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:58.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:58.734 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:58.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:58.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:58.877 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:58.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:58.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:44:59.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:59.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:59.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:44:59.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:59.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:58 smithi190 ceph-mon[33748]: 6.17 scrub starts 2024-03-24T12:44:59.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:58 smithi190 ceph-mon[33748]: 6.17 scrub ok 2024-03-24T12:44:59.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:58 smithi190 ceph-mon[33748]: 10.a scrub starts 2024-03-24T12:44:59.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:58 smithi190 ceph-mon[33748]: 3.4 scrub starts 2024-03-24T12:44:59.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:58 smithi190 ceph-mon[33748]: 6.12 scrub starts 2024-03-24T12:44:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:58 smithi190 ceph-mon[33748]: 3.4 scrub ok 2024-03-24T12:44:59.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:58 smithi190 ceph-mon[33748]: 6.12 scrub ok 2024-03-24T12:44:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:59.144 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:59.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[29919]: 6.17 scrub starts 2024-03-24T12:44:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[29919]: 6.17 scrub ok 2024-03-24T12:44:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[29919]: 10.a scrub starts 2024-03-24T12:44:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[29919]: 3.4 scrub starts 2024-03-24T12:44:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[29919]: 6.12 scrub starts 2024-03-24T12:44:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[29919]: 3.4 scrub ok 2024-03-24T12:44:59.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[29919]: 6.12 scrub ok 2024-03-24T12:44:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[41500]: 6.17 scrub starts 2024-03-24T12:44:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[41500]: 6.17 scrub ok 2024-03-24T12:44:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[41500]: 10.a scrub starts 2024-03-24T12:44:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[41500]: 3.4 scrub starts 2024-03-24T12:44:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[41500]: 6.12 scrub starts 2024-03-24T12:44:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[41500]: 3.4 scrub ok 2024-03-24T12:44:59.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:58 smithi029 ceph-mon[41500]: 6.12 scrub ok 2024-03-24T12:44:59.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:59.288 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:59.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:44:59.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:44:59.420 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:59.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:44:59.522 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:44:59.546 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:44:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:59.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:44:59.648 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:44:59.690 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:59.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:59.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:44:59.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:44:59.832 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:59.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:59.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:44:59.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:44:59.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:44:59.980 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:44:59.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:00.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:00.128 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:00.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[29919]: 10.a scrub ok 2024-03-24T12:45:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[29919]: 10.b deep-scrub starts 2024-03-24T12:45:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:45:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[29919]: pgmap v827: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 160 active+clean; 596 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 901/29178 objects degraded (3.088%); 552/29178 objects misplaced (1.892%); 4.9 MiB/s, 79 objects/s recovering 2024-03-24T12:45:00.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:45:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[41500]: 10.a scrub ok 2024-03-24T12:45:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[41500]: 10.b deep-scrub starts 2024-03-24T12:45:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:45:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[41500]: pgmap v827: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 160 active+clean; 596 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 901/29178 objects degraded (3.088%); 552/29178 objects misplaced (1.892%); 4.9 MiB/s, 79 objects/s recovering 2024-03-24T12:45:00.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:44:59 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:45:00.230 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:00.259 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:00.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:00.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:00.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:59 smithi190 ceph-mon[33748]: 10.a scrub ok 2024-03-24T12:45:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:59 smithi190 ceph-mon[33748]: 10.b deep-scrub starts 2024-03-24T12:45:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:59 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:45:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:59 smithi190 ceph-mon[33748]: pgmap v827: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 1 peering, 160 active+clean; 596 MiB data, 6.7 GiB used, 619 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 901/29178 objects degraded (3.088%); 552/29178 objects misplaced (1.892%); 4.9 MiB/s, 79 objects/s recovering 2024-03-24T12:45:00.387 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:44:59 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:45:00.403 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:00.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:00.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:00.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:00.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:00.542 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:00.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:00.642 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:00.678 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:00.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:00.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:00.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:00.823 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:00.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:00.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:00.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:00.963 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:00.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:01.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:01.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:01.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:01.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:01.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[41500]: 6.7 scrub starts 2024-03-24T12:45:01.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[41500]: 6.7 scrub ok 2024-03-24T12:45:01.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:45:01.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:45:01.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[41500]: 10.f scrub starts 2024-03-24T12:45:01.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[41500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:45:01.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 901/29178 objects degraded (3.088%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:01.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[41500]: osdmap e362: 8 total, 7 up, 7 in 2024-03-24T12:45:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[29919]: 6.7 scrub starts 2024-03-24T12:45:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[29919]: 6.7 scrub ok 2024-03-24T12:45:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:45:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:45:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[29919]: 10.f scrub starts 2024-03-24T12:45:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[29919]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:45:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 901/29178 objects degraded (3.088%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:01.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:00 smithi029 ceph-mon[29919]: osdmap e362: 8 total, 7 up, 7 in 2024-03-24T12:45:01.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:01.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:01.239 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:01.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:01.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:01.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:01.382 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:01.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:00 smithi190 ceph-mon[33748]: 6.7 scrub starts 2024-03-24T12:45:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:00 smithi190 ceph-mon[33748]: 6.7 scrub ok 2024-03-24T12:45:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:00 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:45:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:00 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:45:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:00 smithi190 ceph-mon[33748]: 10.f scrub starts 2024-03-24T12:45:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:00 smithi190 ceph-mon[33748]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:45:01.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:00 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 901/29178 objects degraded (3.088%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:01.384 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:00 smithi190 ceph-mon[33748]: osdmap e362: 8 total, 7 up, 7 in 2024-03-24T12:45:01.483 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:01.511 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:01.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:01.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:01.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:01.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:01.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:01.739 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:01.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:01.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:01.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:01.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:01.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:01.897 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:01.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:01.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:02.038 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:02.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:02.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:02.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[29919]: 6.d deep-scrub starts 2024-03-24T12:45:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[29919]: 10.f scrub ok 2024-03-24T12:45:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[29919]: 6.d deep-scrub ok 2024-03-24T12:45:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[29919]: 5.6 scrub starts 2024-03-24T12:45:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[29919]: 5.6 scrub ok 2024-03-24T12:45:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[29919]: pgmap v829: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 161 active+clean; 539 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 862 B/s rd, 0 op/s; 772/26415 objects degraded (2.923%); 552/26415 objects misplaced (2.090%) 2024-03-24T12:45:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:45:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:45:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[41500]: 6.d deep-scrub starts 2024-03-24T12:45:02.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[41500]: 10.f scrub ok 2024-03-24T12:45:02.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[41500]: 6.d deep-scrub ok 2024-03-24T12:45:02.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[41500]: 5.6 scrub starts 2024-03-24T12:45:02.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[41500]: 5.6 scrub ok 2024-03-24T12:45:02.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[41500]: pgmap v829: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 161 active+clean; 539 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 862 B/s rd, 0 op/s; 772/26415 objects degraded (2.923%); 552/26415 objects misplaced (2.090%) 2024-03-24T12:45:02.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:45:02.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:01 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:45:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:02.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:02.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:02.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:02.320 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:02.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:02.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:01 smithi190 ceph-mon[33748]: 6.d deep-scrub starts 2024-03-24T12:45:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:01 smithi190 ceph-mon[33748]: 10.f scrub ok 2024-03-24T12:45:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:01 smithi190 ceph-mon[33748]: 6.d deep-scrub ok 2024-03-24T12:45:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:01 smithi190 ceph-mon[33748]: 5.6 scrub starts 2024-03-24T12:45:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:01 smithi190 ceph-mon[33748]: 5.6 scrub ok 2024-03-24T12:45:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:01 smithi190 ceph-mon[33748]: pgmap v829: 168 pgs: 2 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 161 active+clean; 539 MiB data, 6.6 GiB used, 619 GiB / 626 GiB avail; 862 B/s rd, 0 op/s; 772/26415 objects degraded (2.923%); 552/26415 objects misplaced (2.090%) 2024-03-24T12:45:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:01 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:45:02.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:01 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:45:02.421 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:02.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:02.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:02.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:02.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:02.596 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:02.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:02.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:02.732 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:02.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:02.834 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:02.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:02.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:02.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:02.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:03.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:03.013 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:03.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:03.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:03.138 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:03.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:03.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:03.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:02 smithi029 ceph-mon[29919]: 10.b deep-scrub ok 2024-03-24T12:45:03.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:02 smithi029 ceph-mon[29919]: osdmap e363: 8 total, 7 up, 7 in 2024-03-24T12:45:03.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:02 smithi029 ceph-mon[41500]: 10.b deep-scrub ok 2024-03-24T12:45:03.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:02 smithi029 ceph-mon[41500]: osdmap e363: 8 total, 7 up, 7 in 2024-03-24T12:45:03.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:03.267 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:03.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:03.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:03.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:02 smithi190 ceph-mon[33748]: 10.b deep-scrub ok 2024-03-24T12:45:03.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:02 smithi190 ceph-mon[33748]: osdmap e363: 8 total, 7 up, 7 in 2024-03-24T12:45:03.400 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:03.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:03.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:03.540 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:03.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:03.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:03.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:45:03.264+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:45:03.679 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:03.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:03.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:03.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:03.818 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:03.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:03.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:03.966 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:03.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:04.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:04.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:04.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:04.193 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:04.219 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:04.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:04.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:04.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:04.357 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:04.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:03 smithi190 ceph-mon[33748]: 3.17 scrub starts 2024-03-24T12:45:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:03 smithi190 ceph-mon[33748]: 3.17 scrub ok 2024-03-24T12:45:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:03 smithi190 ceph-mon[33748]: pgmap v831: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 450 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 772/22164 objects degraded (3.483%); 552/22164 objects misplaced (2.491%) 2024-03-24T12:45:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:03 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:45:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:03 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:45:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[41500]: 3.17 scrub starts 2024-03-24T12:45:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[41500]: 3.17 scrub ok 2024-03-24T12:45:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[41500]: pgmap v831: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 450 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 772/22164 objects degraded (3.483%); 552/22164 objects misplaced (2.491%) 2024-03-24T12:45:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:45:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:45:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:04.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[29919]: 3.17 scrub starts 2024-03-24T12:45:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[29919]: 3.17 scrub ok 2024-03-24T12:45:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[29919]: pgmap v831: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 450 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s; 772/22164 objects degraded (3.483%); 552/22164 objects misplaced (2.491%) 2024-03-24T12:45:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:45:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:45:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:04.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:04.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:04.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:04.497 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:04.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:04.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:04.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:04.631 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:04.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:04.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:04.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:04.776 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:04.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:04.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:04.913 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:04.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:05.014 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:05.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:05.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:05.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:05.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:05.186 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:05.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:05.287 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:05.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:05.313 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:05.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:04 smithi190 ceph-mon[33748]: 6.4 deep-scrub starts 2024-03-24T12:45:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:04 smithi190 ceph-mon[33748]: 6.4 deep-scrub ok 2024-03-24T12:45:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:04 smithi190 ceph-mon[33748]: 6.1a scrub starts 2024-03-24T12:45:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:04 smithi190 ceph-mon[33748]: 6.1a scrub ok 2024-03-24T12:45:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:04 smithi190 ceph-mon[33748]: osdmap e364: 8 total, 7 up, 7 in 2024-03-24T12:45:05.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:04 smithi029 ceph-mon[29919]: 6.4 deep-scrub starts 2024-03-24T12:45:05.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:04 smithi029 ceph-mon[29919]: 6.4 deep-scrub ok 2024-03-24T12:45:05.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:04 smithi029 ceph-mon[29919]: 6.1a scrub starts 2024-03-24T12:45:05.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:04 smithi029 ceph-mon[29919]: 6.1a scrub ok 2024-03-24T12:45:05.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:04 smithi029 ceph-mon[29919]: osdmap e364: 8 total, 7 up, 7 in 2024-03-24T12:45:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:04 smithi029 ceph-mon[41500]: 6.4 deep-scrub starts 2024-03-24T12:45:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:04 smithi029 ceph-mon[41500]: 6.4 deep-scrub ok 2024-03-24T12:45:05.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:04 smithi029 ceph-mon[41500]: 6.1a scrub starts 2024-03-24T12:45:05.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:04 smithi029 ceph-mon[41500]: 6.1a scrub ok 2024-03-24T12:45:05.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:04 smithi029 ceph-mon[41500]: osdmap e364: 8 total, 7 up, 7 in 2024-03-24T12:45:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:05.453 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:05.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:05.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:05.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:05.578 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:05.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:05.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:05.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:05.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:05.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:05.815 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:05.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:05.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:05.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:05.949 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:05.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:05.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:05.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:06.000 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Removed 126971 objects 2024-03-24T12:45:06.000 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Clean up completed and total clean up time :151.7 2024-03-24T12:45:06.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:06.107 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:06.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:06.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:06.235 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:06.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:06.336 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:06.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:06.363 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:06.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:05 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:45:06.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:05 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:45:06.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:05 smithi190 ceph-mon[33748]: pgmap v833: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 223 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 250/11274 objects degraded (2.217%); 193/11274 objects misplaced (1.712%) 2024-03-24T12:45:06.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:05 smithi190 ceph-mon[33748]: osdmap e365: 8 total, 7 up, 7 in 2024-03-24T12:45:06.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:05 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 250/11274 objects degraded (2.217%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:06.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:05 smithi190 ceph-mon[33748]: osdmap e366: 8 total, 7 up, 7 in 2024-03-24T12:45:06.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:05 smithi190 ceph-mon[33748]: 3.1f deep-scrub starts 2024-03-24T12:45:06.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:05 smithi190 ceph-mon[33748]: 3.1f deep-scrub ok 2024-03-24T12:45:06.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:45:06.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:45:06.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[29919]: pgmap v833: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 223 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 250/11274 objects degraded (2.217%); 193/11274 objects misplaced (1.712%) 2024-03-24T12:45:06.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[29919]: osdmap e365: 8 total, 7 up, 7 in 2024-03-24T12:45:06.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 250/11274 objects degraded (2.217%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:06.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[29919]: osdmap e366: 8 total, 7 up, 7 in 2024-03-24T12:45:06.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[29919]: 3.1f deep-scrub starts 2024-03-24T12:45:06.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[29919]: 3.1f deep-scrub ok 2024-03-24T12:45:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:45:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:45:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[41500]: pgmap v833: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 223 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 250/11274 objects degraded (2.217%); 193/11274 objects misplaced (1.712%) 2024-03-24T12:45:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[41500]: osdmap e365: 8 total, 7 up, 7 in 2024-03-24T12:45:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 250/11274 objects degraded (2.217%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[41500]: osdmap e366: 8 total, 7 up, 7 in 2024-03-24T12:45:06.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[41500]: 3.1f deep-scrub starts 2024-03-24T12:45:06.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:05 smithi029 ceph-mon[41500]: 3.1f deep-scrub ok 2024-03-24T12:45:06.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:06.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:06.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:06.592 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:06.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:06.617 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:06.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:06.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:06.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:06.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:06.848 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:06.873 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:06.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:06.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:07.001 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:07.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:07.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:07.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:07.128 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:07.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:07.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:07.253 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:07.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:07.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:07.379 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:07.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:07.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:07.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:07.606 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:07.634 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:07.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:07.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:07.736 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:07.762 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:07.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:07.863 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:07.892 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:07.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:07.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:07.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:45:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:45:07.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:08.020 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:08.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:08.115 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-24T12:45:08.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-24T12:45:08.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[41500]: pgmap v836: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 189 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s; 250/9648 objects degraded (2.591%); 193/9648 objects misplaced (2.000%) 2024-03-24T12:45:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[41500]: 5.1a scrub starts 2024-03-24T12:45:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[41500]: 5.1a scrub ok 2024-03-24T12:45:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[41500]: 5.d scrub starts 2024-03-24T12:45:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[41500]: 5.d scrub ok 2024-03-24T12:45:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:45:08.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:45:08.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[29919]: pgmap v836: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 189 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s; 250/9648 objects degraded (2.591%); 193/9648 objects misplaced (2.000%) 2024-03-24T12:45:08.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[29919]: 5.1a scrub starts 2024-03-24T12:45:08.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[29919]: 5.1a scrub ok 2024-03-24T12:45:08.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[29919]: 5.d scrub starts 2024-03-24T12:45:08.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[29919]: 5.d scrub ok 2024-03-24T12:45:08.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:45:08.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:07 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:45:08.182 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:08.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:08.259 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:45:08.259 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:45:08.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:45:08.260 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:45:08.260 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.260 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104312 9.b 9:d3e4c742:::benchmark_data_smithi029_134815_object104311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:45:08.260 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.099394+0000", 2024-03-24T12:45:08.260 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 418.16223245200001, 2024-03-24T12:45:08.260 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1656896990000001, 2024-03-24T12:45:08.260 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.260 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.260 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104312 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099394+0000", 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.261 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099394+0000", 2024-03-24T12:45:08.262 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.262 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099395+0000", 2024-03-24T12:45:08.262 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2249999999999999e-06 2024-03-24T12:45:08.262 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.262 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.262 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099446+0000", 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0138e-05 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099455+0000", 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8849999999999994e-06 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.263 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099464+0000", 2024-03-24T12:45:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7309999999999999e-06 2024-03-24T12:45:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099570+0000", 2024-03-24T12:45:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105568 2024-03-24T12:45:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.264 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.099579+0000", 2024-03-24T12:45:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1190000000000003e-06 2024-03-24T12:45:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073282+0000", 2024-03-24T12:45:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.97370271399999997 2024-03-24T12:45:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.265 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073293+0000", 2024-03-24T12:45:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1156e-05 2024-03-24T12:45:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.266 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.225494+0000", 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1522010009999999 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.225625+0000", 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013125300000000001 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.267 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.226198+0000", 2024-03-24T12:45:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00057334900000000002 2024-03-24T12:45:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.241089+0000", 2024-03-24T12:45:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.014890944 2024-03-24T12:45:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.268 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250249+0000", 2024-03-24T12:45:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0091596839999999995 2024-03-24T12:45:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.264890+0000", 2024-03-24T12:45:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.014640604 2024-03-24T12:45:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.269 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.264958+0000", 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8332999999999998e-05 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265084+0000", 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000125995 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.270 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104318 9.b 9:d43bf6ba:::benchmark_data_smithi029_134815_object104317:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.109135+0000", 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 418.15249201300003, 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1563207370000002, 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104318 2024-03-24T12:45:08.271 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109135+0000", 2024-03-24T12:45:08.272 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.273 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109135+0000", 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109135+0000", 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4700000000000001e-07 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.274 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109181+0000", 2024-03-24T12:45:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5513999999999998e-05 2024-03-24T12:45:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109188+0000", 2024-03-24T12:45:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.644e-06 2024-03-24T12:45:08.275 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.109195+0000", 2024-03-24T12:45:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2400000000000004e-06 2024-03-24T12:45:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.276 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.111000+0000", 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0018054429999999999 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.111011+0000", 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.134e-05 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073314+0000", 2024-03-24T12:45:08.277 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.96230284899999996 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073321+0000", 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.002e-06 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.241134+0000", 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.167813244 2024-03-24T12:45:08.278 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.241273+0000", 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013887400000000001 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.241450+0000", 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000176226 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.279 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250310+0000", 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0088607200000000007 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257152+0000", 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.006841387 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.280 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265168+0000", 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0080164239999999994 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265349+0000", 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018142200000000001 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.281 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265455+0000", 2024-03-24T12:45:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010586099999999999 2024-03-24T12:45:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.282 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104334 9.b 9:df2efe4b:::benchmark_data_smithi029_134815_object104333:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:45:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.128443+0000", 2024-03-24T12:45:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 418.13318341600001, 2024-03-24T12:45:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1374394969999999, 2024-03-24T12:45:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.283 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104334 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128443+0000", 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128443+0000", 2024-03-24T12:45:08.284 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128445+0000", 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.516e-06 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128495+0000", 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9907999999999997e-05 2024-03-24T12:45:08.285 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128504+0000", 2024-03-24T12:45:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3510000000000008e-06 2024-03-24T12:45:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128511+0000", 2024-03-24T12:45:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3889999999999999e-06 2024-03-24T12:45:08.286 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128580+0000", 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9028000000000004e-05 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128591+0000", 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0794e-05 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.287 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073342+0000", 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94475095399999998 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073349+0000", 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5220000000000004e-06 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.288 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.244214+0000", 2024-03-24T12:45:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.170865115 2024-03-24T12:45:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.244346+0000", 2024-03-24T12:45:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000131777 2024-03-24T12:45:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.289 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.244514+0000", 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016814999999999999 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250351+0000", 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0058376089999999997 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257198+0000", 2024-03-24T12:45:08.290 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0068467659999999998 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265468+0000", 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0082702160000000004 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265777+0000", 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030915599999999999 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265883+0000", 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105246 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104336 9.b 9:d1900f74:::benchmark_data_smithi029_134815_object104335:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.129851+0000", 2024-03-24T12:45:08.292 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 418.13177561999998, 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1363790969999998, 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104336 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129851+0000", 2024-03-24T12:45:08.293 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129851+0000", 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129852+0000", 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.009e-06 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.294 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129902+0000", 2024-03-24T12:45:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9784000000000003e-05 2024-03-24T12:45:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129922+0000", 2024-03-24T12:45:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0543e-05 2024-03-24T12:45:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.129936+0000", 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3386e-05 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.130013+0000", 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7442999999999997e-05 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.296 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.130023+0000", 2024-03-24T12:45:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8519999999999999e-06 2024-03-24T12:45:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.297 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073367+0000", 2024-03-24T12:45:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94334440600000002 2024-03-24T12:45:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.299 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073374+0000", 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3500000000000002e-06 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.247267+0000", 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.17389327 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.300 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.247413+0000", 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000146064 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.247597+0000", 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000183914 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250393+0000", 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027959180000000001 2024-03-24T12:45:08.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257238+0000", 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0068453170000000001 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.265896+0000", 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086582129999999997 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.302 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.266117+0000", 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00022031700000000001 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.266230+0000", 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113311 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104365 9.b 9:d74e8c62:::benchmark_data_smithi029_134815_object104364:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.178308+0000", 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 418.08331862199998, 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.089454752, 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104365 2024-03-24T12:45:08.304 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178308+0000", 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178308+0000", 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.305 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178310+0000", 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.795e-06 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178334+0000", 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3892000000000001e-05 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.306 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178355+0000", 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1787e-05 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178366+0000", 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0705e-05 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178440+0000", 2024-03-24T12:45:08.307 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4006999999999998e-05 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.178452+0000", 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1494999999999999e-05 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073397+0000", 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.89494537900000004 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.308 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073404+0000", 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5069999999999997e-06 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250424+0000", 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1770203910000001 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.309 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250552+0000", 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012809099999999999 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.250738+0000", 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018606100000000001 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257280+0000", 2024-03-24T12:45:08.310 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0065419579999999996 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257325+0000", 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5188999999999997e-05 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.266239+0000", 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0089141340000000006 2024-03-24T12:45:08.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.267645+0000", 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0014058460000000001 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.267763+0000", 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000117515 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.312 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104372 9.b 9:d40eb526:::benchmark_data_smithi029_134815_object104371:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.195928+0000", 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 418.065698365, 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0720185619999998, 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.313 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104372 2024-03-24T12:45:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195928+0000", 2024-03-24T12:45:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.314 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195928+0000", 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195929+0000", 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2359999999999999e-06 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195950+0000", 2024-03-24T12:45:08.315 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0792e-05 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195962+0000", 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1226999999999999e-05 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.195970+0000", 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7239999999999998e-06 2024-03-24T12:45:08.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.196035+0000", 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4597999999999994e-05 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.196047+0000", 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1941e-05 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.317 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073425+0000", 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.87737854400000004 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.073432+0000", 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6429999999999998e-06 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.318 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.253708+0000", 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1802758440000001 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.253849+0000", 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014084199999999999 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.254020+0000", 2024-03-24T12:45:08.319 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000171261 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257368+0000", 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0033482170000000001 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.257415+0000", 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6925000000000002e-05 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.267778+0000", 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010363140999999999 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.267833+0000", 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4313e-05 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.321 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.267947+0000", 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000114314 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172139 9.b 9:dc246dd6:::benchmark_data_smithi029_134815_object60073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.674435+0000", 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 334.58719198199998, 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4605824350000001, 2024-03-24T12:45:08.322 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172139 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674435+0000", 2024-03-24T12:45:08.323 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674435+0000", 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674436+0000", 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5900000000000005e-07 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674440+0000", 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0860000000000004e-06 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674453+0000", 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3131e-05 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.325 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.674461+0000", 2024-03-24T12:45:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4849999999999992e-06 2024-03-24T12:45:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680515+0000", 2024-03-24T12:45:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060537869999999997 2024-03-24T12:45:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.326 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680647+0000", 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000131624 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.681361+0000", 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00071422300000000005 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.691102+0000", 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0097411960000000006 2024-03-24T12:45:08.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.691182+0000", 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0073000000000004e-05 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745633+0000", 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.054451039999999999 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745655+0000", 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1627000000000001e-05 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750438+0000", 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0047831860000001 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750458+0000", 2024-03-24T12:45:08.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0426000000000001e-05 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.134965+0000", 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.384506811 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.135017+0000", 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1780999999999998e-05 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.330 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172143 9.b 9:d403d143:::benchmark_data_smithi029_134815_object60077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.680416+0000", 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 334.58121034200002, 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4554223509999999, 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.331 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172143 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680416+0000", 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.332 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680416+0000", 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680417+0000", 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0240000000000001e-06 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680421+0000", 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9299999999999996e-06 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680453+0000", 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1365999999999998e-05 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.680463+0000", 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0066e-05 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.684801+0000", 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.004338639 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.684935+0000", 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013339999999999999 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.685063+0000", 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000128718 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.691380+0000", 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0063161479999999997 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.691418+0000", 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8167999999999998e-05 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.336 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745728+0000", 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.054309907999999997 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745737+0000", 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6549999999999999e-06 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750567+0000", 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.004829897 2024-03-24T12:45:08.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750579+0000", 2024-03-24T12:45:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1317e-05 2024-03-24T12:45:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.135741+0000", 2024-03-24T12:45:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38516228699999999 2024-03-24T12:45:08.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.135839+0000", 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7827999999999999e-05 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172149 9.b 9:d2e9c91b:::benchmark_data_smithi029_134815_object60083:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.682707+0000", 2024-03-24T12:45:08.339 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 334.57891933000002, 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.453752683, 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172149 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.340 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682707+0000", 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682707+0000", 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682708+0000", 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8500000000000002e-07 2024-03-24T12:45:08.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682710+0000", 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4210000000000002e-06 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682716+0000", 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8699999999999997e-06 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.682721+0000", 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0830000000000003e-06 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.692301+0000", 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.009580224 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.692458+0000", 2024-03-24T12:45:08.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015688200000000001 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.692594+0000", 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013530700000000001 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.695140+0000", 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002546338 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.695394+0000", 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025376799999999997 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745799+0000", 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.050405028999999997 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.345 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745824+0000", 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4890000000000001e-05 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750612+0000", 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0047884199999999 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750621+0000", 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3330000000000003e-06 2024-03-24T12:45:08.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136400+0000", 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38577870399999997 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136460+0000", 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9929000000000002e-05 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.347 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172172 9.b 9:d5266a5d:::benchmark_data_smithi029_134815_object60106:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.697248+0000", 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 334.56437874599999, 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4392927250000001, 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.348 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172172 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697248+0000", 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697248+0000", 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697249+0000", 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.46e-06 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697254+0000", 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0259999999999998e-06 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.350 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697270+0000", 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5804e-05 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697278+0000", 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1620000000000008e-06 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697424+0000", 2024-03-24T12:45:08.351 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014525700000000001 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697569+0000", 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014558799999999999 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.697705+0000", 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013550699999999999 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.352 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.700575+0000", 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028701170000000002 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.701307+0000", 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073196299999999997 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745878+0000", 2024-03-24T12:45:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.044571350000000003 2024-03-24T12:45:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745889+0000", 2024-03-24T12:45:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0577999999999999e-05 2024-03-24T12:45:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.354 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750652+0000", 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.004763383 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750661+0000", 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1179999999999998e-06 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136490+0000", 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38582929500000002 2024-03-24T12:45:08.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.356 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136541+0000", 2024-03-24T12:45:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0117000000000001e-05 2024-03-24T12:45:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.357 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172194 9.b 9:d9b9c02c:::benchmark_data_smithi029_134815_object60128:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.718569+0000", 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 334.54305718000001, 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4180509909999999, 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.358 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172194 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718569+0000", 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718569+0000", 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718570+0000", 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9800000000000002e-07 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718575+0000", 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8670000000000002e-06 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718586+0000", 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0492000000000001e-05 2024-03-24T12:45:08.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718592+0000", 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.711e-06 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718636+0000", 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4610999999999997e-05 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718740+0000", 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010396899999999999 2024-03-24T12:45:08.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.718811+0000", 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0983999999999997e-05 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.721173+0000", 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0023618150000000002 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.362 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.721297+0000", 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000124263 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745952+0000", 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.024654850999999998 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.745963+0000", 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0806e-05 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.363 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750691+0000", 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.004728445 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750700+0000", 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0659999999999993e-06 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136572+0000", 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38587177099999997 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136620+0000", 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8341999999999999e-05 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172198 9.b 9:d32e9228:::benchmark_data_smithi029_134815_object60132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.725029+0000", 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 334.53659741799999, 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.41167083, 2024-03-24T12:45:08.365 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172198 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725029+0000", 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.366 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725029+0000", 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725030+0000", 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.850000000000001e-07 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725035+0000", 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.459e-06 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725045+0000", 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0522e-05 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725053+0000", 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9270000000000005e-06 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725103+0000", 2024-03-24T12:45:08.368 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0380999999999998e-05 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725193+0000", 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9495999999999998e-05 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.725270+0000", 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7470000000000002e-05 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.369 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.727388+0000", 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002118048 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.727403+0000", 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4909000000000001e-05 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.746084+0000", 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.018681039999999999 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.746095+0000", 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0424999999999999e-05 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750731+0000", 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.004635827 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750740+0000", 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0299999999999999e-06 2024-03-24T12:45:08.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136651+0000", 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38591176799999999 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136700+0000", 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8542999999999999e-05 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.372 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172205 9.b 9:d50f1c31:::benchmark_data_smithi029_134815_object60139:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.794173+0000", 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 334.46745332400002, 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3523102300000001, 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172205 2024-03-24T12:45:08.373 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794173+0000", 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794173+0000", 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.374 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794175+0000", 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2359999999999999e-06 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794179+0000", 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6829999999999999e-06 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794190+0000", 2024-03-24T12:45:08.375 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.046e-05 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794198+0000", 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5459999999999992e-06 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794253+0000", 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5228e-05 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.794263+0000", 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0016e-05 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750765+0000", 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.95650199599999997 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750773+0000", 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9689999999999994e-06 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136726+0000", 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38595213699999997 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136829+0000", 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010315900000000001 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.136948+0000", 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000119073 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146237+0000", 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0092890230000000004 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146250+0000", 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3419e-05 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146339+0000", 2024-03-24T12:45:08.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8263999999999994e-05 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146386+0000", 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7568000000000003e-05 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146484+0000", 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7453000000000004e-05 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172221 9.b 9:df06907a:::benchmark_data_smithi029_134815_object60155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.872329+0000", 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 334.38929802500002, 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2750098759999999, 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172221 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.381 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872329+0000", 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872329+0000", 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872330+0000", 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.807e-06 2024-03-24T12:45:08.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872338+0000", 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5460000000000003e-06 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872355+0000", 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6626999999999999e-05 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872365+0000", 2024-03-24T12:45:08.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0359e-05 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872441+0000", 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6285000000000006e-05 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.872454+0000", 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2812e-05 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750796+0000", 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.87834196399999998 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.750804+0000", 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8630000000000008e-06 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.139760+0000", 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38895642000000002 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.139863+0000", 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010245 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.139975+0000", 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000112643 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146256+0000", 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0062804260000000004 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146292+0000", 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6346000000000001e-05 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147145+0000", 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00085249000000000002 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147213+0000", 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8195999999999996e-05 2024-03-24T12:45:08.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147338+0000", 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000125642 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172223 9.b 9:dc9df42b:::benchmark_data_smithi029_134815_object60157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.874452+0000", 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 334.38717430899999, 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.273117195, 2024-03-24T12:45:08.388 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172223 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874452+0000", 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.389 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874452+0000", 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874454+0000", 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4699999999999999e-06 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874461+0000", 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8129999999999998e-06 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874479+0000", 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8745000000000001e-05 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874490+0000", 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0209e-05 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874624+0000", 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013484099999999999 2024-03-24T12:45:08.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.874638+0000", 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3506000000000001e-05 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.751235+0000", 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.87659726599999999 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.751246+0000", 2024-03-24T12:45:08.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0856000000000001e-05 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.142969+0000", 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.39172287300000003 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.143104+0000", 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013542599999999999 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:45:08.393 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.143213+0000", 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000108628 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146305+0000", 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0030918769999999998 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.146613+0000", 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030804699999999999 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.394 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147399+0000", 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00078603199999999999 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147459+0000", 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9757999999999999e-05 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.147570+0000", 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011084800000000001 2024-03-24T12:45:08.395 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:114605 10.4 10:242b400c:::benchmark_data_smithi029_160664_object114604:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:42:22.373681+0000", 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 165.887945634, 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.40016160000000001, 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:08.396 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 114605 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373681+0000", 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373681+0000", 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373683+0000", 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.57e-06 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373705+0000", 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2201999999999999e-05 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373717+0000", 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2068e-05 2024-03-24T12:45:08.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373726+0000", 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7819999999999999e-06 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.373907+0000", 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018135699999999999 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.374046+0000", 2024-03-24T12:45:08.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013883600000000001 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.374190+0000", 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000144299 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.380596+0000", 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.006406017 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.381497+0000", 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00090137699999999997 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.773726+0000", 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.39222809199999997 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.773788+0000", 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2230999999999994e-05 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.773843+0000", 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4768999999999997e-05 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:114621 10.4 10:24abf80b:::benchmark_data_smithi029_160664_object114620:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:42:22.383519+0000", 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 165.87810734199999, 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.400178706, 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.402 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 114621 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.383519+0000", 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.383519+0000", 2024-03-24T12:45:08.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.383520+0000", 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.065e-06 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.383541+0000", 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0275000000000002e-05 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.383570+0000", 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9005000000000001e-05 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.383578+0000", 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2819999999999996e-06 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.383634+0000", 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6496000000000003e-05 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.405 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.406 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.383761+0000", 2024-03-24T12:45:08.406 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012685800000000001 2024-03-24T12:45:08.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.406 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:07 smithi190 ceph-mon[33748]: pgmap v836: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 189 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.14k op/s; 250/9648 objects degraded (2.591%); 193/9648 objects misplaced (2.000%) 2024-03-24T12:45:08.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:07 smithi190 ceph-mon[33748]: 5.1a scrub starts 2024-03-24T12:45:08.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:07 smithi190 ceph-mon[33748]: 5.1a scrub ok 2024-03-24T12:45:08.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:07 smithi190 ceph-mon[33748]: 5.d scrub starts 2024-03-24T12:45:08.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:07 smithi190 ceph-mon[33748]: 5.d scrub ok 2024-03-24T12:45:08.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:07 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:45:08.407 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:07 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.383926+0000", 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016503199999999999 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.387393+0000", 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0034670870000000002 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.387579+0000", 2024-03-24T12:45:08.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000185599 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.783575+0000", 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.39599642699999998 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.783647+0000", 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1219999999999999e-05 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.783698+0000", 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1360000000000003e-05 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:114632 10.4 10:273c04af:::benchmark_data_smithi029_160664_object114631:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:42:22.394402+0000", 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 165.867224193, 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38942293300000003, 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.410 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 114632 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.394402+0000", 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.394402+0000", 2024-03-24T12:45:08.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.394404+0000", 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.519e-06 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.394425+0000", 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0803000000000001e-05 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.394438+0000", 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3589e-05 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.394447+0000", 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0080000000000008e-06 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.394505+0000", 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7745999999999999e-05 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.394659+0000", 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000153533 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.395018+0000", 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00035902599999999997 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.397758+0000", 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027398779999999998 2024-03-24T12:45:08.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.398109+0000", 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000351522 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.783743+0000", 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38563426000000001 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.783789+0000", 2024-03-24T12:45:08.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5315000000000002e-05 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.783825+0000", 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6733999999999998e-05 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:114637 10.4 10:25f311ac:::benchmark_data_smithi029_160664_object114636:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:42:22.398481+0000", 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 165.863145293, 2024-03-24T12:45:08.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.388927474, 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 114637 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.398481+0000", 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.398481+0000", 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.398483+0000", 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.53e-06 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.398504+0000", 2024-03-24T12:45:08.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1627000000000001e-05 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.398519+0000", 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4252e-05 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.398527+0000", 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.619e-06 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.398581+0000", 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3860999999999998e-05 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.398744+0000", 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016292800000000001 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.398940+0000", 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019598400000000001 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.401460+0000", 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0025201590000000001 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.401582+0000", 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000121549 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.787300+0000", 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.38571768699999998 2024-03-24T12:45:08.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.787356+0000", 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6567999999999998e-05 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:22.787409+0000", 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2710000000000002e-05 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:138933 10.4 10:2294c5bb:::benchmark_data_smithi029_160664_object11959:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e318)", 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:42:50.266404+0000", 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 137.99522275999999, 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.454978139, 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 138933 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.423 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.266404+0000", 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.266404+0000", 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.266405+0000", 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5820000000000001e-06 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.266411+0000", 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0519999999999998e-06 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.266426+0000", 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4066000000000001e-05 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.266436+0000", 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0205e-05 2024-03-24T12:45:08.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.516773+0000", 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.25033710100000001 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.516915+0000", 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014185200000000001 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 3,4", 2024-03-24T12:45:08.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.517054+0000", 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000139092 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.522543+0000", 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0054891519999999997 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.522587+0000", 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3735000000000001e-05 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:08.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.721270+0000", 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.198683784 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.721318+0000", 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7070999999999997e-05 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:42:50.721382+0000", 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4447000000000001e-05 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.428 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:08.429 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:08.429 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:45:08.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:08.430 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:08.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:08.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:08.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:08.556 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:08.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:08.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:08.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:08.686 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:08.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:08.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:08.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:08.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:45:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:45:08.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:08.942 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:08.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:08.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:09.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:09.068 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:09.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:09.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:45:09.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:45:09.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[41500]: osdmap e367: 8 total, 7 up, 7 in 2024-03-24T12:45:09.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[41500]: 4.19 deep-scrub starts 2024-03-24T12:45:09.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[41500]: 4.19 deep-scrub ok 2024-03-24T12:45:09.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[41500]: 5.15 scrub starts 2024-03-24T12:45:09.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[41500]: 5.15 scrub ok 2024-03-24T12:45:09.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:45:09.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:45:09.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[29919]: osdmap e367: 8 total, 7 up, 7 in 2024-03-24T12:45:09.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[29919]: 4.19 deep-scrub starts 2024-03-24T12:45:09.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[29919]: 4.19 deep-scrub ok 2024-03-24T12:45:09.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[29919]: 5.15 scrub starts 2024-03-24T12:45:09.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:08 smithi029 ceph-mon[29919]: 5.15 scrub ok 2024-03-24T12:45:09.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:09.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:09.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:09.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:09.246 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:08 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:45:09.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:08 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:45:09.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:08 smithi190 ceph-mon[33748]: osdmap e367: 8 total, 7 up, 7 in 2024-03-24T12:45:09.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:08 smithi190 ceph-mon[33748]: 4.19 deep-scrub starts 2024-03-24T12:45:09.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:08 smithi190 ceph-mon[33748]: 4.19 deep-scrub ok 2024-03-24T12:45:09.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:08 smithi190 ceph-mon[33748]: 5.15 scrub starts 2024-03-24T12:45:09.247 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:08 smithi190 ceph-mon[33748]: 5.15 scrub ok 2024-03-24T12:45:09.303 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:09.329 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:09.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:09.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:09.456 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:09.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:09.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:09.586 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:09.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:09.687 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:09.715 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:09.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:09.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:09.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:09.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:09.841 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:09.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:09.942 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:09.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:09.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:09.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:10.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:10.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:10.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:10.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:10.228 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:10.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:10.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:10.330 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:10.354 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:10.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:10.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:10.370 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:10.370 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[29919]: pgmap v838: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 21 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 799 op/s; 65/1563 objects degraded (4.159%) 2024-03-24T12:45:10.370 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:45:10.370 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[29919]: 5.b scrub starts 2024-03-24T12:45:10.370 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[29919]: 5.b scrub ok 2024-03-24T12:45:10.370 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:45:10.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:10.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[41500]: pgmap v838: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 21 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 799 op/s; 65/1563 objects degraded (4.159%) 2024-03-24T12:45:10.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:45:10.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[41500]: 5.b scrub starts 2024-03-24T12:45:10.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[41500]: 5.b scrub ok 2024-03-24T12:45:10.371 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:09 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:45:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:09 smithi190 ceph-mon[33748]: pgmap v838: 168 pgs: 2 active+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 21 MiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 799 op/s; 65/1563 objects degraded (4.159%) 2024-03-24T12:45:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:09 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:45:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:09 smithi190 ceph-mon[33748]: 5.b scrub starts 2024-03-24T12:45:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:09 smithi190 ceph-mon[33748]: 5.b scrub ok 2024-03-24T12:45:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:09 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:45:10.456 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:10.484 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:10.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:10.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:10.610 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:10.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:10.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:10.712 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:10.737 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:10.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:10.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:10.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T12:45:10.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:10.866 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:10.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:10.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:10.992 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:10.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:10.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:11.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:11.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:45:11.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[29919]: 4.11 scrub ok 2024-03-24T12:45:11.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 65/1563 objects degraded (4.159%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:11.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[29919]: osdmap e368: 8 total, 7 up, 7 in 2024-03-24T12:45:11.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:45:11.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:45:11.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:45:11.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[41500]: 4.11 scrub ok 2024-03-24T12:45:11.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 65/1563 objects degraded (4.159%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:11.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[41500]: osdmap e368: 8 total, 7 up, 7 in 2024-03-24T12:45:11.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:45:11.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:10 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:45:11.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:11.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:11.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:11.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:11.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:11.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:10 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:45:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:10 smithi190 ceph-mon[33748]: 4.11 scrub ok 2024-03-24T12:45:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:10 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 65/1563 objects degraded (4.159%), 3 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:10 smithi190 ceph-mon[33748]: osdmap e368: 8 total, 7 up, 7 in 2024-03-24T12:45:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:10 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:45:11.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:10 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:45:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:11.387 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:11.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:11.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:11.512 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:11.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:11.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:11.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:11.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:11.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a[29896]: 2024-03-24T12:45:11.469+0000 7f0eeb338700 -1 mon.a@0(leader).osd e368 definitely_dead 0 2024-03-24T12:45:11.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:11.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:11.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:11.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:11.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:11.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:11.910 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:11.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:12.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:12.045 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:12.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:12.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:12.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:12.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-mon[41500]: pgmap v840: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 3.3 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 870 B/s rd, 0 B/s wr, 512 op/s; 43/723 objects degraded (5.947%) 2024-03-24T12:45:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-mon[41500]: 5.13 scrub starts 2024-03-24T12:45:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-mon[41500]: 5.13 scrub ok 2024-03-24T12:45:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4109533799' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-24T12:45:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-mon[41500]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-24T12:45:12.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-mon[29919]: pgmap v840: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 3.3 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 870 B/s rd, 0 B/s wr, 512 op/s; 43/723 objects degraded (5.947%) 2024-03-24T12:45:12.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-mon[29919]: 5.13 scrub starts 2024-03-24T12:45:12.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-mon[29919]: 5.13 scrub ok 2024-03-24T12:45:12.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4109533799' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-24T12:45:12.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:11 smithi029 ceph-mon[29919]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-24T12:45:12.171 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:12.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:marked out osd.6. 2024-03-24T12:45:12.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:12.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:12.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:11 smithi190 ceph-mon[33748]: pgmap v840: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 163 active+clean; 3.3 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 870 B/s rd, 0 B/s wr, 512 op/s; 43/723 objects degraded (5.947%) 2024-03-24T12:45:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:11 smithi190 ceph-mon[33748]: 5.13 scrub starts 2024-03-24T12:45:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:11 smithi190 ceph-mon[33748]: 5.13 scrub ok 2024-03-24T12:45:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:11 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4109533799' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-03-24T12:45:12.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:11 smithi190 ceph-mon[33748]: Client client.admin marked osd.6 out, while it was still marked up 2024-03-24T12:45:12.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:12.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:12.423 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:12.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:12.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:12.552 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:12.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:12.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:12.679 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:12.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:12.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:12.805 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:12.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:12.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:12.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:12.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:12.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:13.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:13.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:13.062 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:13.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:13.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:13.188 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:13.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:13.219 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:45:13.219 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:12 smithi029 ceph-mon[41500]: 5.7 scrub starts 2024-03-24T12:45:13.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 5.7 scrub ok 2024-03-24T12:45:13.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 5.a deep-scrub starts 2024-03-24T12:45:13.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 5.a deep-scrub ok 2024-03-24T12:45:13.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4109533799' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-24T12:45:13.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: osdmap e369: 8 total, 7 up, 6 in 2024-03-24T12:45:13.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2124771405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:45:13.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 3.12 scrub starts 2024-03-24T12:45:13.220 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 3.12 scrub ok 2024-03-24T12:45:13.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:12 smithi029 ceph-mon[29919]: 5.7 scrub starts 2024-03-24T12:45:13.220 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 5.7 scrub ok 2024-03-24T12:45:13.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 5.a deep-scrub starts 2024-03-24T12:45:13.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 5.a deep-scrub ok 2024-03-24T12:45:13.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4109533799' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-24T12:45:13.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: osdmap e369: 8 total, 7 up, 6 in 2024-03-24T12:45:13.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2124771405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:45:13.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 3.12 scrub starts 2024-03-24T12:45:13.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 3.12 scrub ok 2024-03-24T12:45:13.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:13.327 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:13.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:13.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:12 smithi190 ceph-mon[33748]: 5.7 scrub starts 2024-03-24T12:45:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:12 smithi190 ceph-mon[33748]: 5.7 scrub ok 2024-03-24T12:45:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:12 smithi190 ceph-mon[33748]: 5.a deep-scrub starts 2024-03-24T12:45:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:12 smithi190 ceph-mon[33748]: 5.a deep-scrub ok 2024-03-24T12:45:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:12 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4109533799' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-03-24T12:45:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:12 smithi190 ceph-mon[33748]: osdmap e369: 8 total, 7 up, 6 in 2024-03-24T12:45:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:12 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2124771405' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:45:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:12 smithi190 ceph-mon[33748]: 3.12 scrub starts 2024-03-24T12:45:13.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:12 smithi190 ceph-mon[33748]: 3.12 scrub ok 2024-03-24T12:45:13.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:13.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:13.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:13.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:13.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:13.582 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:13.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:13.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:13.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:13.709 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:13.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:13.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:13.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:13.835 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:13.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:13.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:13.961 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:13.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:14.062 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:14.087 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:14.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:14.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:14.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:14.213 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:14.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:14.314 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:14.346 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:14.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:45:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:45:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: pgmap v842: 168 pgs: 30 peering, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 130 active+clean; 3.3 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 6.5 KiB/s rd, 37 KiB/s wr, 509 op/s; 43/723 objects degraded (5.947%); 35 KiB/s, 0 objects/s recovering 2024-03-24T12:45:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:45:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:45:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T12:45:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:45:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2124771405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:45:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: osdmap e370: 8 total, 7 up, 6 in 2024-03-24T12:45:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: 10.16 starting backfill to osd.2 from (0'0,0'0] MAX to 366'12545 2024-03-24T12:45:14.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:13 smithi190 ceph-mon[33748]: 10.16 continuing backfill to osd.4 from (309'7488,366'12545] MIN to 366'12545 2024-03-24T12:45:14.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:45:14.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:45:14.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: pgmap v842: 168 pgs: 30 peering, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 130 active+clean; 3.3 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 6.5 KiB/s rd, 37 KiB/s wr, 509 op/s; 43/723 objects degraded (5.947%); 35 KiB/s, 0 objects/s recovering 2024-03-24T12:45:14.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2124771405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: osdmap e370: 8 total, 7 up, 6 in 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 10.16 starting backfill to osd.2 from (0'0,0'0] MAX to 366'12545 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[41500]: 10.16 continuing backfill to osd.4 from (309'7488,366'12545] MIN to 366'12545 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: pgmap v842: 168 pgs: 30 peering, 1 active+undersized+remapped+backfill_toofull, 2 active+remapped+backfill_toofull, 3 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 130 active+clean; 3.3 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 6.5 KiB/s rd, 37 KiB/s wr, 509 op/s; 43/723 objects degraded (5.947%); 35 KiB/s, 0 objects/s recovering 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:45:14.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:45:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-03-24T12:45:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:45:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2124771405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:45:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: osdmap e370: 8 total, 7 up, 6 in 2024-03-24T12:45:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 10.16 starting backfill to osd.2 from (0'0,0'0] MAX to 366'12545 2024-03-24T12:45:14.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:13 smithi029 ceph-mon[29919]: 10.16 continuing backfill to osd.4 from (309'7488,366'12545] MIN to 366'12545 2024-03-24T12:45:14.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:14.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:14.472 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:14.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:14.573 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:14.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:14.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:14.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:14.700 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:14.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:14.725 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:14.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:14.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:14.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:14.850 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:14.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:14.951 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:14.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:14.976 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:14.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:15.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:15.102 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:15.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:15.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:15.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:15.235 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:15.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:15.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:15.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:15.362 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:15.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:15.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:15.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:15.487 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:15.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:15.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:15.613 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:15.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:15.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 366'12901 2024-03-24T12:45:15.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: osdmap e371: 8 total, 7 up, 6 in 2024-03-24T12:45:15.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 366'15946 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 366'22158 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.12 starting backfill to osd.3 from (0'0,0'0] MAX to 366'12634 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.17 starting backfill to osd.1 from (0'0,0'0] MAX to 366'12875 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.7 continuing backfill to osd.2 from (309'7664,366'12814] MIN to 366'12814 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.7 starting backfill to osd.4 from (0'0,0'0] MAX to 366'12814 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 366'12872 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 366'16261 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 366'21571 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.11 starting backfill to osd.2 from (0'0,0'0] MAX to 366'12556 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.12 continuing backfill to osd.7 from (309'7480,366'12634] MIN to 366'12634 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: 10.a starting backfill to osd.3 from (0'0,0'0] MAX to 366'16050 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 43/723 objects degraded (5.947%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:15.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:15 smithi190 ceph-mon[33748]: osdmap e372: 8 total, 7 up, 6 in 2024-03-24T12:45:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 366'12901 2024-03-24T12:45:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: osdmap e371: 8 total, 7 up, 6 in 2024-03-24T12:45:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 366'15946 2024-03-24T12:45:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 366'22158 2024-03-24T12:45:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.12 starting backfill to osd.3 from (0'0,0'0] MAX to 366'12634 2024-03-24T12:45:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.17 starting backfill to osd.1 from (0'0,0'0] MAX to 366'12875 2024-03-24T12:45:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.7 continuing backfill to osd.2 from (309'7664,366'12814] MIN to 366'12814 2024-03-24T12:45:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.7 starting backfill to osd.4 from (0'0,0'0] MAX to 366'12814 2024-03-24T12:45:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 366'12872 2024-03-24T12:45:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 366'16261 2024-03-24T12:45:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 366'21571 2024-03-24T12:45:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.11 starting backfill to osd.2 from (0'0,0'0] MAX to 366'12556 2024-03-24T12:45:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.12 continuing backfill to osd.7 from (309'7480,366'12634] MIN to 366'12634 2024-03-24T12:45:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: 10.a starting backfill to osd.3 from (0'0,0'0] MAX to 366'16050 2024-03-24T12:45:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 43/723 objects degraded (5.947%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[41500]: osdmap e372: 8 total, 7 up, 6 in 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.10 starting backfill to osd.0 from (0'0,0'0] MAX to 366'12901 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: osdmap e371: 8 total, 7 up, 6 in 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 366'15946 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 366'22158 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.12 starting backfill to osd.3 from (0'0,0'0] MAX to 366'12634 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.17 starting backfill to osd.1 from (0'0,0'0] MAX to 366'12875 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.7 continuing backfill to osd.2 from (309'7664,366'12814] MIN to 366'12814 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.7 starting backfill to osd.4 from (0'0,0'0] MAX to 366'12814 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 366'12872 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 366'16261 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 366'21571 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.11 starting backfill to osd.2 from (0'0,0'0] MAX to 366'12556 2024-03-24T12:45:15.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.12 continuing backfill to osd.7 from (309'7480,366'12634] MIN to 366'12634 2024-03-24T12:45:15.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: 10.a starting backfill to osd.3 from (0'0,0'0] MAX to 366'16050 2024-03-24T12:45:15.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 43/723 objects degraded (5.947%), 2 pgs degraded (PG_DEGRADED) 2024-03-24T12:45:15.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:15 smithi029 ceph-mon[29919]: osdmap e372: 8 total, 7 up, 6 in 2024-03-24T12:45:15.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:15.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:15.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:15.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:15.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:15.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:15.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:16.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:16.118 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:16.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:16.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:16.254 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:16.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:16.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:16.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:16.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:16 smithi029 ceph-mon[41500]: pgmap v845: 168 pgs: 1 active+clean+remapped, 1 active+undersized+remapped+backfill_toofull, 2 active+undersized+remapped, 52 peering, 1 active+remapped+backfill_toofull, 7 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 102 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 19 KiB/s rd, 142 KiB/s wr, 24 op/s; 43/723 objects degraded (5.947%); 180 KiB/s, 0 objects/s recovering 2024-03-24T12:45:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:16 smithi029 ceph-mon[41500]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:45:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:16 smithi029 ceph-mon[41500]: osdmap e373: 8 total, 7 up, 6 in 2024-03-24T12:45:16.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:16 smithi029 ceph-mon[29919]: pgmap v845: 168 pgs: 1 active+clean+remapped, 1 active+undersized+remapped+backfill_toofull, 2 active+undersized+remapped, 52 peering, 1 active+remapped+backfill_toofull, 7 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 102 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 19 KiB/s rd, 142 KiB/s wr, 24 op/s; 43/723 objects degraded (5.947%); 180 KiB/s, 0 objects/s recovering 2024-03-24T12:45:16.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:16 smithi029 ceph-mon[29919]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:45:16.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:16 smithi029 ceph-mon[29919]: osdmap e373: 8 total, 7 up, 6 in 2024-03-24T12:45:16.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:16.506 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:16.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:16.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:16.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:16.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:16 smithi190 ceph-mon[33748]: pgmap v845: 168 pgs: 1 active+clean+remapped, 1 active+undersized+remapped+backfill_toofull, 2 active+undersized+remapped, 52 peering, 1 active+remapped+backfill_toofull, 7 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 102 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 19 KiB/s rd, 142 KiB/s wr, 24 op/s; 43/723 objects degraded (5.947%); 180 KiB/s, 0 objects/s recovering 2024-03-24T12:45:16.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:16 smithi190 ceph-mon[33748]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-24T12:45:16.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:16 smithi190 ceph-mon[33748]: osdmap e373: 8 total, 7 up, 6 in 2024-03-24T12:45:16.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:16.633 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:16.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:16.734 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:16.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:16.759 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:16.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:16.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:16.884 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:16.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:16.986 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:17.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:17.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:17.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:17.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:17.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:17.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:17.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:17.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:17.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:17.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:17.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:17.387 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:17.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:17.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:17 smithi029 ceph-mon[41500]: pgmap v848: 168 pgs: 2 active+clean+remapped, 1 active+undersized+remapped+backfill_toofull, 2 active+undersized+remapped, 52 peering, 1 active+remapped+backfill_toofull, 7 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 101 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 13 KiB/s rd, 110 KiB/s wr, 16 op/s; 43/723 objects degraded (5.947%); 156 KiB/s, 6 objects/s recovering 2024-03-24T12:45:17.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:17 smithi029 ceph-mon[29919]: pgmap v848: 168 pgs: 2 active+clean+remapped, 1 active+undersized+remapped+backfill_toofull, 2 active+undersized+remapped, 52 peering, 1 active+remapped+backfill_toofull, 7 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 101 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 13 KiB/s rd, 110 KiB/s wr, 16 op/s; 43/723 objects degraded (5.947%); 156 KiB/s, 6 objects/s recovering 2024-03-24T12:45:17.488 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:17.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:17.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:17.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:17.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:17.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:17 smithi190 ceph-mon[33748]: pgmap v848: 168 pgs: 2 active+clean+remapped, 1 active+undersized+remapped+backfill_toofull, 2 active+undersized+remapped, 52 peering, 1 active+remapped+backfill_toofull, 7 unknown, 2 active+undersized+degraded+remapped+backfill_toofull, 101 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 13 KiB/s rd, 110 KiB/s wr, 16 op/s; 43/723 objects degraded (5.947%); 156 KiB/s, 6 objects/s recovering 2024-03-24T12:45:17.639 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:17.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:17.722 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:45:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:45:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:45:17.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:17.765 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:17.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:17.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:17.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:17.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:17.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:18.021 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:18.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:18.080 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:18.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:18.146 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:18.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:18.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:18.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:18.275 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:18.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:18.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:18.401 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:18.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:18.447 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:45:18 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:45:18.277+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:45:18.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:18.527 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:18.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:18.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:18.657 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:18.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:18.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:18.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:18.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:18.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:18.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:45:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:45:18.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:18.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:18.913 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:18.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:19.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:19.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:19.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:19.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:19.167 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:19.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:19.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:19.293 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:19.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:19.394 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:19.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:19.420 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:19.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:19.468 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:19 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:19.468 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:19 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:19.468 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:19.468 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:19 smithi029 ceph-mon[29919]: pgmap v849: 168 pgs: 2 active+clean+remapped, 1 active+undersized+remapped+backfill_toofull, 2 active+undersized+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 546 B/s rd, 0 op/s; 43/723 objects degraded (5.947%); 30/723 objects misplaced (4.149%); 103 KiB/s, 14 objects/s recovering 2024-03-24T12:45:19.468 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:19 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:19.468 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:19 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:19.469 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:19.469 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:19 smithi029 ceph-mon[41500]: pgmap v849: 168 pgs: 2 active+clean+remapped, 1 active+undersized+remapped+backfill_toofull, 2 active+undersized+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 546 B/s rd, 0 op/s; 43/723 objects degraded (5.947%); 30/723 objects misplaced (4.149%); 103 KiB/s, 14 objects/s recovering 2024-03-24T12:45:19.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:19.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:19.545 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:19.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:19.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:19 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:19.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:19 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:19.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:19.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:19 smithi190 ceph-mon[33748]: pgmap v849: 168 pgs: 2 active+clean+remapped, 1 active+undersized+remapped+backfill_toofull, 2 active+undersized+remapped, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 156 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 546 B/s rd, 0 op/s; 43/723 objects degraded (5.947%); 30/723 objects misplaced (4.149%); 103 KiB/s, 14 objects/s recovering 2024-03-24T12:45:19.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:19.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:19.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:19.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:19.687 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:45:19.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:45:19.726 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:45:19.746 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:45:19.767 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:45:19.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:19.843 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:45:19.843 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:45:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:45:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:45:19.844 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:45:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:45:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:45:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:45:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:45:19.845 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:45:19.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:19.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:19.970 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:19.999 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:19.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:20.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:20.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:20.125 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:20.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:20.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:20.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:20.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:20.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:20.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:20.376 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:20.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:20.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:45:20.477 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:20.541 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:20.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:20.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:20.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:20 smithi190 ceph-mon[33748]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-03-24T12:45:20.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:20 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3647228361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:45:20.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:20 smithi190 ceph-mon[33748]: osdmap e374: 8 total, 7 up, 6 in 2024-03-24T12:45:20.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:20.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:20 smithi029 ceph-mon[41500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-03-24T12:45:20.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:20 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3647228361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:45:20.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:20 smithi029 ceph-mon[41500]: osdmap e374: 8 total, 7 up, 6 in 2024-03-24T12:45:20.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:20 smithi029 ceph-mon[29919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-03-24T12:45:20.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:20 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3647228361' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:45:20.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:20 smithi029 ceph-mon[29919]: osdmap e374: 8 total, 7 up, 6 in 2024-03-24T12:45:20.671 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:20.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:20.772 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:20.801 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:20.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:20.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:20.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:20.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:20.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:21.028 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:21.057 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:21.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:21.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:21.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:21.184 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:21.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:21.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:21.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:21.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:21.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:21.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:21.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:21.444 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:21.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:21.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:21.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:21.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:21.571 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:21.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:21.672 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:21.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:21.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:21.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:21.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:21.828 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:21.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:21.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:21.956 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:21.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:22.013 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7] out_osds: [5, 6] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:45:22.014 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:45:22.014 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.33693844979059434 2024-03-24T12:45:22.014 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd reweight 3 0.33693844979059434 2024-03-24T12:45:22.057 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:22.085 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:22.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:22.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:22.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:22.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:22.212 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:22.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:22.288 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'unique_pool_3' removed 2024-03-24T12:45:22.288 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:22 smithi029 ceph-mon[41500]: pgmap v851: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 160 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 43/723 objects degraded (5.947%); 30/723 objects misplaced (4.149%); 54 KiB/s, 13 objects/s recovering 2024-03-24T12:45:22.289 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:22 smithi029 ceph-mon[41500]: osdmap e375: 8 total, 7 up, 6 in 2024-03-24T12:45:22.289 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:22 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4091923090' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:45:22.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:22 smithi029 ceph-mon[29919]: pgmap v851: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 160 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 43/723 objects degraded (5.947%); 30/723 objects misplaced (4.149%); 54 KiB/s, 13 objects/s recovering 2024-03-24T12:45:22.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:22 smithi029 ceph-mon[29919]: osdmap e375: 8 total, 7 up, 6 in 2024-03-24T12:45:22.289 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:22 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4091923090' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:45:22.314 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:22.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:22.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:22.460 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:22.462 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:45:22.463 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:45:22.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:22.484 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:22.485 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:22.585 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:22.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:22.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:22.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:22 smithi190 ceph-mon[33748]: pgmap v851: 168 pgs: 1 active+undersized+remapped+backfill_toofull, 5 active+remapped+backfill_toofull, 2 active+undersized+degraded+remapped+backfill_toofull, 160 active+clean; 3.4 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 511 B/s rd, 0 op/s; 43/723 objects degraded (5.947%); 30/723 objects misplaced (4.149%); 54 KiB/s, 13 objects/s recovering 2024-03-24T12:45:22.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:22 smithi190 ceph-mon[33748]: osdmap e375: 8 total, 7 up, 6 in 2024-03-24T12:45:22.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:22 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4091923090' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:45:22.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:22.765 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:22.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:22.866 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:22.891 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:22.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:22.993 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:23.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:23.018 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:23.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:23.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:23.143 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:23.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:23.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:23.273 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:23.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:23.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:23.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:23.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:23 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4091923090' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:45:23.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:23 smithi029 ceph-mon[29919]: osdmap e376: 8 total, 7 up, 6 in 2024-03-24T12:45:23.382 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:23 smithi029 ceph-mon[29919]: pgmap v854: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T12:45:23.382 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:23 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4091923090' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:45:23.382 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:23 smithi029 ceph-mon[41500]: osdmap e376: 8 total, 7 up, 6 in 2024-03-24T12:45:23.383 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:23 smithi029 ceph-mon[41500]: pgmap v854: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T12:45:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:23.400 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:23.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:23.500 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:23.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:23.515 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:45:23.515 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:45:23.534 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:23.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:23.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:23 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4091923090' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:45:23.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:23 smithi190 ceph-mon[33748]: osdmap e376: 8 total, 7 up, 6 in 2024-03-24T12:45:23.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:23 smithi190 ceph-mon[33748]: pgmap v854: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T12:45:23.637 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:23.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:23.661 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:23.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:23.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:23.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:23.787 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:23.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:23.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:23.912 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:23.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:23.993 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-24T12:45:23.994 INFO:tasks.radosbench:Beginning radosbench... 2024-03-24T12:45:23.994 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-24T12:45:23.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-24T12:45:24.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:24.041 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:24.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:24.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:24.167 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:24.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:24.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:24.269 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:24.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:24.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:24.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:24.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:24.425 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:24.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:24.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:45:24.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:24.555 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:24.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:24.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:24.569 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:24 smithi029 ceph-mon[41500]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull) 2024-03-24T12:45:24.569 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:24 smithi029 ceph-mon[41500]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/723 objects degraded (5.947%), 2 pgs degraded) 2024-03-24T12:45:24.569 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:24 smithi029 ceph-mon[29919]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull) 2024-03-24T12:45:24.569 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:24 smithi029 ceph-mon[29919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/723 objects degraded (5.947%), 2 pgs degraded) 2024-03-24T12:45:24.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:24 smithi190 ceph-mon[33748]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 8 pgs backfill_toofull) 2024-03-24T12:45:24.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:24 smithi190 ceph-mon[33748]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 43/723 objects degraded (5.947%), 2 pgs degraded) 2024-03-24T12:45:24.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:24.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:24.683 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:24.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:24.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:24.812 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:24.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:24.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:24.940 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:24.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:24.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:25.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:25.065 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:25.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:25.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:25.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:25.194 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:25.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:25.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:25.323 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:25.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:25.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:25.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:25 smithi029 ceph-mon[29919]: pgmap v855: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T12:45:25.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:25 smithi029 ceph-mon[41500]: pgmap v855: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T12:45:25.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:25.450 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:25.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:25.551 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:25.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:25.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:25.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:25.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:25 smithi190 ceph-mon[33748]: pgmap v855: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-24T12:45:25.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:25.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:25.707 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:25.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:25.809 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:25.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:25.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:25.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:25.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:25.968 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:25.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:26.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:26.095 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:26.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:26.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:26.197 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:26.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:26.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:26.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:26.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:26.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:26.350 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:26.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:26.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:26.481 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:26.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:26.583 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:26.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:26.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:26.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:26.737 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:26.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:26.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T12:45:26.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:26.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:26.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:26.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:26.994 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:26.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:26.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:27.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:27.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:27.121 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:27.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:27.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:27.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:27.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:27.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:27.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:27.376 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:27.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:27.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:27.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:27.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:27.503 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:27.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:27.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:27.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:27.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:45:27.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:45:27.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:27.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:27.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:27.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:45:27.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:45:27.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:27.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:27.629 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:27.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:27.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:45:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:45:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:27.731 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:27.759 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:27.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:27.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:27.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:27.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:27.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:27.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:27.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:45:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:45:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:45:27.991 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:28.019 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:28.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:28.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:28.120 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:28.148 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:28.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:28.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:28.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:28.274 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:28.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:28.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:28.375 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:28.399 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:28.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:28.400 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:28.468 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:28 smithi190 ceph-mon[33748]: pgmap v856: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 617 B/s rd, 0 op/s 2024-03-24T12:45:28.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:28 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2538036666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:45:28.469 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:28 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:45:28.501 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:28.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:28.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:28.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:28.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:28.655 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:28.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:28.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:28.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:28 smithi029 ceph-mon[41500]: pgmap v856: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 617 B/s rd, 0 op/s 2024-03-24T12:45:28.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:28 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2538036666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:45:28.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:28 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:45:28.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:28 smithi029 ceph-mon[29919]: pgmap v856: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 617 B/s rd, 0 op/s 2024-03-24T12:45:28.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:28 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2538036666' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:45:28.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:28 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:45:28.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:28.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:28.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:28.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:45:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:45:28.884 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:28.911 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:28.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:28.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:29.013 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:29.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:29.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:29.165 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:29.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:29.266 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:29.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:29.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:29.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:29.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:29.423 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:29.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:29.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:29.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:29.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:29.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:29.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:29 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:45:29.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:29 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:45:29.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:29 smithi190 ceph-mon[33748]: osdmap e377: 8 total, 7 up, 6 in 2024-03-24T12:45:29.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:29.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:29 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:45:29.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:29 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:45:29.670 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:29 smithi029 ceph-mon[41500]: osdmap e377: 8 total, 7 up, 6 in 2024-03-24T12:45:29.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:29 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:45:29.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:29 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:45:29.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:29 smithi029 ceph-mon[29919]: osdmap e377: 8 total, 7 up, 6 in 2024-03-24T12:45:29.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:29.681 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:29.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:29.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:29.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:29.810 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:29.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:29.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:29.939 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:29.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:30.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:30.069 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:30.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:30.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:30.199 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:30.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:30.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:30.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:30.328 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:30.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:30.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:30.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:30.456 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:30.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:30.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:30.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:30.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:30.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:30 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:30 smithi190 ceph-mon[33748]: pgmap v858: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:45:30.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:30 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:30.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:30 smithi029 ceph-mon[29919]: pgmap v858: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:45:30.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:30 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:30.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:30 smithi029 ceph-mon[41500]: pgmap v858: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:45:30.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:30.716 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:30.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:30.818 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:30.846 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:30.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:30.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:30.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:30.976 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:30.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:30.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:31.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:31.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:31.107 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:31.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:31.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:31.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:31.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:31.367 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:31.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:31.372 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:45:31.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:31.496 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:31.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:31.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:31.612 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:31.630 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:31.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:31.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:31 smithi190 ceph-mon[33748]: osdmap e378: 8 total, 7 up, 6 in 2024-03-24T12:45:31.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:31 smithi029 ceph-mon[41500]: osdmap e378: 8 total, 7 up, 6 in 2024-03-24T12:45:31.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:31 smithi029 ceph-mon[29919]: osdmap e378: 8 total, 7 up, 6 in 2024-03-24T12:45:31.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:31.757 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:31.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:31.858 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:31.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:31.923 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:31.923 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:31.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:32.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:32.014 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:32.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:32.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:32.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:32.140 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:32.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:32.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:32.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:32.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:32.366 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:32.373 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:32.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:32.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:32.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:32 smithi029 ceph-mon[29919]: pgmap v860: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:45:32.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:32 smithi029 ceph-mon[41500]: pgmap v860: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:45:32.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:32.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:32.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:32.619 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:32 smithi190 ceph-mon[33748]: pgmap v860: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:45:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:32.644 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:32.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:32.745 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:32.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:32.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:32.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:32.875 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:32.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:32.900 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:32.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:33.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:33.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:33.026 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:33.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:33.127 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:33.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:33.156 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:33.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:33.257 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:33.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:33.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:33.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:33.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:33.411 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:33.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:33.438 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:33 smithi029 ceph-mon[41500]: pgmap v861: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:45:33.439 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:33 smithi029 ceph-mon[29919]: pgmap v861: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:45:33.439 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:45:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:45:33.292+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:45:33.513 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:33.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:33.538 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:33.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:33 smithi190 ceph-mon[33748]: pgmap v861: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:45:33.638 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:33.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:33.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:33.769 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:33.796 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:33.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:33.897 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:33.922 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:33.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:33.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:34.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:34.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:34.050 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:34.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:34.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:34.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:34.175 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:34.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:34.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:34.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:34.301 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:34.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:34.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:34.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:34.427 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:34.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:34.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:34.552 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:34.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:34.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:34.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:34.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:34.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:34.678 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:34.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:34.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:34.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:34.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:34.804 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:34.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:34.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:34.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:45:34.933 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:34.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:34.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:35.035 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:35.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:35.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:35.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:35.190 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:35.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:35.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:35.320 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:35.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:35.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:35.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:35.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:35.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:35.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:35.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:35.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:35.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:35.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:35.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:35.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:35.707 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:35.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:35.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:35.833 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:35.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:35.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:35.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:35.959 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:35.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:36.060 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:36.088 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:36.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:36.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:36.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:36.218 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:36.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:36.319 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:36.347 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:36.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:36.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:36.449 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:36.477 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:36.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:36.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:36.526 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:36 smithi029 ceph-mon[41500]: pgmap v862: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:45:36.527 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:36 smithi029 ceph-mon[29919]: pgmap v862: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:45:36.579 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:36.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:36.607 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:36.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:36 smithi190 ceph-mon[33748]: pgmap v862: 142 pgs: 142 active+clean; 740 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:45:36.709 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:36.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:36.736 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:36.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:36.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:36.865 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:36.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:36.967 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:36.993 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:36.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:36.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:37.095 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:37.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:37.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:37.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:37.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:37.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:37.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:37.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:37.374 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:37.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:37.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:37.500 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:37.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:37.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:37.602 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:37.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:37.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:37.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:37.666 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:45:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:45:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:45:37.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:37.756 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:37.757 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:37.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:37.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:37.882 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:37.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:37.983 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:38.011 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:38.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:38.114 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:38.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:38.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:38.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:38.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:38.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:38.271 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:38.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:38.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:38.400 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:38.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:38.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:38 smithi029 ceph-mon[41500]: pgmap v863: 142 pgs: 142 active+clean; 740 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 592 B/s rd, 0 op/s 2024-03-24T12:45:38.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:38 smithi029 ceph-mon[29919]: pgmap v863: 142 pgs: 142 active+clean; 740 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 592 B/s rd, 0 op/s 2024-03-24T12:45:38.479 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:38 smithi190 ceph-mon[33748]: pgmap v863: 142 pgs: 142 active+clean; 740 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 592 B/s rd, 0 op/s 2024-03-24T12:45:38.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:38.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:38.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:38.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:38.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:38.655 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:38.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:38.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:38.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:38.781 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:38.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:38.784 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:45:38.784 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:45:38.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:45:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:45:38.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:38.907 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:38.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:39.008 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:39.036 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:39.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:39.138 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:39.165 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:39.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:39.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:39.235 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.3 to 0.336938 (5641) 2024-03-24T12:45:39.267 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:39.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:39.294 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:39.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:39.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:39.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:39 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2382508240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.33693844979059434}]: dispatch 2024-03-24T12:45:39.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:39 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2382508240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.33693844979059434}]: dispatch 2024-03-24T12:45:39.425 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:39.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:39.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:39.527 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:39.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T12:45:39.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:39.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:39.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:39 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2382508240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.33693844979059434}]: dispatch 2024-03-24T12:45:39.655 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:39.679 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:39.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:39.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:39.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:39.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:39.806 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:39.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:39.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:39.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:39.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:40.038 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:40.066 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:40.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:40.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:40.168 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:40.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:40.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:40.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:40.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:40.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:40.324 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:40.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:40.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:40 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:40.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:40 smithi029 ceph-mon[41500]: pgmap v864: 142 pgs: 142 active+clean; 740 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:45:40.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:40 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2382508240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.33693844979059434}]': finished 2024-03-24T12:45:40.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:40 smithi029 ceph-mon[41500]: osdmap e379: 8 total, 7 up, 6 in 2024-03-24T12:45:40.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:40 smithi029 ceph-mon[41500]: osdmap e380: 8 total, 7 up, 6 in 2024-03-24T12:45:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:40 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:40 smithi029 ceph-mon[29919]: pgmap v864: 142 pgs: 142 active+clean; 740 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:45:40.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:40 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2382508240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.33693844979059434}]': finished 2024-03-24T12:45:40.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:40 smithi029 ceph-mon[29919]: osdmap e379: 8 total, 7 up, 6 in 2024-03-24T12:45:40.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:40 smithi029 ceph-mon[29919]: osdmap e380: 8 total, 7 up, 6 in 2024-03-24T12:45:40.426 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:40.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:40.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:40.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:40.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:40.579 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:40.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:40.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:40 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:40.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:40 smithi190 ceph-mon[33748]: pgmap v864: 142 pgs: 142 active+clean; 740 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:45:40.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:40 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2382508240' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.33693844979059434}]': finished 2024-03-24T12:45:40.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:40 smithi190 ceph-mon[33748]: osdmap e379: 8 total, 7 up, 6 in 2024-03-24T12:45:40.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:40 smithi190 ceph-mon[33748]: osdmap e380: 8 total, 7 up, 6 in 2024-03-24T12:45:40.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:40.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:40.705 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:40.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:40.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:40.831 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:40.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:40.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:40.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:40.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:40.957 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:40.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:41.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:41.082 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:41.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:41.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:41.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:41.207 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:41.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:41.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:41.338 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:41.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:41.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:41 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:45:41.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:41 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:45:41.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:41 smithi029 ceph-mon[29919]: pgmap v867: 142 pgs: 20 peering, 122 active+clean; 740 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 383 B/s rd, 0 op/s 2024-03-24T12:45:41.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:41 smithi029 ceph-mon[29919]: osdmap e381: 8 total, 7 up, 6 in 2024-03-24T12:45:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:41 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:45:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:41 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:45:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:41 smithi029 ceph-mon[41500]: pgmap v867: 142 pgs: 20 peering, 122 active+clean; 740 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 383 B/s rd, 0 op/s 2024-03-24T12:45:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:41 smithi029 ceph-mon[41500]: osdmap e381: 8 total, 7 up, 6 in 2024-03-24T12:45:41.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:41.464 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:41.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:41.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:41.566 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:41.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:41.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:41.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:41 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:45:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:41 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:45:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:41 smithi190 ceph-mon[33748]: pgmap v867: 142 pgs: 20 peering, 122 active+clean; 740 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 383 B/s rd, 0 op/s 2024-03-24T12:45:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:41 smithi190 ceph-mon[33748]: osdmap e381: 8 total, 7 up, 6 in 2024-03-24T12:45:41.687 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:41.696 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:41.707 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:45:41.711 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:41.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:41.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:41.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:41.730 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:45:41.751 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:45:41.770 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:45:41.791 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:45:41.812 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:45:41.825 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:41.833 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:45:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:45:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:45:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:45:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:45:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:45:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:45:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:45:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:45:41.853 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:41.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:41.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:41.979 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:41.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:42.080 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:42.108 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:42.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:42.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:42.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:42.226 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'unique_pool_4' created 2024-03-24T12:45:42.239 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:42.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:42.244 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[41500]: Health check failed: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-24T12:45:42.244 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4199274261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:45:42.244 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:45:42.244 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1254460335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:45:42.244 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-24T12:45:42.244 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[41500]: osdmap e382: 8 total, 7 up, 6 in 2024-03-24T12:45:42.244 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[29919]: Health check failed: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-24T12:45:42.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4199274261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:45:42.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:45:42.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1254460335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:45:42.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-24T12:45:42.245 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:42 smithi029 ceph-mon[29919]: osdmap e382: 8 total, 7 up, 6 in 2024-03-24T12:45:42.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:42.365 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:42.366 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:42.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:42.491 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:42.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:42.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:42.621 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:42.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:42 smithi190 ceph-mon[33748]: Health check failed: Reduced data availability: 4 pgs inactive, 7 pgs peering (PG_AVAILABILITY) 2024-03-24T12:45:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:42 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4199274261' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:45:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:42 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-24T12:45:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:42 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1254460335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:45:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:42 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-24T12:45:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:42 smithi190 ceph-mon[33748]: osdmap e382: 8 total, 7 up, 6 in 2024-03-24T12:45:42.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:42.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:42.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:42.851 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:42.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:42.878 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:42.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:42.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:43.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:43.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:43.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:43.130 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:43.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:43.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:43.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:43.258 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:43.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:43.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:43.384 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:43.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:43.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:43 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:45:43.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:43 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:45:43.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:43 smithi029 ceph-mon[41500]: pgmap v870: 158 pgs: 6 creating+peering, 10 unknown, 22 peering, 120 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 7.7 KiB/s rd, 55 KiB/s wr, 9 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T12:45:43.444 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:43 smithi029 ceph-mon[41500]: osdmap e383: 8 total, 7 up, 6 in 2024-03-24T12:45:43.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:43 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:45:43.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:43 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:45:43.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:43 smithi029 ceph-mon[29919]: pgmap v870: 158 pgs: 6 creating+peering, 10 unknown, 22 peering, 120 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 7.7 KiB/s rd, 55 KiB/s wr, 9 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T12:45:43.445 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:43 smithi029 ceph-mon[29919]: osdmap e383: 8 total, 7 up, 6 in 2024-03-24T12:45:43.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:43.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:43.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:43.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:43 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:45:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:43 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:45:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:43 smithi190 ceph-mon[33748]: pgmap v870: 158 pgs: 6 creating+peering, 10 unknown, 22 peering, 120 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 7.7 KiB/s rd, 55 KiB/s wr, 9 op/s; 0 B/s, 4 objects/s recovering 2024-03-24T12:45:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:43 smithi190 ceph-mon[33748]: osdmap e383: 8 total, 7 up, 6 in 2024-03-24T12:45:43.641 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:43.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:45:43.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:45:43.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:43.771 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:43.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:43.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:43.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:43.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:43.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:43.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:44.005 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:44.029 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:44.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:44.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:44.154 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:44.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:44.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:44.256 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:44.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:44.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:44.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:44.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:44.413 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:44.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:44.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:44 smithi029 ceph-mon[41500]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:45:44.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:44 smithi029 ceph-mon[41500]: osdmap e384: 8 total, 7 up, 6 in 2024-03-24T12:45:44.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:44 smithi029 ceph-mon[29919]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:45:44.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:44 smithi029 ceph-mon[29919]: osdmap e384: 8 total, 7 up, 6 in 2024-03-24T12:45:44.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:44.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:44.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:44 smithi190 ceph-mon[33748]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T12:45:44.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:44 smithi190 ceph-mon[33748]: osdmap e384: 8 total, 7 up, 6 in 2024-03-24T12:45:44.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:44.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:44.666 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:44.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:44.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:44.790 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:44.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:44.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:44.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:44.916 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:44.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:45.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:45.042 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:45.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:45.143 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:45.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:45.173 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:45.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:45.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:45.303 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:45.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:45.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:45.361 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:45 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:45:45.361 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:45 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:45:45.361 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:45 smithi029 ceph-mon[41500]: pgmap v873: 158 pgs: 12 creating+peering, 2 unknown, 13 peering, 131 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 8.0 KiB/s rd, 55 KiB/s wr, 9 op/s; 63 KiB/s, 15 objects/s recovering 2024-03-24T12:45:45.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:45 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:45:45.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:45 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:45:45.362 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:45 smithi029 ceph-mon[29919]: pgmap v873: 158 pgs: 12 creating+peering, 2 unknown, 13 peering, 131 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 8.0 KiB/s rd, 55 KiB/s wr, 9 op/s; 63 KiB/s, 15 objects/s recovering 2024-03-24T12:45:45.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:45.434 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:45.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:45.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:45.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:45.561 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:45.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:45.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:45 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:45:45.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:45 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:45:45.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:45 smithi190 ceph-mon[33748]: pgmap v873: 158 pgs: 12 creating+peering, 2 unknown, 13 peering, 131 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 8.0 KiB/s rd, 55 KiB/s wr, 9 op/s; 63 KiB/s, 15 objects/s recovering 2024-03-24T12:45:45.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-24T12:45:45.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:45:45.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:45.695 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:45.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:45.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:45.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:45.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:45.824 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:45.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:45.847 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7] out_osds: [5, 6] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:45:45.847 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:45:45.847 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-24T12:45:45.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:45:45.926 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:45.951 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:45.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:45.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:46.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:46.076 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:46.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:46.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:46.178 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:46.207 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:46.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:46.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:46.251 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:45:46.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:46.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:46.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:46.439 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:46.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:46.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:46.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:46.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:46.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:46.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:46.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:46.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:46 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1614229177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:45:46.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:46 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1614229177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:45:46.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:46 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1614229177' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:45:46.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:46.727 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:46.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:46.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:46.829 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:46.857 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:46.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:46.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:46.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:46.983 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:46.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:46.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:47.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:47.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:47.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:47.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:47.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:47.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:47.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:47.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:47.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:47.361 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:47.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:47.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:47.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:45:47.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1614229177' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:45:47.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[29919]: osdmap e385: 8 total, 7 up, 6 in 2024-03-24T12:45:47.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[29919]: pgmap v875: 158 pgs: 12 creating+peering, 3 peering, 143 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 6.9 KiB/s rd, 47 KiB/s wr, 8 op/s; 55 KiB/s, 16 objects/s recovering 2024-03-24T12:45:47.400 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[29919]: 6.1b scrub starts 2024-03-24T12:45:47.401 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[29919]: 6.1b scrub ok 2024-03-24T12:45:47.401 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:45:47.401 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1614229177' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:45:47.401 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[41500]: osdmap e385: 8 total, 7 up, 6 in 2024-03-24T12:45:47.401 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[41500]: pgmap v875: 158 pgs: 12 creating+peering, 3 peering, 143 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 6.9 KiB/s rd, 47 KiB/s wr, 8 op/s; 55 KiB/s, 16 objects/s recovering 2024-03-24T12:45:47.401 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[41500]: 6.1b scrub starts 2024-03-24T12:45:47.401 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-mon[41500]: 6.1b scrub ok 2024-03-24T12:45:47.464 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:47.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:47.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:47.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:47.615 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:47.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:47.624 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:47.624 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:47 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:45:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:47 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1614229177' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:45:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:47 smithi190 ceph-mon[33748]: osdmap e385: 8 total, 7 up, 6 in 2024-03-24T12:45:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:47 smithi190 ceph-mon[33748]: pgmap v875: 158 pgs: 12 creating+peering, 3 peering, 143 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 6.9 KiB/s rd, 47 KiB/s wr, 8 op/s; 55 KiB/s, 16 objects/s recovering 2024-03-24T12:45:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:47 smithi190 ceph-mon[33748]: 6.1b scrub starts 2024-03-24T12:45:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:47 smithi190 ceph-mon[33748]: 6.1b scrub ok 2024-03-24T12:45:47.665 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:45:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:45:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:45:47.666 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:45:47.666 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:45:47.666 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:45:47.666 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:45:47.666 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.666 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104325 9.0 9:05e930a3:::benchmark_data_smithi029_134815_object104324:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:45:47.666 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.117256+0000", 2024-03-24T12:45:47.666 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 457.55041905000002, 2024-03-24T12:45:47.666 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.081903713, 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104325 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117256+0000", 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.667 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117256+0000", 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117258+0000", 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5710000000000001e-06 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117282+0000", 2024-03-24T12:45:47.668 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4168999999999998e-05 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117294+0000", 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1813e-05 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.117303+0000", 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9670000000000007e-06 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.669 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.118795+0000", 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0014927600000000001 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.118807+0000", 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1378e-05 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077574+0000", 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.958767339 2024-03-24T12:45:47.670 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077587+0000", 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3011e-05 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.176190+0000", 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0986029450000001 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.671 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.176338+0000", 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014812700000000001 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2,6", 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.176830+0000", 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00049169399999999999 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.191858+0000", 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.015028036999999999 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.673 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.673 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.191903+0000", 2024-03-24T12:45:47.673 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5529999999999999e-05 2024-03-24T12:45:47.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.673 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.673 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.673 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.193131+0000", 2024-03-24T12:45:47.673 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001227552 2024-03-24T12:45:47.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.673 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.674 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.674 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.198956+0000", 2024-03-24T12:45:47.675 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0058248479999999997 2024-03-24T12:45:47.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.675 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.675 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.675 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199033+0000", 2024-03-24T12:45:47.675 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6946000000000005e-05 2024-03-24T12:45:47.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.675 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.675 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199160+0000", 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000127026 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104332 9.0 9:0a464280:::benchmark_data_smithi029_134815_object104331:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.128520+0000", 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 457.539155403, 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0715006140000001, 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.676 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104332 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128520+0000", 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128520+0000", 2024-03-24T12:45:47.677 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.678 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.678 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.678 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.678 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128521+0000", 2024-03-24T12:45:47.678 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.815e-06 2024-03-24T12:45:47.678 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.678 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.682 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.682 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128548+0000", 2024-03-24T12:45:47.682 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.6149999999999999e-05 2024-03-24T12:45:47.682 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.682 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.682 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.682 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128568+0000", 2024-03-24T12:45:47.682 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0421e-05 2024-03-24T12:45:47.682 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.683 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.683 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.683 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128579+0000", 2024-03-24T12:45:47.683 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0565e-05 2024-03-24T12:45:47.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128649+0000", 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0376000000000005e-05 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.128660+0000", 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1141999999999999e-05 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077614+0000", 2024-03-24T12:45:47.684 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.94895389900000005 2024-03-24T12:45:47.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.685 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.685 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.685 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077623+0000", 2024-03-24T12:45:47.685 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1479999999999995e-06 2024-03-24T12:45:47.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.685 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.685 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.685 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.184183+0000", 2024-03-24T12:45:47.686 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1065596470000001 2024-03-24T12:45:47.686 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.686 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.686 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.687 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.184310+0000", 2024-03-24T12:45:47.687 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012726999999999999 2024-03-24T12:45:47.687 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.687 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.687 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2,6", 2024-03-24T12:45:47.687 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.184474+0000", 2024-03-24T12:45:47.687 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016396 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.191923+0000", 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0074493980000000003 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.192359+0000", 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00043587299999999998 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.688 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.193191+0000", 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00083197800000000004 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199218+0000", 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060271320000000001 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.199289+0000", 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0358000000000006e-05 2024-03-24T12:45:47.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.200020+0000", 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00073148199999999999 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:104356 9.0 9:08e6f257:::benchmark_data_smithi029_134815_object104355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e232)", 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:38:10.157948+0000", 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 457.50972708299997, 2024-03-24T12:45:47.690 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0423164210000002, 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 104356 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157948+0000", 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.691 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157948+0000", 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157950+0000", 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6619999999999999e-06 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157972+0000", 2024-03-24T12:45:47.692 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2824e-05 2024-03-24T12:45:47.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.693 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.693 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.693 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157987+0000", 2024-03-24T12:45:47.693 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4649e-05 2024-03-24T12:45:47.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.693 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.693 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.693 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.157995+0000", 2024-03-24T12:45:47.693 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2199999999999992e-06 2024-03-24T12:45:47.694 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.694 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.694 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.694 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.158067+0000", 2024-03-24T12:45:47.694 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1746000000000001e-05 2024-03-24T12:45:47.694 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.694 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.694 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.694 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:10.158075+0000", 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7980000000000006e-06 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077650+0000", 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.91957491400000002 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:11.077659+0000", 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8459999999999997e-06 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.695 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.189246+0000", 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1115876039999999 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.189379+0000", 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013289700000000001 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2,6", 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.189577+0000", 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019751600000000001 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.696 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.192372+0000", 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027951769999999998 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.193087+0000", 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00071518999999999999 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.194000+0000", 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00091256499999999995 2024-03-24T12:45:47.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.200074+0000", 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060745529999999999 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.200135+0000", 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1086000000000004e-05 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.698 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:38:12.200264+0000", 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012917399999999999 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172155 9.0 9:07da41af:::benchmark_data_smithi029_134815_object60089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.686738+0000", 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 373.98093720000003, 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.508465065, 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.699 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172155 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686738+0000", 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686738+0000", 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.700 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686739+0000", 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5209999999999999e-06 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686744+0000", 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7589999999999997e-06 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686755+0000", 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0711e-05 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.702 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.703 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686763+0000", 2024-03-24T12:45:47.703 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8010000000000004e-06 2024-03-24T12:45:47.703 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.703 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.704 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.704 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686822+0000", 2024-03-24T12:45:47.704 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9033000000000002e-05 2024-03-24T12:45:47.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.704 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.704 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686972+0000", 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000149894 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.687082+0000", 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000110303 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.689792+0000", 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027105240000000002 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.705 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.692544+0000", 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027517980000000002 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.751725+0000", 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.059180952000000002 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.751741+0000", 2024-03-24T12:45:47.706 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5579000000000001e-05 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738213+0000", 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.98647265900000003 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738227+0000", 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3541e-05 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.195103+0000", 2024-03-24T12:45:47.707 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45687653900000003 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.195203+0000", 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9450999999999994e-05 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172200 9.0 9:061b9bea:::benchmark_data_smithi029_134815_object60134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.785371+0000", 2024-03-24T12:45:47.708 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 373.88230433799998, 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.419709179, 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172200 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785371+0000", 2024-03-24T12:45:47.709 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785371+0000", 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785372+0000", 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4789999999999999e-06 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.710 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785378+0000", 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2260000000000002e-06 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785390+0000", 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1697e-05 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785399+0000", 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1880000000000008e-06 2024-03-24T12:45:47.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785495+0000", 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6033999999999996e-05 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785502+0000", 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5849999999999996e-06 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738257+0000", 2024-03-24T12:45:47.712 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.95275513099999998 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738266+0000", 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0219999999999993e-06 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.196663+0000", 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45839733799999999 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.713 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.196781+0000", 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000118023 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2,6", 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.196907+0000", 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012570600000000001 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202007+0000", 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0050995379999999998 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.714 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202150+0000", 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000142955 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202201+0000", 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1745e-05 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.204877+0000", 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026757270000000001 2024-03-24T12:45:47.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.204947+0000", 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9937000000000005e-05 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.205080+0000", 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000132848 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.716 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214257 10.14 10:2ceef5e0:::benchmark_data_smithi029_160664_object87283:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.675879+0000", 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.991796502, 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8521024619999999, 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214257 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.717 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675879+0000", 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675879+0000", 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675880+0000", 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.392e-06 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.718 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.719 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675886+0000", 2024-03-24T12:45:47.719 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7270000000000004e-06 2024-03-24T12:45:47.719 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.719 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.719 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.719 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675895+0000", 2024-03-24T12:45:47.719 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2289999999999992e-06 2024-03-24T12:45:47.719 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.719 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.719 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675905+0000", 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8400000000000007e-06 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.676001+0000", 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6440999999999994e-05 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.676009+0000", 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9889999999999992e-06 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.720 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.527872+0000", 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8518627940000001 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.527981+0000", 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010904999999999999 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.721 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214269 10.14 10:2d22cebe:::benchmark_data_smithi029_160664_object87295:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.675923+0000", 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.991751605000005, 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.873380117, 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214269 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.722 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675923+0000", 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675923+0000", 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675924+0000", 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6100000000000001e-07 2024-03-24T12:45:47.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.724 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.724 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.724 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675927+0000", 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7539999999999998e-06 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675935+0000", 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2390000000000002e-06 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675942+0000", 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7560000000000002e-06 2024-03-24T12:45:47.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.676030+0000", 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8160999999999998e-05 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.676035+0000", 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.356e-06 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.528231+0000", 2024-03-24T12:45:47.726 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8521956849999999 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for missing object", 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.528328+0000", 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6831999999999997e-05 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549214+0000", 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.020886693000000001 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.727 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549304+0000", 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9179999999999997e-05 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214277 10.14 10:2bb49b9c:::benchmark_data_smithi029_160664_object87303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.684362+0000", 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.983313426999999, 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.85330631, 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.728 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214277 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684362+0000", 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684362+0000", 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684364+0000", 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8920000000000001e-06 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684370+0000", 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0580000000000003e-06 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684382+0000", 2024-03-24T12:45:47.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2167999999999999e-05 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684389+0000", 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7179999999999996e-06 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684452+0000", 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2269999999999998e-05 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.731 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684461+0000", 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4599999999999992e-06 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.528389+0000", 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.843927989 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.528485+0000", 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6043000000000002e-05 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.528752+0000", 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026698599999999998 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.531958+0000", 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0032062269999999999 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534548+0000", 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002589694 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537515+0000", 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0029667629999999999 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537568+0000", 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3471999999999999e-05 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537668+0000", 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9569999999999997e-05 2024-03-24T12:45:47.734 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214303 10.14 10:2887a57d:::benchmark_data_smithi029_160664_object87329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.708096+0000", 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.959579212999998, 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8297232299999999, 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.735 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214303 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708096+0000", 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708096+0000", 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708097+0000", 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3069999999999999e-06 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708101+0000", 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.405e-06 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708110+0000", 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0129999999999999e-06 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708116+0000", 2024-03-24T12:45:47.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2219999999999999e-06 2024-03-24T12:45:47.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.738 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708562+0000", 2024-03-24T12:45:47.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044582200000000002 2024-03-24T12:45:47.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708572+0000", 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0559e-05 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.530125+0000", 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8215530129999999 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.530242+0000", 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000116858 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.530479+0000", 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000237149 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534509+0000", 2024-03-24T12:45:47.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0040294320000000003 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534589+0000", 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0791000000000006e-05 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537678+0000", 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0030884240000000002 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537723+0000", 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5048e-05 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537819+0000", 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6187000000000006e-05 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214310 10.14 10:2f4ebc59:::benchmark_data_smithi029_160664_object87336:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.714353+0000", 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.953322376000003, 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.82368043, 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214310 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.743 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714353+0000", 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714353+0000", 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714354+0000", 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.483e-06 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.744 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714360+0000", 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8889999999999999e-06 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714403+0000", 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3237e-05 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714413+0000", 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.944e-06 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.718231+0000", 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003818213 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.718244+0000", 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2064e-05 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.531998+0000", 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8137548749999999 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.532132+0000", 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133435 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.532382+0000", 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025042400000000001 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534635+0000", 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0022523920000000002 2024-03-24T12:45:47.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537828+0000", 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0031931860000000002 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537876+0000", 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8340000000000001e-05 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537933+0000", 2024-03-24T12:45:47.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700999999999998e-05 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538033+0000", 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000100247 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214355 10.14 10:2f432cd7:::benchmark_data_smithi029_160664_object87381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.747009+0000", 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.920665679999999, 2024-03-24T12:45:47.749 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7913908009999999, 2024-03-24T12:45:47.750 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.750 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.750 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.750 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.750 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.750 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214355 2024-03-24T12:45:47.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.750 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747009+0000", 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747009+0000", 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747011+0000", 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2190000000000001e-06 2024-03-24T12:45:47.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747015+0000", 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7849999999999999e-06 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747061+0000", 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5359999999999999e-05 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747070+0000", 2024-03-24T12:45:47.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5079999999999994e-06 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747132+0000", 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1600000000000007e-05 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747142+0000", 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0634e-05 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534668+0000", 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.787525668 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534802+0000", 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000134225 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.535024+0000", 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000221862 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538084+0000", 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003059562 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538142+0000", 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8168999999999998e-05 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538242+0000", 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010018600000000001 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538286+0000", 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3784e-05 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538400+0000", 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000114239 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214453 10.14 10:2d8c7f68:::benchmark_data_smithi029_160664_object87479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.756 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.824146+0000", 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.843529425, 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7145637759999999, 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214453 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824146+0000", 2024-03-24T12:45:47.757 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824146+0000", 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824147+0000", 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5e-06 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824152+0000", 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2009999999999998e-06 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824163+0000", 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0815000000000001e-05 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824171+0000", 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1270000000000003e-06 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.759 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824230+0000", 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8696999999999997e-05 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824239+0000", 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3039999999999994e-06 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.536110+0000", 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7118707930000001 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.536215+0000", 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105368 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.536411+0000", 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000195935 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538414+0000", 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020029729999999999 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538461+0000", 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6696999999999997e-05 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538513+0000", 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1714e-05 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538566+0000", 2024-03-24T12:45:47.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3233999999999999e-05 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538709+0000", 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000143418 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214474 10.14 10:29041696:::benchmark_data_smithi029_160664_object87500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.841093+0000", 2024-03-24T12:45:47.763 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.826581891000004, 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7080451330000002, 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214474 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841093+0000", 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.764 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841093+0000", 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841094+0000", 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7800000000000001e-07 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841099+0000", 2024-03-24T12:45:47.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5909999999999998e-06 2024-03-24T12:45:47.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.766 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.766 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.766 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841110+0000", 2024-03-24T12:45:47.766 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1664999999999999e-05 2024-03-24T12:45:47.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.766 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.767 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.767 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841118+0000", 2024-03-24T12:45:47.767 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9859999999999994e-06 2024-03-24T12:45:47.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.767 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.767 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.767 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841189+0000", 2024-03-24T12:45:47.767 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0298999999999999e-05 2024-03-24T12:45:47.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841197+0000", 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.568e-06 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538749+0000", 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6975519669999999 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538876+0000", 2024-03-24T12:45:47.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012745700000000001 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.539126+0000", 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024978899999999999 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.545132+0000", 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060055610000000004 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.769 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.548627+0000", 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0034948169999999999 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.548973+0000", 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00034671000000000002 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549025+0000", 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1919999999999998e-05 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549138+0000", 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113025 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214487 10.14 10:2b7e884d:::benchmark_data_smithi029_160664_object87513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.850412+0000", 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.817262583000002, 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.69917728, 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.771 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214487 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850412+0000", 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.772 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850412+0000", 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850414+0000", 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0109999999999999e-06 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850417+0000", 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0299999999999998e-06 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.773 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850424+0000", 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4569999999999999e-06 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850430+0000", 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8769999999999998e-06 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850499+0000", 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9182000000000003e-05 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850508+0000", 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7600000000000008e-06 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.540291+0000", 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6897836399999999 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.540398+0000", 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106465 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.540593+0000", 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019509900000000001 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549150+0000", 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085569410000000002 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549360+0000", 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020971799999999999 2024-03-24T12:45:47.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549437+0000", 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6941999999999996e-05 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549490+0000", 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3306000000000001e-05 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549590+0000", 2024-03-24T12:45:47.777 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9852000000000006e-05 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214509 10.14 10:2d062f92:::benchmark_data_smithi029_160664_object87535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.869591+0000", 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.798084513999996, 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6836662840000001, 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.778 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214509 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869591+0000", 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869591+0000", 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869592+0000", 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5850000000000001e-06 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869597+0000", 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2950000000000001e-06 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.780 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869611+0000", 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3186e-05 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869620+0000", 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4809999999999995e-06 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869683+0000", 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2731999999999995e-05 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869693+0000", 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0645000000000001e-05 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.541935+0000", 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6722419739999999 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.542069+0000", 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013380299999999999 2024-03-24T12:45:47.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.542284+0000", 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021516600000000001 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549161+0000", 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0068762490000000001 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549650+0000", 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00048899499999999999 2024-03-24T12:45:47.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553089+0000", 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0034392509999999999 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553148+0000", 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9278999999999999e-05 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553257+0000", 2024-03-24T12:45:47.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000108643 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214518 10.14 10:2c5d038c:::benchmark_data_smithi029_160664_object87544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.874901+0000", 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.792773721000003, 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.678556089, 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.785 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214518 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874901+0000", 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874901+0000", 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874903+0000", 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2580000000000001e-06 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874907+0000", 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2660000000000003e-06 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874929+0000", 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2572999999999999e-05 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874938+0000", 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4570000000000005e-06 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874999+0000", 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0687999999999999e-05 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.875012+0000", 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3312000000000001e-05 2024-03-24T12:45:47.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.789 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.789 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.789 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.543447+0000", 2024-03-24T12:45:47.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:47.790 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6684352900000001 2024-03-24T12:45:47.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.790 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.543566+0000", 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011909 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.543771+0000", 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020472900000000001 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549167+0000", 2024-03-24T12:45:47.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0053959000000000003 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549709+0000", 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00054198799999999997 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553306+0000", 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0035967490000000002 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553356+0000", 2024-03-24T12:45:47.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9925000000000001e-05 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553457+0000", 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000101864 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214533 10.14 10:2be34e47:::benchmark_data_smithi029_160664_object87559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.888437+0000", 2024-03-24T12:45:47.793 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.779237699999996, 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.665250957, 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214533 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.794 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888437+0000", 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888437+0000", 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888439+0000", 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4050000000000001e-06 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.795 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888444+0000", 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9309999999999999e-06 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888459+0000", 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4902999999999999e-05 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888468+0000", 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9749999999999996e-06 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888523+0000", 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5294999999999998e-05 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888532+0000", 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5680000000000005e-06 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.545170+0000", 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6566373140000001 2024-03-24T12:45:47.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.545291+0000", 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000121144 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.545514+0000", 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000223229 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549173+0000", 2024-03-24T12:45:47.798 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0036587550000000001 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549777+0000", 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00060382399999999998 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553540+0000", 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0037631269999999998 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553593+0000", 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2966000000000001e-05 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553688+0000", 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5520999999999996e-05 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214541 10.14 10:2d9371c9:::benchmark_data_smithi029_160664_object87567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.898613+0000", 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.769061718000003, 2024-03-24T12:45:47.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6554179869999999, 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214541 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898613+0000", 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898613+0000", 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898615+0000", 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.548e-06 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898620+0000", 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1869999999999996e-06 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898634+0000", 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3512999999999999e-05 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898641+0000", 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4109999999999999e-06 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898695+0000", 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3850000000000001e-05 2024-03-24T12:45:47.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898704+0000", 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9099999999999994e-06 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.547035+0000", 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6483308700000001 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.547162+0000", 2024-03-24T12:45:47.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000127198 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.547394+0000", 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023246599999999999 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549303+0000", 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001908397 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553500+0000", 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0041976440000000004 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553742+0000", 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000241747 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553795+0000", 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2720999999999999e-05 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.554031+0000", 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023652499999999999 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214561 10.14 10:2d91c4e6:::benchmark_data_smithi029_160664_object87587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.926538+0000", 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.741137107, 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6337680830000001, 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.807 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214561 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926538+0000", 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926538+0000", 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926540+0000", 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5570000000000001e-06 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926546+0000", 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1879999999999999e-06 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926558+0000", 2024-03-24T12:45:47.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2166e-05 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926566+0000", 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6899999999999992e-06 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926634+0000", 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8573999999999998e-05 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.810 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926645+0000", 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0539e-05 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549826+0000", 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6231816079999999 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549951+0000", 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012484400000000001 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.550213+0000", 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026170399999999997 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.557401+0000", 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0071880670000000002 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.558955+0000", 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0015537400000000001 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.560214+0000", 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0012589719999999999 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.560266+0000", 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2240000000000001e-05 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.560306+0000", 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0194000000000002e-05 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.813 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214562 10.14 10:29c2d725:::benchmark_data_smithi029_160664_object87588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.927210+0000", 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 80.740464861999996, 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.634147952, 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214562 2024-03-24T12:45:47.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927210+0000", 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927210+0000", 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927211+0000", 2024-03-24T12:45:47.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9699999999999995e-07 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927215+0000", 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3229999999999999e-06 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927225+0000", 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5820000000000008e-06 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:45:47.816 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927233+0000", 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6629999999999993e-06 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927288+0000", 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.533e-05 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927298+0000", 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8579999999999995e-06 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:45:47.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.551528+0000", 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.624230362 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.551640+0000", 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000112281 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.551868+0000", 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00022806899999999999 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.558961+0000", 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0070923169999999999 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.559809+0000", 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00084865600000000004 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.561263+0000", 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001453661 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.561318+0000", 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5315999999999997e-05 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.561358+0000", 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9737000000000003e-05 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:47.820 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:45:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:47.844 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:47.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:47.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:47.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:47.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:47.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:48.075 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:48.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:48.100 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:48.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:48.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:48.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:48.226 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:48.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:48.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:48.351 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:48.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:48.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:48.476 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:48.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:48.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:48.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:48.606 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:48.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:48.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:48.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:45:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:45:48.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:48 smithi190 ceph-mon[33748]: 6.6 scrub starts 2024-03-24T12:45:48.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:48 smithi190 ceph-mon[33748]: 6.6 scrub ok 2024-03-24T12:45:48.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:48 smithi190 ceph-mon[33748]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-03-24T12:45:48.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:48 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:45:48.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:48 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:45:48.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:45:48.308+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:45:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-mon[29919]: 6.6 scrub starts 2024-03-24T12:45:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-mon[29919]: 6.6 scrub ok 2024-03-24T12:45:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-mon[29919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-03-24T12:45:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:45:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:45:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-mon[41500]: 6.6 scrub starts 2024-03-24T12:45:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-mon[41500]: 6.6 scrub ok 2024-03-24T12:45:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-mon[41500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 5 pgs peering) 2024-03-24T12:45:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:45:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:48 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:45:48.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:48.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:48.733 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:48.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:48.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:48.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:48.862 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:48.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:48.963 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:48.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:48.987 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:48.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:49.088 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:49.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:49.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:49.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:45:49.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:49.244 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:49.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:49.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:49.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T12:45:49.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:49.371 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:49.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:49.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:49.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:49.503 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:49.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:49.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[29919]: 4.17 scrub starts 2024-03-24T12:45:49.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[29919]: 4.17 scrub ok 2024-03-24T12:45:49.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:49.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:49.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:49.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[29919]: pgmap v876: 158 pgs: 158 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s; 43 KiB/s, 8 objects/s recovering 2024-03-24T12:45:49.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[29919]: 1.4 scrub starts 2024-03-24T12:45:49.504 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[29919]: 1.4 scrub ok 2024-03-24T12:45:49.505 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[41500]: 4.17 scrub starts 2024-03-24T12:45:49.505 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[41500]: 4.17 scrub ok 2024-03-24T12:45:49.505 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:49.505 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:49.505 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:49.505 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[41500]: pgmap v876: 158 pgs: 158 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s; 43 KiB/s, 8 objects/s recovering 2024-03-24T12:45:49.505 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[41500]: 1.4 scrub starts 2024-03-24T12:45:49.505 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:49 smithi029 ceph-mon[41500]: 1.4 scrub ok 2024-03-24T12:45:49.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:49.630 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:49.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:49.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:49 smithi190 ceph-mon[33748]: 4.17 scrub starts 2024-03-24T12:45:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:49 smithi190 ceph-mon[33748]: 4.17 scrub ok 2024-03-24T12:45:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:45:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:45:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:49 smithi190 ceph-mon[33748]: pgmap v876: 158 pgs: 158 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s; 43 KiB/s, 8 objects/s recovering 2024-03-24T12:45:49.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:49 smithi190 ceph-mon[33748]: 1.4 scrub starts 2024-03-24T12:45:49.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:49 smithi190 ceph-mon[33748]: 1.4 scrub ok 2024-03-24T12:45:49.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:49.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:49.759 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:49.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:49.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:49.889 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:49.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:49.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:50.015 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:50.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:50.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:50.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:50.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:50.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:50.267 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:50.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:50.368 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:50.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:50.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:50.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:50.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:50.523 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:50.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:50.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:50 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:45:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:50 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:45:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:50 smithi190 ceph-mon[33748]: 6.13 scrub starts 2024-03-24T12:45:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:50 smithi190 ceph-mon[33748]: 6.13 scrub ok 2024-03-24T12:45:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:50 smithi190 ceph-mon[33748]: 1.7 scrub starts 2024-03-24T12:45:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:50 smithi190 ceph-mon[33748]: 1.7 scrub ok 2024-03-24T12:45:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:50 smithi190 ceph-mon[33748]: osdmap e386: 8 total, 7 up, 6 in 2024-03-24T12:45:50.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:50.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:50.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:45:50.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:45:50.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[29919]: 6.13 scrub starts 2024-03-24T12:45:50.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[29919]: 6.13 scrub ok 2024-03-24T12:45:50.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[29919]: 1.7 scrub starts 2024-03-24T12:45:50.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[29919]: 1.7 scrub ok 2024-03-24T12:45:50.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[29919]: osdmap e386: 8 total, 7 up, 6 in 2024-03-24T12:45:50.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:45:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:45:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[41500]: 6.13 scrub starts 2024-03-24T12:45:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[41500]: 6.13 scrub ok 2024-03-24T12:45:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[41500]: 1.7 scrub starts 2024-03-24T12:45:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[41500]: 1.7 scrub ok 2024-03-24T12:45:50.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:50 smithi029 ceph-mon[41500]: osdmap e386: 8 total, 7 up, 6 in 2024-03-24T12:45:50.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:50.783 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:50.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:50.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:50.915 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:50.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:51.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:51.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:51.045 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:51.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:51.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:51.170 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:51.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:51.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:51.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:51.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:51.297 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:51.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:51.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:51.422 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:51.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:51.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:51.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:51.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:51.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:45:51.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:45:51.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[29919]: 1.6 scrub starts 2024-03-24T12:45:51.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[29919]: 1.6 scrub ok 2024-03-24T12:45:51.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[29919]: 1.3 scrub starts 2024-03-24T12:45:51.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[29919]: 1.3 scrub ok 2024-03-24T12:45:51.595 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[29919]: pgmap v878: 158 pgs: 158 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 772 B/s rd, 0 op/s; 39 KiB/s, 6 objects/s recovering 2024-03-24T12:45:51.596 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:45:51.596 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:45:51.596 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[41500]: 1.6 scrub starts 2024-03-24T12:45:51.596 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[41500]: 1.6 scrub ok 2024-03-24T12:45:51.596 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[41500]: 1.3 scrub starts 2024-03-24T12:45:51.596 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[41500]: 1.3 scrub ok 2024-03-24T12:45:51.597 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:51 smithi029 ceph-mon[41500]: pgmap v878: 158 pgs: 158 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 772 B/s rd, 0 op/s; 39 KiB/s, 6 objects/s recovering 2024-03-24T12:45:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:51 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:45:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:51 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:45:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:51 smithi190 ceph-mon[33748]: 1.6 scrub starts 2024-03-24T12:45:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:51 smithi190 ceph-mon[33748]: 1.6 scrub ok 2024-03-24T12:45:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:51 smithi190 ceph-mon[33748]: 1.3 scrub starts 2024-03-24T12:45:51.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:51 smithi190 ceph-mon[33748]: 1.3 scrub ok 2024-03-24T12:45:51.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:51 smithi190 ceph-mon[33748]: pgmap v878: 158 pgs: 158 active+clean; 472 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 772 B/s rd, 0 op/s; 39 KiB/s, 6 objects/s recovering 2024-03-24T12:45:51.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:51.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:51.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:51.785 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:51.802 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:51.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:51.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:51.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:51.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:51.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:52.042 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:52.070 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:52.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:52.132 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:45:52.132 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:45:52.132 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:45:52.148 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:45:52.169 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:45:52.172 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:52.190 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:45:52.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:52.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:52.211 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:45:52.211 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:45:52.211 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:45:52.211 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:45:52.212 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:45:52.212 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:45:52.212 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:45:52.212 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:45:52.212 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:45:52.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:52.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:52.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:52.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:52.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:52.453 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:52.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:52.522 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:52 smithi029 ceph-mon[29919]: 5.17 scrub starts 2024-03-24T12:45:52.523 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:52 smithi029 ceph-mon[29919]: 5.17 scrub ok 2024-03-24T12:45:52.523 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:52 smithi029 ceph-mon[29919]: 5.14 scrub starts 2024-03-24T12:45:52.523 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:52 smithi029 ceph-mon[29919]: 5.14 scrub ok 2024-03-24T12:45:52.523 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:52 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3492886879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:45:52.523 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:52 smithi029 ceph-mon[41500]: 5.17 scrub starts 2024-03-24T12:45:52.523 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:52 smithi029 ceph-mon[41500]: 5.17 scrub ok 2024-03-24T12:45:52.523 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:52 smithi029 ceph-mon[41500]: 5.14 scrub starts 2024-03-24T12:45:52.523 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:52 smithi029 ceph-mon[41500]: 5.14 scrub ok 2024-03-24T12:45:52.523 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:52 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3492886879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:45:52.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:52.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:52.580 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:52.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:52.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:52 smithi190 ceph-mon[33748]: 5.17 scrub starts 2024-03-24T12:45:52.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:52 smithi190 ceph-mon[33748]: 5.17 scrub ok 2024-03-24T12:45:52.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:52 smithi190 ceph-mon[33748]: 5.14 scrub starts 2024-03-24T12:45:52.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:52 smithi190 ceph-mon[33748]: 5.14 scrub ok 2024-03-24T12:45:52.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:52 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3492886879' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:45:52.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:52.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:52.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:52.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:52.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:52.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:52.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:52.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:52.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:52.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:52.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:52.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:53.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:53.094 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:53.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:53.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:53.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:53.220 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:53.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:53.317 INFO:teuthology.orchestra.run.smithi029.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-24T12:45:53.322 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:53.349 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:53.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:53.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:53.476 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:53.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:53.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: 3.18 scrub starts 2024-03-24T12:45:53.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: 3.18 scrub ok 2024-03-24T12:45:53.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:45:53.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:45:53.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: pgmap v879: 158 pgs: 158 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-03-24T12:45:53.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2205043248' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: 1.9 scrub starts 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: 1.9 scrub ok 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: 4.14 scrub starts 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[29919]: 4.14 scrub ok 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: 3.18 scrub starts 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: 3.18 scrub ok 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: pgmap v879: 158 pgs: 158 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-03-24T12:45:53.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2205043248' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:45:53.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:45:53.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: 1.9 scrub starts 2024-03-24T12:45:53.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: 1.9 scrub ok 2024-03-24T12:45:53.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: 4.14 scrub starts 2024-03-24T12:45:53.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:53 smithi029 ceph-mon[41500]: 4.14 scrub ok 2024-03-24T12:45:53.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:53.603 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:53.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: 3.18 scrub starts 2024-03-24T12:45:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: 3.18 scrub ok 2024-03-24T12:45:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:45:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:45:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: pgmap v879: 158 pgs: 158 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 0 B/s, 0 objects/s recovering 2024-03-24T12:45:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2205043248' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:45:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T12:45:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: 1.9 scrub starts 2024-03-24T12:45:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: 1.9 scrub ok 2024-03-24T12:45:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: 4.14 scrub starts 2024-03-24T12:45:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:53 smithi190 ceph-mon[33748]: 4.14 scrub ok 2024-03-24T12:45:53.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:53.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:53.729 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:53.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:53.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:53.858 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:53.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:53.962 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:53.990 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:53.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:53.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:54.092 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:54.117 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:54.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:54.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:54.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:54.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:54.244 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:54.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:54.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:54.370 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:54.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:54.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:54.495 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:54.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:54.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:54.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:54.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:54.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:54 smithi190 ceph-mon[33748]: 4.0 scrub starts 2024-03-24T12:45:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:54 smithi190 ceph-mon[33748]: 4.0 scrub ok 2024-03-24T12:45:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:54 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:45:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:54 smithi190 ceph-mon[33748]: osdmap e387: 8 total, 7 up, 6 in 2024-03-24T12:45:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:54 smithi029 ceph-mon[29919]: 4.0 scrub starts 2024-03-24T12:45:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:54 smithi029 ceph-mon[29919]: 4.0 scrub ok 2024-03-24T12:45:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:54 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:45:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:54 smithi029 ceph-mon[29919]: osdmap e387: 8 total, 7 up, 6 in 2024-03-24T12:45:54.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:54 smithi029 ceph-mon[41500]: 4.0 scrub starts 2024-03-24T12:45:54.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:54 smithi029 ceph-mon[41500]: 4.0 scrub ok 2024-03-24T12:45:54.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:54 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-24T12:45:54.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:54 smithi029 ceph-mon[41500]: osdmap e387: 8 total, 7 up, 6 in 2024-03-24T12:45:54.727 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:54.756 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:54.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:54.831 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:54.831 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:45:54.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:54.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:54.882 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:54.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:54.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:55.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:55.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:55.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:55.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:55.136 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:55.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:55.238 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:55.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:55.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:55.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:55.393 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:55.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:55.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:55 smithi029 ceph-mon[29919]: 5.3 scrub starts 2024-03-24T12:45:55.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:55 smithi029 ceph-mon[29919]: 5.3 scrub ok 2024-03-24T12:45:55.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:55 smithi029 ceph-mon[29919]: pgmap v881: 158 pgs: 158 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:45:55.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:55 smithi029 ceph-mon[29919]: osdmap e388: 8 total, 7 up, 6 in 2024-03-24T12:45:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:55 smithi029 ceph-mon[41500]: 5.3 scrub starts 2024-03-24T12:45:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:55 smithi029 ceph-mon[41500]: 5.3 scrub ok 2024-03-24T12:45:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:55 smithi029 ceph-mon[41500]: pgmap v881: 158 pgs: 158 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:45:55.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:55 smithi029 ceph-mon[41500]: osdmap e388: 8 total, 7 up, 6 in 2024-03-24T12:45:55.494 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:55.514 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:45:55.514 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:45:55.518 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:55.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:55.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:55 smithi190 ceph-mon[33748]: 5.3 scrub starts 2024-03-24T12:45:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:55 smithi190 ceph-mon[33748]: 5.3 scrub ok 2024-03-24T12:45:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:55 smithi190 ceph-mon[33748]: pgmap v881: 158 pgs: 158 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:45:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:55 smithi190 ceph-mon[33748]: osdmap e388: 8 total, 7 up, 6 in 2024-03-24T12:45:55.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:55.645 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:55.645 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:55.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:55.774 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:55.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:55.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:55.792 DEBUG:teuthology.orchestra.run.smithi029:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-24T12:45:55.794 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-24T12:45:55.794 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-24T12:45:55.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:55.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:55.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:55.907 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:hints = 1 2024-03-24T12:45:55.907 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-24T12:45:55.907 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Object prefix: benchmark_data_smithi029_192894 2024-03-24T12:45:55.908 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:45:55.908 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 0 0 0 0 0 0 - 0 2024-03-24T12:45:56.004 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:45:56.006 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:56.007 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:45:56.007 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 388, 2024-03-24T12:45:56.007 INFO:teuthology.orchestra.run.smithi029.stdout: "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:45:56.007 INFO:teuthology.orchestra.run.smithi029.stdout: "created": "2024-03-24T12:20:18.712998+0000", 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "modified": "2024-03-24T12:45:55.216773+0000", 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "last_up_change": "2024-03-24T12:35:57.677873+0000", 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "last_in_change": "2024-03-24T12:45:12.212720+0000", 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_num": 5804032, 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_set": [ 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "nodeep-scrub", 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "pglog_hardlimit", 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "purged_snapdirs", 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_deletes", 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: "sortbitwise" 2024-03-24T12:45:56.008 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_version": 18, 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_snapshot": "", 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_max": 11, 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd": 8, 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "min_compat_client": "luminous", 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "require_osd_release": "squid", 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_crimson": false, 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "pools": [ 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.009 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 1, 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".mgr", 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:23:41.491597+0000", 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:45:56.010 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 11, 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 11, 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 11, 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 11, 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 11, 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:45:56.011 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "295", 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "293", 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:45:56.012 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:45:56.013 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:45:56.014 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_max": 32, 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_min": 1 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": {} 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.9100000858306885, 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.9100000858306885, 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:45:56.015 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.1800000667572021, 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2.1800000667572021, 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 2, 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "datapool", 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:01.465080+0000", 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 8193, 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:45:56.016 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 3, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 3, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 3, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 3, 2024-03-24T12:45:56.017 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 3, 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "71", 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:45:56.018 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 2, 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 71, 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:45:56.019 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:45:56.020 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd": {} 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:45:56.021 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.6600000858306885, 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.6600000858306885, 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.5, 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.3300000429153442, 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.3300000429153442, 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 3, 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".rgw.root", 2024-03-24T12:45:56.022 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:02.100021+0000", 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:45:56.023 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "68", 2024-03-24T12:45:56.024 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:45:56.025 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:45:56.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:45:56.027 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.25, 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.25, 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.690000057220459, 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.690000057220459, 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:45:56.028 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 4, 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:03.473862+0000", 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:45:56.029 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:45:56.030 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:45:56.030 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:45:56.030 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:45:56.030 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:45:56.030 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:45:56.030 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:45:56.030 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:45:56.030 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:45:56.030 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "70", 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:45:56.031 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:45:56.032 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:45:56.033 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:45:56.034 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 3, 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 3, 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.25, 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2.25, 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 5, 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:45:56.035 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:05.573828+0000", 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:45:56.036 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "72", 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:45:56.038 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:45:56.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:45:56.040 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:45:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:45:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:45:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2, 2024-03-24T12:45:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2, 2024-03-24T12:45:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:45:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.5, 2024-03-24T12:45:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.5, 2024-03-24T12:45:56.041 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:45:56.042 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 6, 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:07.645083+0000", 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:45:56.043 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:45:56.044 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "74", 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:45:56.045 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:45:56.046 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:45:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_bias": 4 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:45:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2, 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2, 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.5, 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.5, 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 11, 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "unique_pool_4", 2024-03-24T12:45:56.049 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:45:41.907152+0000", 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:45:56.050 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 16, 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 16, 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 16, 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 16, 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 16, 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "387", 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:45:56.051 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:45:56.052 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:45:56.053 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: "rados": {} 2024-03-24T12:45:56.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.5099999904632568, 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.5099999904632568, 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:45:56.055 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:45:56.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.056 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:45:56.056 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b", 2024-03-24T12:45:56.056 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:45:56.056 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:45:56.056 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:45:56.056 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:45:56.056 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:45:56.056 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 8, 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 382, 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6801", 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.057 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6802", 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.058 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6804", 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6803", 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.059 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6801/2514818464", 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6802/2514818464", 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6804/2514818464", 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6803/2514818464", 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc", 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:45:56.060 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 0.575775146484375, 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 13, 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 382, 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6805", 2024-03-24T12:45:56.061 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6806", 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.062 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6808", 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6807", 2024-03-24T12:45:56.063 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6805/2923396572", 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6806/2923396572", 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6808/2923396572", 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6807/2923396572", 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.064 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "c3f83505-25ee-4217-b60a-081c36092e92", 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 18, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 382, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6809", 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6810", 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:45:56.066 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6812", 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:45:56.067 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6811", 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6809/126032915", 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6810/126032915", 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6812/126032915", 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6811/126032915", 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:45:56.068 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96", 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 0.3369293212890625, 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 27, 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 382, 2024-03-24T12:45:56.069 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6813", 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6814", 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6816", 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.071 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6815", 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6813/1080171824", 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6814/1080171824", 2024-03-24T12:45:56.072 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6816/1080171824", 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6815/1080171824", 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "5e420304-2798-4d11-9688-d0d89482008c", 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:45:56.073 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 33, 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 382, 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6800", 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:45:56.074 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6801", 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6803", 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6802", 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.076 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6800/3690258869", 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6801/3690258869", 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6803/3690258869", 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6802/3690258869", 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:45:56.077 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "1213431a-652c-4cbf-b32a-36335430c666", 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 0, 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 0, 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 0, 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 42, 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 204, 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.078 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6804", 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6805", 2024-03-24T12:45:56.079 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:45:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:56.080 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:56.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:56.081 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.081 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.081 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:45:56.081 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.081 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6807", 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6806", 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.082 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6804/1327220488", 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6805/1327220488", 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6807/1327220488", 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6806/1327220488", 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: "exists" 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c", 2024-03-24T12:45:56.083 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 0, 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 0, 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 51, 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 369, 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.084 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6808", 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6809", 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.085 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6811", 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.086 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6810", 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6808/1295064976", 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6809/1295064976", 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6811/1295064976", 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6810/1295064976", 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.087 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda", 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 0.828521728515625, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 60, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 382, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:45:56.088 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6812", 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6813", 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:45:56.089 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6815", 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:45:56.090 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6814", 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6812/110963949", 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6813/110963949", 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6815/110963949", 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6814/110963949", 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:45:56.091 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_xinfo": [ 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:45:07.385141+0000", 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:45:56.092 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:45:40.143627+0000", 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:45:56.093 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:45:01.034618+0000", 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:45:56.094 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:45:42.684943+0000", 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:45:02.995647+0000", 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.095 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 65536, 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:35:04.518484+0000", 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 203 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:45:56.096 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 65536, 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:45:44.075331+0000", 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:45:04.429228+0000", 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:45:56.097 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap": [], 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_items": [ 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.b", 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.098 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.f", 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_primaries": [], 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_temp": [], 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_temp": [], 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklist": { 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2633022206": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:45:56.099 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2612078930": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2365028933": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4113609697": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4130171883": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3697529912": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2929430696": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/327626505": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3715448582": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/861146182": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2109454792": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1631806715": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/852212806": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1823252077": "2024-03-25T12:21:15.151901+0000" 2024-03-24T12:45:56.100 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "range_blocklist": {}, 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profiles": { 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "default": { 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-failure-domain": "osd", 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "k": "2", 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "m": "1", 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "plugin": "jerasure", 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "technique": "reed_sol_van" 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps_queue": [], 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "new_removed_snaps": [], 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "new_purged_snaps": [], 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_node_flags": {}, 2024-03-24T12:45:56.101 INFO:teuthology.orchestra.run.smithi029.stdout: "device_class_flags": {}, 2024-03-24T12:45:56.102 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode": { 2024-03-24T12:45:56.102 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_enabled": false, 2024-03-24T12:45:56.102 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_bucket_count": 0, 2024-03-24T12:45:56.102 INFO:teuthology.orchestra.run.smithi029.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:45:56.102 INFO:teuthology.orchestra.run.smithi029.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:45:56.102 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_bucket": 0 2024-03-24T12:45:56.102 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:45:56.102 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:45:56.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:56.223 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:56.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:56.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:56.325 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:56.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:56.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:56.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:56.455 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:56.495 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:56.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:56.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:56.572 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:45:56.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:45:56.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[41500]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:45:56.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1263828020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:45:56.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[41500]: 3.7 scrub starts 2024-03-24T12:45:56.573 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[41500]: 3.7 scrub ok 2024-03-24T12:45:56.573 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:45:56.574 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:45:56.574 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[29919]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:45:56.574 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1263828020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:45:56.574 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[29919]: 3.7 scrub starts 2024-03-24T12:45:56.574 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:56 smithi029 ceph-mon[29919]: 3.7 scrub ok 2024-03-24T12:45:56.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:56.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:56.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:56 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:45:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:56 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:45:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:56 smithi190 ceph-mon[33748]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T12:45:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:56 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1263828020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:45:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:56 smithi190 ceph-mon[33748]: 3.7 scrub starts 2024-03-24T12:45:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:56 smithi190 ceph-mon[33748]: 3.7 scrub ok 2024-03-24T12:45:56.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:56.769 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:56.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:56.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:56.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:56.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:56.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:56.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:56.908 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 1 15 2989 2974 185.864 185.875 0.00368208 0.00536071 2024-03-24T12:45:57.002 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:57.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:57.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:57.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:57.163 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:57.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:57.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:57.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:57.305 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:57.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:57.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:57.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:57.448 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:57.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:57.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:57.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:45:57.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:57.576 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:57.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:57.577 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:57.606 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[41500]: 3.1b scrub starts 2024-03-24T12:45:57.606 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[41500]: 3.1b scrub ok 2024-03-24T12:45:57.606 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[41500]: 4.1d scrub starts 2024-03-24T12:45:57.606 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[41500]: 4.1d scrub ok 2024-03-24T12:45:57.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[41500]: 3.1a scrub starts 2024-03-24T12:45:57.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[41500]: 3.1a scrub ok 2024-03-24T12:45:57.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[41500]: pgmap v883: 158 pgs: 158 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:45:57.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[29919]: 3.1b scrub starts 2024-03-24T12:45:57.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[29919]: 3.1b scrub ok 2024-03-24T12:45:57.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[29919]: 4.1d scrub starts 2024-03-24T12:45:57.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[29919]: 4.1d scrub ok 2024-03-24T12:45:57.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[29919]: 3.1a scrub starts 2024-03-24T12:45:57.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[29919]: 3.1a scrub ok 2024-03-24T12:45:57.607 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-mon[29919]: pgmap v883: 158 pgs: 158 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:45:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:57 smithi190 ceph-mon[33748]: 3.1b scrub starts 2024-03-24T12:45:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:57 smithi190 ceph-mon[33748]: 3.1b scrub ok 2024-03-24T12:45:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:57 smithi190 ceph-mon[33748]: 4.1d scrub starts 2024-03-24T12:45:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:57 smithi190 ceph-mon[33748]: 4.1d scrub ok 2024-03-24T12:45:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:57 smithi190 ceph-mon[33748]: 3.1a scrub starts 2024-03-24T12:45:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:57 smithi190 ceph-mon[33748]: 3.1a scrub ok 2024-03-24T12:45:57.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:57 smithi190 ceph-mon[33748]: pgmap v883: 158 pgs: 158 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:45:57.678 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:57.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:57.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:57.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:57.878 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:57.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:57.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:57.894 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:45:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:45:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:45:57.908 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 2 16 5766 5750 179.597 173.5 0.00584254 0.00555457 2024-03-24T12:45:57.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:58.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:58.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:58.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:58.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:58.151 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:58.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:58.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:58.277 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:58.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:58.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:58.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:58.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:58.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:58.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:58.551 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:58.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:58.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:58 smithi190 ceph-mon[33748]: 3.b scrub starts 2024-03-24T12:45:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:58 smithi190 ceph-mon[33748]: 3.b scrub ok 2024-03-24T12:45:58.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:45:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:45:58.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:45:58.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:58 smithi029 ceph-mon[29919]: 3.b scrub starts 2024-03-24T12:45:58.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:58 smithi029 ceph-mon[29919]: 3.b scrub ok 2024-03-24T12:45:58.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:58 smithi029 ceph-mon[41500]: 3.b scrub starts 2024-03-24T12:45:58.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:58 smithi029 ceph-mon[41500]: 3.b scrub ok 2024-03-24T12:45:58.692 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:58.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:58.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:45:58.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:45:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:58.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:58.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:45:58.909 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 3 16 8033 8017 166.96 141.688 0.00378393 0.00597796 2024-03-24T12:45:58.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:58.962 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:58.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:58.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:59.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:59.112 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:59.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:59.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:45:59.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:45:59.256 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:59.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:59.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:45:59.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:59.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:59.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:59.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:59.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:45:59.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:59.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:45:59.622 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:45:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:59 smithi190 ceph-mon[33748]: 3.8 scrub starts 2024-03-24T12:45:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:59 smithi190 ceph-mon[33748]: 3.8 scrub ok 2024-03-24T12:45:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:59 smithi190 ceph-mon[33748]: pgmap v884: 158 pgs: 158 active+clean; 262 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 33 MiB/s wr, 514 op/s 2024-03-24T12:45:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:59 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:45:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:45:59 smithi190 ceph-mon[33748]: 3.13 scrub ok 2024-03-24T12:45:59.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:59.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:59.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:45:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[41500]: 3.8 scrub starts 2024-03-24T12:45:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[41500]: 3.8 scrub ok 2024-03-24T12:45:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[41500]: pgmap v884: 158 pgs: 158 active+clean; 262 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 33 MiB/s wr, 514 op/s 2024-03-24T12:45:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:45:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[41500]: 3.13 scrub ok 2024-03-24T12:45:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[29919]: 3.8 scrub starts 2024-03-24T12:45:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[29919]: 3.8 scrub ok 2024-03-24T12:45:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:45:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[29919]: pgmap v884: 158 pgs: 158 active+clean; 262 MiB data, 4.5 GiB used, 532 GiB / 536 GiB avail; 33 MiB/s wr, 514 op/s 2024-03-24T12:45:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:45:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:45:59 smithi029 ceph-mon[29919]: 3.13 scrub ok 2024-03-24T12:45:59.753 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:45:59.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:45:59.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:59.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:45:59.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:45:59.910 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 4 16 10653 10637 166.112 163.75 0.00885734 0.00601044 2024-03-24T12:45:59.929 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:45:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:45:59.930 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:00.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:00.072 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:00.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:00.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:00.142 INFO:tasks.thrashosds.thrasher:j is {'epoch': 388, 'fsid': 'bc017092-e9d8-11ee-95cd-87774f69a715', 'created': '2024-03-24T12:20:18.712998+0000', 'modified': '2024-03-24T12:45:55.216773+0000', 'last_up_change': '2024-03-24T12:35:57.677873+0000', 'last_in_change': '2024-03-24T12:45:12.212720+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T12:23:41.491597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '295', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '293', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9100000858306885, 'score_stable': 2.9100000858306885, 'optimal_score': 0.75, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-24T12:26:01.465080+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6600000858306885, 'score_stable': 2.6600000858306885, 'optimal_score': 0.5, 'raw_score_acting': 1.3300000429153442, 'raw_score_stable': 1.3300000429153442, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T12:26:02.100021+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T12:26:03.473862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T12:26:05.573828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T12:26:07.645083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-24T12:45:41.907152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '387', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '0aa1c49b-7887-4f74-b03c-08cc0adec90b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6801', 'nonce': 2514818464}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6802', 'nonce': 2514818464}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6804', 'nonce': 2514818464}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6803', 'nonce': 2514818464}]}, 'public_addr': '172.21.15.29:6801/2514818464', 'cluster_addr': '172.21.15.29:6802/2514818464', 'heartbeat_back_addr': '172.21.15.29:6804/2514818464', 'heartbeat_front_addr': '172.21.15.29:6803/2514818464', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99d8ef8f-eb59-4882-ab51-b398e6b53bcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.575775146484375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6805', 'nonce': 2923396572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6806', 'nonce': 2923396572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6808', 'nonce': 2923396572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6807', 'nonce': 2923396572}]}, 'public_addr': '172.21.15.29:6805/2923396572', 'cluster_addr': '172.21.15.29:6806/2923396572', 'heartbeat_back_addr': '172.21.15.29:6808/2923396572', 'heartbeat_front_addr': '172.21.15.29:6807/2923396572', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c3f83505-25ee-4217-b60a-081c36092e92', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6809', 'nonce': 126032915}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6810', 'nonce': 126032915}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6812', 'nonce': 126032915}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6811', 'nonce': 126032915}]}, 'public_addr': '172.21.15.29:6809/126032915', 'cluster_addr': '172.21.15.29:6810/126032915', 'heartbeat_back_addr': '172.21.15.29:6812/126032915', 'heartbeat_front_addr': '172.21.15.29:6811/126032915', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f5e479a8-459a-4315-a808-711a7b84bb96', 'up': 1, 'in': 1, 'weight': 0.3369293212890625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6813', 'nonce': 1080171824}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6814', 'nonce': 1080171824}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6816', 'nonce': 1080171824}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6815', 'nonce': 1080171824}]}, 'public_addr': '172.21.15.29:6813/1080171824', 'cluster_addr': '172.21.15.29:6814/1080171824', 'heartbeat_back_addr': '172.21.15.29:6816/1080171824', 'heartbeat_front_addr': '172.21.15.29:6815/1080171824', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '5e420304-2798-4d11-9688-d0d89482008c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 3690258869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6801', 'nonce': 3690258869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6803', 'nonce': 3690258869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 3690258869}]}, 'public_addr': '172.21.15.190:6800/3690258869', 'cluster_addr': '172.21.15.190:6801/3690258869', 'heartbeat_back_addr': '172.21.15.190:6803/3690258869', 'heartbeat_front_addr': '172.21.15.190:6802/3690258869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1213431a-652c-4cbf-b32a-36335430c666', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 158, 'down_at': 204, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 1327220488}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6805', 'nonce': 1327220488}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6807', 'nonce': 1327220488}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 1327220488}]}, 'public_addr': '172.21.15.190:6804/1327220488', 'cluster_addr': '172.21.15.190:6805/1327220488', 'heartbeat_back_addr': '172.21.15.190:6807/1327220488', 'heartbeat_front_addr': '172.21.15.190:6806/1327220488', 'state': ['exists']}, {'osd': 6, 'uuid': '0712e511-0257-44f3-a82f-ee09ab957c0c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 369, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 1295064976}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6809', 'nonce': 1295064976}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6811', 'nonce': 1295064976}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 1295064976}]}, 'public_addr': '172.21.15.190:6808/1295064976', 'cluster_addr': '172.21.15.190:6809/1295064976', 'heartbeat_back_addr': '172.21.15.190:6811/1295064976', 'heartbeat_front_addr': '172.21.15.190:6810/1295064976', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7e36bde4-14d0-4e7e-9a82-2bb97859dbda', 'up': 1, 'in': 1, 'weight': 0.828521728515625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 110963949}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6813', 'nonce': 110963949}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6815', 'nonce': 110963949}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 110963949}]}, 'public_addr': '172.21.15.190:6812/110963949', 'cluster_addr': '172.21.15.190:6813/110963949', 'heartbeat_back_addr': '172.21.15.190:6815/110963949', 'heartbeat_front_addr': '172.21.15.190:6814/110963949', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:45:07.385141+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:45:40.143627+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:45:01.034618+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:45:42.684943+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:45:02.995647+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-24T12:35:04.518484+0000', 'dead_epoch': 203}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-24T12:45:44.075331+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:45:04.429228+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.29:0/2633022206': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2612078930': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2365028933': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4113609697': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4130171883': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/3697529912': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2929430696': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/327626505': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/3715448582': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/861146182': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2109454792': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/1631806715': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:6800/852212806': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/1823252077': '2024-03-25T12:21:15.151901+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:46:00.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:46:00.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:00.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:00.228 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:00.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:00.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:00.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:00.376 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:00.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:00.478 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:00.508 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:00.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:00.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:00.610 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:00.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:00 smithi190 ceph-mon[33748]: 4.b scrub starts 2024-03-24T12:46:00.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:00 smithi190 ceph-mon[33748]: 4.b scrub ok 2024-03-24T12:46:00.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:00 smithi190 ceph-mon[33748]: 6.b scrub starts 2024-03-24T12:46:00.638 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:00.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:00.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:00.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:00 smithi029 ceph-mon[29919]: 4.b scrub starts 2024-03-24T12:46:00.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:00 smithi029 ceph-mon[29919]: 4.b scrub ok 2024-03-24T12:46:00.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:00 smithi029 ceph-mon[29919]: 6.b scrub starts 2024-03-24T12:46:00.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:00 smithi029 ceph-mon[41500]: 4.b scrub starts 2024-03-24T12:46:00.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:00 smithi029 ceph-mon[41500]: 4.b scrub ok 2024-03-24T12:46:00.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:00 smithi029 ceph-mon[41500]: 6.b scrub starts 2024-03-24T12:46:00.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:00.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:00.879 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:00.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:00.910 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 5 15 12806 12791 159.815 134.625 0.00602572 0.00625033 2024-03-24T12:46:00.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:01.010 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:01.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:01.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:01.156 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:01.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:01.167 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:46:01.167 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:46:01.257 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:01.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:01.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:01.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:01.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:01.418 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:01.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:01.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:01.563 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:01.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:01.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:01.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:01 smithi029 ceph-mon[41500]: 6.b scrub ok 2024-03-24T12:46:01.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:01 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:46:01.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:01 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:46:01.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:01 smithi029 ceph-mon[41500]: pgmap v885: 158 pgs: 158 active+clean; 427 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 53 MiB/s wr, 852 op/s 2024-03-24T12:46:01.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:01 smithi029 ceph-mon[29919]: 6.b scrub ok 2024-03-24T12:46:01.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:01 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:46:01.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:01 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:46:01.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:01 smithi029 ceph-mon[29919]: pgmap v885: 158 pgs: 158 active+clean; 427 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 53 MiB/s wr, 852 op/s 2024-03-24T12:46:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:01 smithi190 ceph-mon[33748]: 6.b scrub ok 2024-03-24T12:46:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:01 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:46:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:01 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:46:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:01 smithi190 ceph-mon[33748]: pgmap v885: 158 pgs: 158 active+clean; 427 MiB data, 5.4 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 53 MiB/s wr, 852 op/s 2024-03-24T12:46:01.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:01.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:01.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:01.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:01.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:01.826 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:01.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:01.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:01.910 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 6 16 15039 15023 156.428 139.5 0.00510987 0.00638402 2024-03-24T12:46:01.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:01.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:01.969 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:01.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:02.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:02.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:02.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:02.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:02.237 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:02.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:02.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:02.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:02.357 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:46:02.365 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:02.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:02.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:02.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:02.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:02.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:02.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:02.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:02.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:02.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:46:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:46:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:46:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:46:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:46:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:46:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:46:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:46:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: 6.1f scrub starts 2024-03-24T12:46:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2167499558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:46:02.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:46:02.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:46:02.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: Cluster is now healthy 2024-03-24T12:46:02.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:02 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:46:02.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:46:02.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:46:02.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:46:02.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:46:02.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:46:02.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:46:02.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:46:02.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:46:02.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: 6.1f scrub starts 2024-03-24T12:46:02.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2167499558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:46:02.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:46:02.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:46:02.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: Cluster is now healthy 2024-03-24T12:46:02.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:46:02.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:46:02.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:46:02.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:46:02.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:46:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:46:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:46:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:46:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:46:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: 6.1f scrub starts 2024-03-24T12:46:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2167499558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:46:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:46:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:46:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: Cluster is now healthy 2024-03-24T12:46:02.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:02 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:46:02.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:02.771 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:02.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:02.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:02.873 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:02.910 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 7 16 16898 16882 150.68 116.188 0.00532617 0.00662826 2024-03-24T12:46:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:02.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:02.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:03.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:03.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T12:46:03.045 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:03.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:03.147 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:03.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:03.176 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:03.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:03.206 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:46:03.228 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:46:03.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:03.325 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:03.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:03.330 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:46:03.330+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:46:03.425 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:03.453 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:03.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:03.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:03.606 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:03.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:03 smithi190 ceph-mon[33748]: 6.1f scrub ok 2024-03-24T12:46:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:03 smithi190 ceph-mon[33748]: osdmap e389: 8 total, 7 up, 6 in 2024-03-24T12:46:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:03 smithi190 ceph-mon[33748]: 4.12 scrub starts 2024-03-24T12:46:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:03 smithi190 ceph-mon[33748]: 4.12 scrub ok 2024-03-24T12:46:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:03 smithi190 ceph-mon[33748]: pgmap v887: 158 pgs: 158 active+clean; 689 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T12:46:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:03 smithi190 ceph-mon[33748]: 6.3 deep-scrub starts 2024-03-24T12:46:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:46:03.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:03.649 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[41500]: 6.1f scrub ok 2024-03-24T12:46:03.649 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[41500]: osdmap e389: 8 total, 7 up, 6 in 2024-03-24T12:46:03.649 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[41500]: 4.12 scrub starts 2024-03-24T12:46:03.649 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[41500]: 4.12 scrub ok 2024-03-24T12:46:03.649 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[41500]: pgmap v887: 158 pgs: 158 active+clean; 689 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T12:46:03.650 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[41500]: 6.3 deep-scrub starts 2024-03-24T12:46:03.650 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:46:03.650 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:03.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[29919]: 6.1f scrub ok 2024-03-24T12:46:03.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[29919]: osdmap e389: 8 total, 7 up, 6 in 2024-03-24T12:46:03.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[29919]: 4.12 scrub starts 2024-03-24T12:46:03.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[29919]: 4.12 scrub ok 2024-03-24T12:46:03.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[29919]: pgmap v887: 158 pgs: 158 active+clean; 689 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-24T12:46:03.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[29919]: 6.3 deep-scrub starts 2024-03-24T12:46:03.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:46:03.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:03.708 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:03.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:03.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:03.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:03.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:03.885 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:03.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:03.910 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 8 15 18901 18886 147.5 125.25 0.00544034 0.0067727 2024-03-24T12:46:03.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:04.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:04.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:04.112 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:04.141 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:04.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:04.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:04.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:04.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:04.275 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:04.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:04.376 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:04.417 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:04.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:04.519 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:04.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:04.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:04 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:46:04.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:04 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:46:04.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:04 smithi190 ceph-mon[33748]: 6.3 deep-scrub ok 2024-03-24T12:46:04.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:04 smithi190 ceph-mon[33748]: osdmap e390: 8 total, 7 up, 6 in 2024-03-24T12:46:04.662 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:04 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:46:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:04 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:46:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:04 smithi029 ceph-mon[29919]: 6.3 deep-scrub ok 2024-03-24T12:46:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:04 smithi029 ceph-mon[29919]: osdmap e390: 8 total, 7 up, 6 in 2024-03-24T12:46:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:04 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:46:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:04 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:46:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:04 smithi029 ceph-mon[41500]: 6.3 deep-scrub ok 2024-03-24T12:46:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:04 smithi029 ceph-mon[41500]: osdmap e390: 8 total, 7 up, 6 in 2024-03-24T12:46:04.688 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:04.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:04.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:04.829 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:04.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:04.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:04.910 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 9 15 20558 20543 142.618 103.562 0.00627585 0.00700504 2024-03-24T12:46:04.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:04.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:05.068 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:05.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:05.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:05.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:05.240 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:05.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:05.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:05.373 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:05.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:05.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:05.475 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:05.516 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:05.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:05.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:05 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:46:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:05 smithi190 ceph-mon[33748]: 4.4 scrub ok 2024-03-24T12:46:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:05 smithi190 ceph-mon[33748]: pgmap v889: 158 pgs: 158 active+clean; 1022 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-24T12:46:05.656 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:05.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:05.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:05.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:05 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:46:05.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:05 smithi029 ceph-mon[41500]: 4.4 scrub ok 2024-03-24T12:46:05.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:05 smithi029 ceph-mon[41500]: pgmap v889: 158 pgs: 158 active+clean; 1022 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-24T12:46:05.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:05 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:46:05.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:05 smithi029 ceph-mon[29919]: 4.4 scrub ok 2024-03-24T12:46:05.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:05 smithi029 ceph-mon[29919]: pgmap v889: 158 pgs: 158 active+clean; 1022 MiB data, 6.9 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 128 MiB/s wr, 2.04k op/s 2024-03-24T12:46:05.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:05.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:05.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:05.910 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 10 15 22037 22022 137.6 92.4375 0.00853169 0.00726064 2024-03-24T12:46:06.022 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:06.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:06.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:06.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:06.160 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:06.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:06.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:06.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:06.287 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:06.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:06.388 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:06.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:06.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:06.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:06.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:06.555 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:06.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:06.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:06 smithi190 ceph-mon[33748]: 4.6 scrub starts 2024-03-24T12:46:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:06 smithi190 ceph-mon[33748]: 4.6 scrub ok 2024-03-24T12:46:06.657 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:06.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:06 smithi029 ceph-mon[29919]: 4.6 scrub starts 2024-03-24T12:46:06.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:06 smithi029 ceph-mon[29919]: 4.6 scrub ok 2024-03-24T12:46:06.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:06 smithi029 ceph-mon[41500]: 4.6 scrub starts 2024-03-24T12:46:06.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:06 smithi029 ceph-mon[41500]: 4.6 scrub ok 2024-03-24T12:46:06.682 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:06.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:06.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:06.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:06.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:06.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:06.911 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 11 16 23552 23536 133.691 94.625 0.0132547 0.00747258 2024-03-24T12:46:06.916 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:06.953 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:06.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:07.055 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:07.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:07.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:07.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:07.227 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:07.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:07.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:07.329 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:07.353 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:07.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:07.454 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:07.482 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:07.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:07.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:07.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:07.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:07.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:07.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:07 smithi029 ceph-mon[29919]: 6.18 scrub starts 2024-03-24T12:46:07.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:07 smithi029 ceph-mon[29919]: 6.18 scrub ok 2024-03-24T12:46:07.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:07 smithi029 ceph-mon[29919]: pgmap v890: 158 pgs: 158 active+clean; 1.1 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-24T12:46:07.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:46:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:46:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:46:07.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:07 smithi029 ceph-mon[41500]: 6.18 scrub starts 2024-03-24T12:46:07.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:07 smithi029 ceph-mon[41500]: 6.18 scrub ok 2024-03-24T12:46:07.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:07 smithi029 ceph-mon[41500]: pgmap v890: 158 pgs: 158 active+clean; 1.1 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-24T12:46:07.728 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:07.768 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:07.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:07.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:07.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:07.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:07 smithi190 ceph-mon[33748]: 6.18 scrub starts 2024-03-24T12:46:07.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:07 smithi190 ceph-mon[33748]: 6.18 scrub ok 2024-03-24T12:46:07.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:07 smithi190 ceph-mon[33748]: pgmap v890: 158 pgs: 158 active+clean; 1.1 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-03-24T12:46:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:07.901 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:07.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:07.911 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 12 16 25401 25385 132.18 115.562 0.00318544 0.00755576 2024-03-24T12:46:08.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:08.026 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:08.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:08.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:08.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:08.173 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:08.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:08.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:08.297 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:08.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:08.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:08.424 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:08.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:08.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:08.567 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:08.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:08.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:08 smithi029 ceph-mon[29919]: 6.a scrub starts 2024-03-24T12:46:08.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:08 smithi029 ceph-mon[29919]: 6.a scrub ok 2024-03-24T12:46:08.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:08 smithi029 ceph-mon[29919]: 6.c deep-scrub starts 2024-03-24T12:46:08.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:08 smithi029 ceph-mon[29919]: 6.c deep-scrub ok 2024-03-24T12:46:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:08 smithi029 ceph-mon[41500]: 6.a scrub starts 2024-03-24T12:46:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:08 smithi029 ceph-mon[41500]: 6.a scrub ok 2024-03-24T12:46:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:08 smithi029 ceph-mon[41500]: 6.c deep-scrub starts 2024-03-24T12:46:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:08 smithi029 ceph-mon[41500]: 6.c deep-scrub ok 2024-03-24T12:46:08.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:08.694 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:08.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:08.796 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:08.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:08.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:08.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:08 smithi190 ceph-mon[33748]: 6.a scrub starts 2024-03-24T12:46:08.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:08 smithi190 ceph-mon[33748]: 6.a scrub ok 2024-03-24T12:46:08.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:08 smithi190 ceph-mon[33748]: 6.c deep-scrub starts 2024-03-24T12:46:08.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:08 smithi190 ceph-mon[33748]: 6.c deep-scrub ok 2024-03-24T12:46:08.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:46:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:46:08.911 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 13 15 26232 26217 126.013 52 0.0120122 0.00792935 2024-03-24T12:46:08.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:08.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:09.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:09.105 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:09.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:09.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:09.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:09.231 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:09.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:09.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:09.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:09.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:09.472 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:09.502 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:09.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:09.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:09.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:09.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:09.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:09.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:09.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:09 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:46:09.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:09 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:46:09.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:09.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:09 smithi029 ceph-mon[29919]: pgmap v891: 158 pgs: 158 active+clean; 1.5 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-03-24T12:46:09.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:09 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:46:09.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:09 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:46:09.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:09.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:09 smithi029 ceph-mon[41500]: pgmap v891: 158 pgs: 158 active+clean; 1.5 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-03-24T12:46:09.729 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:09.753 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:09.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:09.855 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:09.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:09 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:46:09.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:09 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:46:09.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:09.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:09 smithi190 ceph-mon[33748]: pgmap v891: 158 pgs: 158 active+clean; 1.5 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-03-24T12:46:09.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:09.883 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:09.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:09.911 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 14 16 27612 27596 123.168 86.1875 0.00319189 0.00811251 2024-03-24T12:46:09.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:10.026 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:10.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:10.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:10.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:10.156 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:10.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:10.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:10.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:10.300 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:10.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:10.301 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:10.402 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:10.433 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:10.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:10.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:10.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:10.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:10.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:10.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:10 smithi029 ceph-mon[29919]: 1.1 scrub starts 2024-03-24T12:46:10.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:10 smithi029 ceph-mon[29919]: 1.1 scrub ok 2024-03-24T12:46:10.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:10 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:46:10.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:10 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:46:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:10 smithi029 ceph-mon[41500]: 1.1 scrub starts 2024-03-24T12:46:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:10 smithi029 ceph-mon[41500]: 1.1 scrub ok 2024-03-24T12:46:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:10 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:46:10.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:10 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:46:10.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:10.705 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:10.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:10.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:10.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:10.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:10 smithi190 ceph-mon[33748]: 1.1 scrub starts 2024-03-24T12:46:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:10 smithi190 ceph-mon[33748]: 1.1 scrub ok 2024-03-24T12:46:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:10 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:46:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:10 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:46:10.911 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 15 16 28949 28933 120.527 83.5625 0.0101717 0.00821468 2024-03-24T12:46:10.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:10.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:10.961 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:10.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:11.062 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:11.086 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:11.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:11.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:11.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:11.230 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:11.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:11.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:11.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:11.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:11.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:11.481 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:11.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:11.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:11.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:11.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:11.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:11.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:46:11.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:46:11.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:46:11.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:46:11.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[41500]: pgmap v892: 158 pgs: 158 active+clean; 1.6 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-24T12:46:11.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:46:11.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:46:11.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:46:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:46:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:46:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:46:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[29919]: pgmap v892: 158 pgs: 158 active+clean; 1.6 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-24T12:46:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:46:11.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:11 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:46:11.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:11.765 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:11.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:11.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:11.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:11 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:46:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:11 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:46:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:11 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:46:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:11 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:46:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:11 smithi190 ceph-mon[33748]: pgmap v892: 158 pgs: 158 active+clean; 1.6 GiB data, 8.9 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 140 MiB/s wr, 2.24k op/s 2024-03-24T12:46:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:11 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:46:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:11 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:46:11.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:11.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:11.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:11.911 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 16 16 29958 29942 116.936 63.0625 0.00556376 0.00854492 2024-03-24T12:46:11.998 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:12.021 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:12.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:12.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:12.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:12.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:12.147 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:12.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:12.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:12.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:12.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:12.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:12.418 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:12.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:12.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:12.564 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:12.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:12.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:12.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:12.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:46:12.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:46:12.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:46:12.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:46:12.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[29919]: 3.e scrub starts 2024-03-24T12:46:12.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[29919]: 3.e scrub ok 2024-03-24T12:46:12.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:46:12.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:46:12.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:46:12.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:46:12.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[41500]: 3.e scrub starts 2024-03-24T12:46:12.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:12 smithi029 ceph-mon[41500]: 3.e scrub ok 2024-03-24T12:46:12.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:12.691 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:12.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:12.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:12.826 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:12.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:12.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:12 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:46:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:12 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:46:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:12 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:46:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:12 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:46:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:12 smithi190 ceph-mon[33748]: 3.e scrub starts 2024-03-24T12:46:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:12 smithi190 ceph-mon[33748]: 3.e scrub ok 2024-03-24T12:46:12.911 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 17 16 31043 31027 114.046 67.8125 0.00994889 0.0087624 2024-03-24T12:46:12.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:12.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:12.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:12.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:13.073 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:13.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:13.099 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:13.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:13.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:13.239 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:13.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:13.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:13.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:13.364 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:13.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:13.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:13.467 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:13.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:13.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:13.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:13.634 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:13.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:46:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:46:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: 3.10 scrub starts 2024-03-24T12:46:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: 3.10 scrub ok 2024-03-24T12:46:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: 3.c scrub starts 2024-03-24T12:46:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: 3.c scrub ok 2024-03-24T12:46:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:46:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:46:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: pgmap v893: 158 pgs: 158 active+clean; 1.7 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 13 KiB/s rd, 106 MiB/s wr, 1.71k op/s 2024-03-24T12:46:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: 5.1e scrub starts 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[29919]: 5.1e scrub ok 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: 3.10 scrub starts 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: 3.10 scrub ok 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: 3.c scrub starts 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: 3.c scrub ok 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: pgmap v893: 158 pgs: 158 active+clean; 1.7 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 13 KiB/s rd, 106 MiB/s wr, 1.71k op/s 2024-03-24T12:46:13.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: 5.1e scrub starts 2024-03-24T12:46:13.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:13 smithi029 ceph-mon[41500]: 5.1e scrub ok 2024-03-24T12:46:13.736 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:13.760 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:13.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:13.862 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:46:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:46:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: 3.10 scrub starts 2024-03-24T12:46:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: 3.10 scrub ok 2024-03-24T12:46:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: 3.c scrub starts 2024-03-24T12:46:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: 3.c scrub ok 2024-03-24T12:46:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:46:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:46:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: pgmap v893: 158 pgs: 158 active+clean; 1.7 GiB data, 9.3 GiB used, 527 GiB / 536 GiB avail; 13 KiB/s rd, 106 MiB/s wr, 1.71k op/s 2024-03-24T12:46:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: 5.1e scrub starts 2024-03-24T12:46:13.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:13 smithi190 ceph-mon[33748]: 5.1e scrub ok 2024-03-24T12:46:13.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:13.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:13.911 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 18 16 31837 31821 110.467 49.625 0.00519449 0.00894248 2024-03-24T12:46:13.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:14.029 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:14.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:14.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:14.174 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:14.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:14.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:14.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:14.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:14.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:14.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:14.403 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:14.431 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:14.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:14.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:14.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:14.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:14.573 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:14.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:14.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:14 smithi029 ceph-mon[29919]: 1.8 scrub starts 2024-03-24T12:46:14.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:14 smithi029 ceph-mon[29919]: 1.8 scrub ok 2024-03-24T12:46:14.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:14 smithi029 ceph-mon[29919]: osdmap e391: 8 total, 7 up, 6 in 2024-03-24T12:46:14.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:14 smithi029 ceph-mon[29919]: 3.1 scrub starts 2024-03-24T12:46:14.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:14 smithi029 ceph-mon[29919]: 3.1 scrub ok 2024-03-24T12:46:14.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:14 smithi029 ceph-mon[41500]: 1.8 scrub starts 2024-03-24T12:46:14.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:14 smithi029 ceph-mon[41500]: 1.8 scrub ok 2024-03-24T12:46:14.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:14 smithi029 ceph-mon[41500]: osdmap e391: 8 total, 7 up, 6 in 2024-03-24T12:46:14.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:14 smithi029 ceph-mon[41500]: 3.1 scrub starts 2024-03-24T12:46:14.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:14 smithi029 ceph-mon[41500]: 3.1 scrub ok 2024-03-24T12:46:14.675 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:14.714 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:14.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:14.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:14.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:14.839 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:14.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:14.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:14 smithi190 ceph-mon[33748]: 1.8 scrub starts 2024-03-24T12:46:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:14 smithi190 ceph-mon[33748]: 1.8 scrub ok 2024-03-24T12:46:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:14 smithi190 ceph-mon[33748]: osdmap e391: 8 total, 7 up, 6 in 2024-03-24T12:46:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:14 smithi190 ceph-mon[33748]: 3.1 scrub starts 2024-03-24T12:46:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:14 smithi190 ceph-mon[33748]: 3.1 scrub ok 2024-03-24T12:46:14.911 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 19 16 32653 32637 107.337 51 0.00363921 0.00925639 2024-03-24T12:46:14.941 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:14.970 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:14.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:15.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:15.102 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:15.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:15.203 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:15.231 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:15.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:15.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:15.372 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:15.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:15.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:15.474 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:15.504 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:15.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:15.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:15.644 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:15.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:15.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:15 smithi029 ceph-mon[41500]: 3.1e scrub starts 2024-03-24T12:46:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:15 smithi029 ceph-mon[41500]: 3.1e scrub ok 2024-03-24T12:46:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:15 smithi029 ceph-mon[41500]: pgmap v895: 158 pgs: 158 active+clean; 1.9 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 30 KiB/s rd, 90 MiB/s wr, 1.49k op/s 2024-03-24T12:46:15.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:15 smithi029 ceph-mon[41500]: osdmap e392: 8 total, 7 up, 6 in 2024-03-24T12:46:15.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:15 smithi029 ceph-mon[29919]: 3.1e scrub starts 2024-03-24T12:46:15.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:15 smithi029 ceph-mon[29919]: 3.1e scrub ok 2024-03-24T12:46:15.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:15 smithi029 ceph-mon[29919]: pgmap v895: 158 pgs: 158 active+clean; 1.9 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 30 KiB/s rd, 90 MiB/s wr, 1.49k op/s 2024-03-24T12:46:15.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:15 smithi029 ceph-mon[29919]: osdmap e392: 8 total, 7 up, 6 in 2024-03-24T12:46:15.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:15.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:15.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:15.871 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:15 smithi190 ceph-mon[33748]: 3.1e scrub starts 2024-03-24T12:46:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:15 smithi190 ceph-mon[33748]: 3.1e scrub ok 2024-03-24T12:46:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:15 smithi190 ceph-mon[33748]: pgmap v895: 158 pgs: 158 active+clean; 1.9 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 30 KiB/s rd, 90 MiB/s wr, 1.49k op/s 2024-03-24T12:46:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:15 smithi190 ceph-mon[33748]: osdmap e392: 8 total, 7 up, 6 in 2024-03-24T12:46:15.899 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:15.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:15.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:15.911 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:46:15.917315+0000 min lat: 0.00142775 max lat: 0.314617 avg lat: 0.00942306 2024-03-24T12:46:15.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:46:15.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 20 16 33962 33946 106.061 81.8125 0.0101197 0.00942306 2024-03-24T12:46:16.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:16.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:16.041 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:16.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:16.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:16.166 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:16.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:16.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:16.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:16.296 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:16.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:16.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:16.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:16.435 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:16.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:16.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:16.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:16.560 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:16.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:16.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:16.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:16.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:16 smithi029 ceph-mon[29919]: 5.1d deep-scrub starts 2024-03-24T12:46:16.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:16 smithi029 ceph-mon[29919]: 5.1d deep-scrub ok 2024-03-24T12:46:16.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:16 smithi029 ceph-mon[29919]: 1.2 scrub starts 2024-03-24T12:46:16.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:16 smithi029 ceph-mon[29919]: 1.2 scrub ok 2024-03-24T12:46:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:16 smithi029 ceph-mon[41500]: 5.1d deep-scrub starts 2024-03-24T12:46:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:16 smithi029 ceph-mon[41500]: 5.1d deep-scrub ok 2024-03-24T12:46:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:16 smithi029 ceph-mon[41500]: 1.2 scrub starts 2024-03-24T12:46:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:16 smithi029 ceph-mon[41500]: 1.2 scrub ok 2024-03-24T12:46:16.685 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:16.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:16.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:16.810 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:16.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:16.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:16 smithi190 ceph-mon[33748]: 5.1d deep-scrub starts 2024-03-24T12:46:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:16 smithi190 ceph-mon[33748]: 5.1d deep-scrub ok 2024-03-24T12:46:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:16 smithi190 ceph-mon[33748]: 1.2 scrub starts 2024-03-24T12:46:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:16 smithi190 ceph-mon[33748]: 1.2 scrub ok 2024-03-24T12:46:16.911 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:16.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 21 16 34737 34721 103.317 48.4375 0.00324247 0.00960242 2024-03-24T12:46:16.936 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:16.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:16.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:17.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:17.062 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:17.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:17.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:17.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:17.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:17.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:17.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:17.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:17.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:17.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:17.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:17.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:17.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:17.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:17.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:17.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:17.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:17.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:17.871 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:17.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:17.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:17 smithi190 ceph-mon[33748]: 3.d scrub starts 2024-03-24T12:46:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:17 smithi190 ceph-mon[33748]: 3.d scrub ok 2024-03-24T12:46:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:17 smithi190 ceph-mon[33748]: pgmap v897: 158 pgs: 158 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 46 KiB/s rd, 58 MiB/s wr, 996 op/s 2024-03-24T12:46:17.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 22 16 35449 35433 100.643 44.5 0.00749821 0.0099305 2024-03-24T12:46:17.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:17 smithi029 ceph-mon[41500]: 3.d scrub starts 2024-03-24T12:46:17.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:17 smithi029 ceph-mon[41500]: 3.d scrub ok 2024-03-24T12:46:17.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:17 smithi029 ceph-mon[41500]: pgmap v897: 158 pgs: 158 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 46 KiB/s rd, 58 MiB/s wr, 996 op/s 2024-03-24T12:46:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:17 smithi029 ceph-mon[29919]: 3.d scrub starts 2024-03-24T12:46:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:17 smithi029 ceph-mon[29919]: 3.d scrub ok 2024-03-24T12:46:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:17 smithi029 ceph-mon[29919]: pgmap v897: 158 pgs: 158 active+clean; 1.9 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 46 KiB/s rd, 58 MiB/s wr, 996 op/s 2024-03-24T12:46:17.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:46:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:46:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:46:17.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:17.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:17.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:17.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:18.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:18.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:18.124 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:18.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:18.225 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:18.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:18.249 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:18.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:18.350 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:18.377 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:18.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:18.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:18.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:18.519 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:18.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:18.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:18.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:18.645 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:18.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:18.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:18.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:46:18.352+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:46:18.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[41500]: 5.0 scrub starts 2024-03-24T12:46:18.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[41500]: 5.0 scrub ok 2024-03-24T12:46:18.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:46:18.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:46:18.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[41500]: 5.1b scrub starts 2024-03-24T12:46:18.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[41500]: 5.1b scrub ok 2024-03-24T12:46:18.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:18.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[29919]: 5.0 scrub starts 2024-03-24T12:46:18.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[29919]: 5.0 scrub ok 2024-03-24T12:46:18.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:46:18.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:46:18.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[29919]: 5.1b scrub starts 2024-03-24T12:46:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[29919]: 5.1b scrub ok 2024-03-24T12:46:18.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:18.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:18.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:18.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:18.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:18.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:18.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:46:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:46:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:18 smithi190 ceph-mon[33748]: 5.0 scrub starts 2024-03-24T12:46:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:18 smithi190 ceph-mon[33748]: 5.0 scrub ok 2024-03-24T12:46:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:18 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:46:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:18 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:46:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:18 smithi190 ceph-mon[33748]: 5.1b scrub starts 2024-03-24T12:46:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:18 smithi190 ceph-mon[33748]: 5.1b scrub ok 2024-03-24T12:46:18.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:18.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 23 16 36451 36435 98.9901 62.625 0.00761709 0.0100953 2024-03-24T12:46:18.917 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:18.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:18.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:19.019 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:19.062 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:19.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:19.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:19.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:19.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:19.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:19.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:19.318 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:19.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:19.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:19.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:19.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:19.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:19.448 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:19.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:19.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:19.587 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:19.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:19.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:19.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:19.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:19.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:19.854 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:19.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:19.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:19 smithi190 ceph-mon[33748]: 3.9 scrub starts 2024-03-24T12:46:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:19 smithi190 ceph-mon[33748]: 3.9 scrub ok 2024-03-24T12:46:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:19 smithi190 ceph-mon[33748]: 3.19 deep-scrub starts 2024-03-24T12:46:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:19 smithi190 ceph-mon[33748]: 3.19 deep-scrub ok 2024-03-24T12:46:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:19 smithi190 ceph-mon[33748]: 6.15 scrub starts 2024-03-24T12:46:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:19 smithi190 ceph-mon[33748]: pgmap v898: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 55 KiB/s rd, 69 MiB/s wr, 1.19k op/s 2024-03-24T12:46:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:19 smithi190 ceph-mon[33748]: 6.15 scrub ok 2024-03-24T12:46:19.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:19 smithi190 ceph-mon[33748]: 6.11 scrub starts 2024-03-24T12:46:19.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:19 smithi190 ceph-mon[33748]: 6.11 scrub ok 2024-03-24T12:46:19.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 24 16 37616 37600 97.8992 72.8125 0.0104569 0.0102089 2024-03-24T12:46:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[41500]: 3.9 scrub starts 2024-03-24T12:46:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[41500]: 3.9 scrub ok 2024-03-24T12:46:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[41500]: 3.19 deep-scrub starts 2024-03-24T12:46:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[41500]: 3.19 deep-scrub ok 2024-03-24T12:46:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[41500]: 6.15 scrub starts 2024-03-24T12:46:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[41500]: pgmap v898: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 55 KiB/s rd, 69 MiB/s wr, 1.19k op/s 2024-03-24T12:46:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[41500]: 6.15 scrub ok 2024-03-24T12:46:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[41500]: 6.11 scrub starts 2024-03-24T12:46:19.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[41500]: 6.11 scrub ok 2024-03-24T12:46:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[29919]: 3.9 scrub starts 2024-03-24T12:46:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[29919]: 3.9 scrub ok 2024-03-24T12:46:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[29919]: 3.19 deep-scrub starts 2024-03-24T12:46:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[29919]: 3.19 deep-scrub ok 2024-03-24T12:46:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[29919]: 6.15 scrub starts 2024-03-24T12:46:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[29919]: pgmap v898: 158 pgs: 158 active+clean; 2.1 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 55 KiB/s rd, 69 MiB/s wr, 1.19k op/s 2024-03-24T12:46:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[29919]: 6.15 scrub ok 2024-03-24T12:46:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[29919]: 6.11 scrub starts 2024-03-24T12:46:19.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:19 smithi029 ceph-mon[29919]: 6.11 scrub ok 2024-03-24T12:46:19.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:19.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:20.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:20.131 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:20.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:20.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:20.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:20.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:20.358 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:20.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:20.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:20.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:20.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:20.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:20.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:20.614 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:20.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:20.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:20.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:20.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:20.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:20.772 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:20.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:20.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:20.911 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:20.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 25 16 38693 38677 96.6756 67.3125 0.00732749 0.0103384 2024-03-24T12:46:20.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:20.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:21.014 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:21.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:21.054 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:21.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:21.155 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:21.182 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:21.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:21.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:21.308 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:21.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:21.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:21.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:21.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:21.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:21.574 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:21.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:21.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:21 smithi190 ceph-mon[33748]: 4.2 scrub starts 2024-03-24T12:46:21.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:21 smithi190 ceph-mon[33748]: 4.2 scrub ok 2024-03-24T12:46:21.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:21 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:46:21.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:21 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:46:21.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:21 smithi190 ceph-mon[33748]: osdmap e393: 8 total, 7 up, 6 in 2024-03-24T12:46:21.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:21 smithi029 ceph-mon[41500]: 4.2 scrub starts 2024-03-24T12:46:21.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:21 smithi029 ceph-mon[41500]: 4.2 scrub ok 2024-03-24T12:46:21.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:21 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:46:21.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:21 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:46:21.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:21 smithi029 ceph-mon[41500]: osdmap e393: 8 total, 7 up, 6 in 2024-03-24T12:46:21.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:21 smithi029 ceph-mon[29919]: 4.2 scrub starts 2024-03-24T12:46:21.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:21 smithi029 ceph-mon[29919]: 4.2 scrub ok 2024-03-24T12:46:21.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:21 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:46:21.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:21 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:46:21.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:21 smithi029 ceph-mon[29919]: osdmap e393: 8 total, 7 up, 6 in 2024-03-24T12:46:21.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:21.717 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:21.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:21.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:21.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:21.847 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:21.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:21.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 26 15 40020 40005 96.1493 83 0.00711963 0.0103948 2024-03-24T12:46:21.950 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:21.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:21.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:22.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:22.126 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:22.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:22.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:22.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:22.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:22.253 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:22.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:22.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:22.378 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:22.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:22.391 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:22 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:46:22.392 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:22 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:46:22.480 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:22.509 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:22.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:22.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:22.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:22.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:22 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:46:22.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:22 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:46:22.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:22 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:46:22.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:22 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:46:22.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:22 smithi190 ceph-mon[33748]: pgmap v900: 158 pgs: 158 active+clean; 2.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 31 KiB/s rd, 49 MiB/s wr, 828 op/s 2024-03-24T12:46:22.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:22.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:22.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:22.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:22 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:46:22.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:22 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:46:22.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:22 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:46:22.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:22 smithi029 ceph-mon[41500]: pgmap v900: 158 pgs: 158 active+clean; 2.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 31 KiB/s rd, 49 MiB/s wr, 828 op/s 2024-03-24T12:46:22.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:22 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:46:22.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:22 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:46:22.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:22 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:46:22.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:22 smithi029 ceph-mon[29919]: pgmap v900: 158 pgs: 158 active+clean; 2.2 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 31 KiB/s rd, 49 MiB/s wr, 828 op/s 2024-03-24T12:46:22.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:22.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:22.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:22.891 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:22.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 27 15 41034 41019 94.9354 63.375 0.0057925 0.0105278 2024-03-24T12:46:22.921 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:22.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:22.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:23.023 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:23.046 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:23.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:23.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:23.148 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:23.173 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:23.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:23.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:23.303 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:23.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:23.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:23.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:23.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:23.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:23.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:23.572 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:23.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:23.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:23 smithi190 ceph-mon[33748]: 3.5 scrub starts 2024-03-24T12:46:23.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:23 smithi190 ceph-mon[33748]: 3.5 scrub ok 2024-03-24T12:46:23.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:23 smithi190 ceph-mon[33748]: 5.1 scrub starts 2024-03-24T12:46:23.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:23 smithi190 ceph-mon[33748]: 5.1 scrub ok 2024-03-24T12:46:23.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:23 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:46:23.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:23 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:46:23.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:23 smithi190 ceph-mon[33748]: 3.16 scrub starts 2024-03-24T12:46:23.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:23 smithi190 ceph-mon[33748]: 3.16 scrub ok 2024-03-24T12:46:23.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[29919]: 3.5 scrub starts 2024-03-24T12:46:23.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[29919]: 3.5 scrub ok 2024-03-24T12:46:23.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[29919]: 5.1 scrub starts 2024-03-24T12:46:23.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[29919]: 5.1 scrub ok 2024-03-24T12:46:23.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:46:23.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:46:23.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[29919]: 3.16 scrub starts 2024-03-24T12:46:23.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[29919]: 3.16 scrub ok 2024-03-24T12:46:23.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[41500]: 3.5 scrub starts 2024-03-24T12:46:23.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[41500]: 3.5 scrub ok 2024-03-24T12:46:23.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[41500]: 5.1 scrub starts 2024-03-24T12:46:23.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[41500]: 5.1 scrub ok 2024-03-24T12:46:23.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:46:23.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:46:23.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[41500]: 3.16 scrub starts 2024-03-24T12:46:23.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:23 smithi029 ceph-mon[41500]: 3.16 scrub ok 2024-03-24T12:46:23.674 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:23.697 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:23.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:23.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:23.841 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:23.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:23.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 28 16 41969 41953 93.6296 58.375 0.00631184 0.0106736 2024-03-24T12:46:23.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:23.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:23.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:24.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:24.108 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:24.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:24.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:24.210 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:24.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:24.235 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:24.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:24.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:24.368 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:24.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:24.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:24.509 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:24.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:24.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:24.611 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:24.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:24 smithi190 ceph-mon[33748]: 3.2 scrub starts 2024-03-24T12:46:24.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:24 smithi190 ceph-mon[33748]: 3.2 scrub ok 2024-03-24T12:46:24.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:24 smithi190 ceph-mon[33748]: pgmap v901: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 19 KiB/s rd, 57 MiB/s wr, 943 op/s 2024-03-24T12:46:24.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:24.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:24.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:24.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:24 smithi029 ceph-mon[29919]: 3.2 scrub starts 2024-03-24T12:46:24.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:24 smithi029 ceph-mon[29919]: 3.2 scrub ok 2024-03-24T12:46:24.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:24 smithi029 ceph-mon[29919]: pgmap v901: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 19 KiB/s rd, 57 MiB/s wr, 943 op/s 2024-03-24T12:46:24.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:24 smithi029 ceph-mon[41500]: 3.2 scrub starts 2024-03-24T12:46:24.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:24 smithi029 ceph-mon[41500]: 3.2 scrub ok 2024-03-24T12:46:24.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:24 smithi029 ceph-mon[41500]: pgmap v901: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 19 KiB/s rd, 57 MiB/s wr, 943 op/s 2024-03-24T12:46:24.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:24.761 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:24.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:24.762 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:24.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:24.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:24.888 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:24.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:24.912 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 29 16 42465 42449 91.47 31 0.0156994 0.0109265 2024-03-24T12:46:24.988 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:25.014 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:25.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:25.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:25.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:25.140 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:25.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:25.241 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:25.265 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:25.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:25.366 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:25.391 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:25.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:25.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:25.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:25.536 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:25.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:25.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:25 smithi190 ceph-mon[33748]: 4.d scrub starts 2024-03-24T12:46:25.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:25 smithi190 ceph-mon[33748]: 4.d scrub ok 2024-03-24T12:46:25.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:25 smithi190 ceph-mon[33748]: osdmap e394: 8 total, 7 up, 6 in 2024-03-24T12:46:25.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:25.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:25 smithi029 ceph-mon[29919]: 4.d scrub starts 2024-03-24T12:46:25.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:25 smithi029 ceph-mon[29919]: 4.d scrub ok 2024-03-24T12:46:25.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:25 smithi029 ceph-mon[29919]: osdmap e394: 8 total, 7 up, 6 in 2024-03-24T12:46:25.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:25 smithi029 ceph-mon[41500]: 4.d scrub starts 2024-03-24T12:46:25.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:25 smithi029 ceph-mon[41500]: 4.d scrub ok 2024-03-24T12:46:25.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:25 smithi029 ceph-mon[41500]: osdmap e394: 8 total, 7 up, 6 in 2024-03-24T12:46:25.676 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:25.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:25.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:25.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:25.819 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:25.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:25.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:25.913 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 30 16 43322 43306 90.2063 53.5625 0.0155708 0.01108 2024-03-24T12:46:25.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:25.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:46:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:25.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:26.066 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:26.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:26.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:26.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:26.241 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:26.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:26.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:26.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:26.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:26 smithi029 ceph-mon[29919]: pgmap v902: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 8.0 KiB/s rd, 61 MiB/s wr, 981 op/s 2024-03-24T12:46:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:26 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:46:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:26 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:46:26.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:26 smithi029 ceph-mon[41500]: pgmap v902: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 8.0 KiB/s rd, 61 MiB/s wr, 981 op/s 2024-03-24T12:46:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:26 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:46:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:26 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:46:26.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:26.512 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:26.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:26.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:26.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:26.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:26 smithi190 ceph-mon[33748]: pgmap v902: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 8.0 KiB/s rd, 61 MiB/s wr, 981 op/s 2024-03-24T12:46:26.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:26 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:46:26.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:26 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:46:26.656 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:26.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:26.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:26.783 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:26.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:26.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:26.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:26.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:26.914 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 31 16 44367 44351 89.4032 65.3125 0.00599811 0.0111699 2024-03-24T12:46:27.015 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:27.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:27.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:27.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:27.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:27.187 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:27.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:27.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:27.289 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:27.312 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:27.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:27.313 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:27.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:27.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:27.459 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:27.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:27.560 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:27.591 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:27.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:27.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:27 smithi029 ceph-mon[41500]: 3.11 scrub starts 2024-03-24T12:46:27.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:27 smithi029 ceph-mon[41500]: 3.11 scrub ok 2024-03-24T12:46:27.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:27 smithi029 ceph-mon[41500]: pgmap v904: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s 2024-03-24T12:46:27.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:27 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:46:27.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:27 smithi029 ceph-mon[29919]: 3.11 scrub starts 2024-03-24T12:46:27.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:27 smithi029 ceph-mon[29919]: 3.11 scrub ok 2024-03-24T12:46:27.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:27 smithi029 ceph-mon[29919]: pgmap v904: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s 2024-03-24T12:46:27.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:27 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:46:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:27 smithi190 ceph-mon[33748]: 3.11 scrub starts 2024-03-24T12:46:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:27 smithi190 ceph-mon[33748]: 3.11 scrub ok 2024-03-24T12:46:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:27 smithi190 ceph-mon[33748]: pgmap v904: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 2.5 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 832 op/s 2024-03-24T12:46:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:27 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:46:27.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:27.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:27.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:27.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:27.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:27.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:27.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:27.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:27.913 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 32 16 45066 45050 87.9746 43.6875 0.015418 0.0113611 2024-03-24T12:46:27.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:46:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:46:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:46:27.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:27.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:27.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:28.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:28.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:28.122 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:28.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:28.223 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:28.247 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:28.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:28.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:28.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:28.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:28.530 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:28.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:28 smithi190 ceph-mon[33748]: 6.1e scrub starts 2024-03-24T12:46:28.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:28 smithi190 ceph-mon[33748]: 6.1e scrub ok 2024-03-24T12:46:28.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:28 smithi190 ceph-mon[33748]: 5.13 deep-scrub starts 2024-03-24T12:46:28.530 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:28 smithi190 ceph-mon[33748]: 5.13 deep-scrub ok 2024-03-24T12:46:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:28.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:28.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:28.656 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:28.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:28.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:28.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:28 smithi029 ceph-mon[29919]: 6.1e scrub starts 2024-03-24T12:46:28.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:28 smithi029 ceph-mon[29919]: 6.1e scrub ok 2024-03-24T12:46:28.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:28 smithi029 ceph-mon[29919]: 5.13 deep-scrub starts 2024-03-24T12:46:28.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:28 smithi029 ceph-mon[29919]: 5.13 deep-scrub ok 2024-03-24T12:46:28.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:28 smithi029 ceph-mon[41500]: 6.1e scrub starts 2024-03-24T12:46:28.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:28 smithi029 ceph-mon[41500]: 6.1e scrub ok 2024-03-24T12:46:28.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:28 smithi029 ceph-mon[41500]: 5.13 deep-scrub starts 2024-03-24T12:46:28.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:28 smithi029 ceph-mon[41500]: 5.13 deep-scrub ok 2024-03-24T12:46:28.758 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:28.795 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:28.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:28.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:46:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:46:28.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:28.913 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 33 16 45947 45931 86.9771 55.0625 0.00494882 0.0114845 2024-03-24T12:46:28.921 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:28.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:28.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:29.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:29.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:29.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:29.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:29.176 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:29.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:29.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:29.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:29.301 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:29.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:29.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:29.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:29.427 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:29.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:29.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:29.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:29.556 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:29.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:29.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:29.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:29 smithi190 ceph-mon[33748]: 6.d scrub starts 2024-03-24T12:46:29.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:29 smithi190 ceph-mon[33748]: 6.d scrub ok 2024-03-24T12:46:29.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:29 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:29.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:29 smithi190 ceph-mon[33748]: pgmap v905: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T12:46:29.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:29 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:46:29.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:29 smithi190 ceph-mon[33748]: 4.11 scrub ok 2024-03-24T12:46:29.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[29919]: 6.d scrub starts 2024-03-24T12:46:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[29919]: 6.d scrub ok 2024-03-24T12:46:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[29919]: pgmap v905: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T12:46:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:46:29.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[29919]: 4.11 scrub ok 2024-03-24T12:46:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[41500]: 6.d scrub starts 2024-03-24T12:46:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[41500]: 6.d scrub ok 2024-03-24T12:46:29.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:29.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[41500]: pgmap v905: 158 pgs: 158 active+clean; 2.7 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 63 MiB/s wr, 1.01k op/s 2024-03-24T12:46:29.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:46:29.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:29 smithi029 ceph-mon[41500]: 4.11 scrub ok 2024-03-24T12:46:29.698 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:29.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:29.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:29.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:29.839 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:29.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:29.913 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 34 15 46626 46611 85.6689 42.5 0.00732106 0.0116665 2024-03-24T12:46:29.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:29.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:29.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:30.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:30.125 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:30.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:30.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:30.226 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:30.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:30.254 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:30.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:30.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:30.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:30.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:30.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:30.485 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:30.513 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:30.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:30.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:30.615 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:30 smithi190 ceph-mon[33748]: 4.7 scrub starts 2024-03-24T12:46:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:30 smithi190 ceph-mon[33748]: 4.7 scrub ok 2024-03-24T12:46:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:30 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:46:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:30 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:46:30.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:30.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:30.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:30.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:30 smithi029 ceph-mon[29919]: 4.7 scrub starts 2024-03-24T12:46:30.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:30 smithi029 ceph-mon[29919]: 4.7 scrub ok 2024-03-24T12:46:30.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:30 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:46:30.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:30 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:46:30.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:30 smithi029 ceph-mon[41500]: 4.7 scrub starts 2024-03-24T12:46:30.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:30 smithi029 ceph-mon[41500]: 4.7 scrub ok 2024-03-24T12:46:30.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:30 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:46:30.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:30 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:46:30.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:30.774 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:30.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:30.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:30.876 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:30.913 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 35 16 47927 47911 85.5417 81.25 0.0114068 0.0116837 2024-03-24T12:46:30.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:30.919 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:30.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:31.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:31.070 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:31.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:31.171 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:31.196 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:31.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:31.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:31.241 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:46:31.298 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:31.325 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:31.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:31.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[29919]: 4.1a scrub starts 2024-03-24T12:46:31.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[29919]: 4.1a scrub ok 2024-03-24T12:46:31.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[29919]: osdmap e395: 8 total, 7 up, 6 in 2024-03-24T12:46:31.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:46:31.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:46:31.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[29919]: pgmap v907: 158 pgs: 158 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 58 MiB/s wr, 924 op/s 2024-03-24T12:46:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[41500]: 4.1a scrub starts 2024-03-24T12:46:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[41500]: 4.1a scrub ok 2024-03-24T12:46:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[41500]: osdmap e395: 8 total, 7 up, 6 in 2024-03-24T12:46:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:46:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:46:31.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:31 smithi029 ceph-mon[41500]: pgmap v907: 158 pgs: 158 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 58 MiB/s wr, 924 op/s 2024-03-24T12:46:31.427 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:31.453 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:31.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:31.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:31.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:31.611 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:31.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:31.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:31 smithi190 ceph-mon[33748]: 4.1a scrub starts 2024-03-24T12:46:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:31 smithi190 ceph-mon[33748]: 4.1a scrub ok 2024-03-24T12:46:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:31 smithi190 ceph-mon[33748]: osdmap e395: 8 total, 7 up, 6 in 2024-03-24T12:46:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:31 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:46:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:31 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:46:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:31 smithi190 ceph-mon[33748]: pgmap v907: 158 pgs: 158 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 639 B/s rd, 58 MiB/s wr, 924 op/s 2024-03-24T12:46:31.713 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:31.740 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:31.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:31.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:31.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:31.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:31.913 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 36 16 49175 49159 85.332 78 0.0101135 0.011713 2024-03-24T12:46:31.960 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:46:31.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:32.038 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:32.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:32.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:32.182 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:32.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:32.284 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:32.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:32.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:32.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:32.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:32.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:32.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:32.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:32.538 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:32 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:46:32.538 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:32 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:46:32.538 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:32 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:46:32.539 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:32 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:46:32.539 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:32 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:46:32.539 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:32 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:46:32.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:32.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:32.614 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:32.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:32 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:46:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:32 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:46:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:32 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:46:32.714 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:32.741 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:32.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:32.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:32.873 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:32.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:32.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:32.913 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 37 16 50512 50496 85.284 83.5625 0.00943195 0.0117196 2024-03-24T12:46:32.975 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:33.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:33.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:33.122 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:33.149 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:33.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:33.250 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:33.276 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:33.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:33.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:33.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:33.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:33.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:33.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-mon[41500]: 5.12 scrub starts 2024-03-24T12:46:33.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-mon[41500]: 5.12 scrub ok 2024-03-24T12:46:33.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-mon[41500]: 3.14 deep-scrub starts 2024-03-24T12:46:33.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-mon[41500]: 3.14 deep-scrub ok 2024-03-24T12:46:33.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-mon[41500]: pgmap v908: 158 pgs: 158 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 851 op/s 2024-03-24T12:46:33.501 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-mon[29919]: 5.12 scrub starts 2024-03-24T12:46:33.501 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-mon[29919]: 5.12 scrub ok 2024-03-24T12:46:33.501 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-mon[29919]: 3.14 deep-scrub starts 2024-03-24T12:46:33.501 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-mon[29919]: 3.14 deep-scrub ok 2024-03-24T12:46:33.501 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-mon[29919]: pgmap v908: 158 pgs: 158 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 851 op/s 2024-03-24T12:46:33.501 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:46:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:46:33.375+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:46:33.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:33.558 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:33.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:33.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:33 smithi190 ceph-mon[33748]: 5.12 scrub starts 2024-03-24T12:46:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:33 smithi190 ceph-mon[33748]: 5.12 scrub ok 2024-03-24T12:46:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:33 smithi190 ceph-mon[33748]: 3.14 deep-scrub starts 2024-03-24T12:46:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:33 smithi190 ceph-mon[33748]: 3.14 deep-scrub ok 2024-03-24T12:46:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:33 smithi190 ceph-mon[33748]: pgmap v908: 158 pgs: 158 active+clean; 2.9 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 851 op/s 2024-03-24T12:46:33.660 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:33.690 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:46:33.690 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:46:33.690 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:46:33.690 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:46:33.690 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:46:33.690 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:46:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:33.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:33.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:33.718 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:46:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:46:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:46:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:46:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:46:33.718 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:46:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:46:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:46:33.719 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:46:33.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:33.858 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:33.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:33.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:33.913 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 38 15 51659 51644 84.9277 71.75 0.0151766 0.0117694 2024-03-24T12:46:33.960 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:33.987 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:33.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:34.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:34.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:34.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:34.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:34.278 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:34.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:34.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:46:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:46:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2810665011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:46:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:46:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:46:34.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[41500]: 4.1 scrub starts 2024-03-24T12:46:34.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[41500]: 4.1 scrub ok 2024-03-24T12:46:34.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:46:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:46:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2810665011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:46:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:46:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:46:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[29919]: 4.1 scrub starts 2024-03-24T12:46:34.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:34 smithi029 ceph-mon[29919]: 4.1 scrub ok 2024-03-24T12:46:34.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:34.421 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:34.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:34.508 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:46:34.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:46:34.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:34.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:34 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:46:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:34 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:46:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:34 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2810665011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:46:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:34 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:46:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:34 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:46:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:34 smithi190 ceph-mon[33748]: 4.1 scrub starts 2024-03-24T12:46:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:34 smithi190 ceph-mon[33748]: 4.1 scrub ok 2024-03-24T12:46:34.667 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:34.695 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:34.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:34.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:34.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:34.843 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:34.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:34.914 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 39 16 52932 52916 84.7884 79.5 0.0254351 0.0117883 2024-03-24T12:46:34.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:34.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:34.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:35.089 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:35.117 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:35.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:35.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:35.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:35.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:35.276 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:35.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:35.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[29919]: 6.19 scrub starts 2024-03-24T12:46:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[29919]: 6.19 scrub ok 2024-03-24T12:46:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[29919]: pgmap v909: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 930 op/s 2024-03-24T12:46:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[29919]: 1.5 scrub starts 2024-03-24T12:46:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[29919]: 1.5 scrub ok 2024-03-24T12:46:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[29919]: osdmap e396: 8 total, 7 up, 6 in 2024-03-24T12:46:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[41500]: 6.19 scrub starts 2024-03-24T12:46:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[41500]: 6.19 scrub ok 2024-03-24T12:46:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[41500]: pgmap v909: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 930 op/s 2024-03-24T12:46:35.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[41500]: 1.5 scrub starts 2024-03-24T12:46:35.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[41500]: 1.5 scrub ok 2024-03-24T12:46:35.420 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:35 smithi029 ceph-mon[41500]: osdmap e396: 8 total, 7 up, 6 in 2024-03-24T12:46:35.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:35.421 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:35.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:35.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:35.566 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:35.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:35 smithi190 ceph-mon[33748]: 6.19 scrub starts 2024-03-24T12:46:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:35 smithi190 ceph-mon[33748]: 6.19 scrub ok 2024-03-24T12:46:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:35 smithi190 ceph-mon[33748]: pgmap v909: 158 pgs: 158 active+clean; 3.1 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.0 KiB/s rd, 58 MiB/s wr, 930 op/s 2024-03-24T12:46:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:35 smithi190 ceph-mon[33748]: 1.5 scrub starts 2024-03-24T12:46:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:35 smithi190 ceph-mon[33748]: 1.5 scrub ok 2024-03-24T12:46:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:35 smithi190 ceph-mon[33748]: osdmap e396: 8 total, 7 up, 6 in 2024-03-24T12:46:35.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:35.709 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:35.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:35.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:35.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:35.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:35.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:35.875 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:46:35.888 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:46:35.914 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:46:35.920384+0000 min lat: 0.00142775 max lat: 0.499653 avg lat: 0.011835 2024-03-24T12:46:35.914 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:46:35.915 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 40 16 54075 54059 84.4525 71.4375 0.0121037 0.011835 2024-03-24T12:46:35.939 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:35.988 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:35.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:35.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:36.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:36.133 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:36.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:36.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:36.262 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:36.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:36.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:36.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:36.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:36.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:36.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:36.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:46:36.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:36.538 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:36.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:36.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:36 smithi029 ceph-mon[29919]: 6.9 scrub starts 2024-03-24T12:46:36.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:36 smithi029 ceph-mon[29919]: 6.9 scrub ok 2024-03-24T12:46:36.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:36 smithi029 ceph-mon[29919]: 6.10 scrub starts 2024-03-24T12:46:36.577 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:36 smithi029 ceph-mon[29919]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:46:36.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:36 smithi029 ceph-mon[41500]: 6.9 scrub starts 2024-03-24T12:46:36.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:36 smithi029 ceph-mon[41500]: 6.9 scrub ok 2024-03-24T12:46:36.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:36 smithi029 ceph-mon[41500]: 6.10 scrub starts 2024-03-24T12:46:36.577 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:36 smithi029 ceph-mon[41500]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:46:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:36 smithi190 ceph-mon[33748]: 6.9 scrub starts 2024-03-24T12:46:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:36 smithi190 ceph-mon[33748]: 6.9 scrub ok 2024-03-24T12:46:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:36 smithi190 ceph-mon[33748]: 6.10 scrub starts 2024-03-24T12:46:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:36 smithi190 ceph-mon[33748]: from='client.15474 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:46:36.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:36.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:36.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:36.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:36.769 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:36.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:36.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:36.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 41 15 55326 55311 84.2995 78.25 0.00940273 0.0118569 2024-03-24T12:46:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:36.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:37.036 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":910,"stamp":"2024-03-24T12:46:35.237786+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3347454302,"num_objects":51267,"num_object_clones":0,"num_object_copies":153801,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51267,"num_whiteouts":0,"num_read":2784,"num_read_kb":2536,"num_write":52077,"num_write_kb":3268461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":795391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":52731,"ondisk_log_size":52731,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":14355144,"kb_used_data":9811604,"kb_used_omap":9,"kb_used_meta":4543478,"kb_avail":548115768,"statfs":{"total":575970213888,"available":561270546432,"internally_reserved":0,"allocated":10047082496,"data_stored":10040427368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9563,"internal_metadata":4652522149},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":637468672,"num_objects":9727,"num_object_clones":0,"num_object_copies":29181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9727,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9727,"num_write_kb":622528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.375153"},"pg_stats":[{"pgid":"11.3","version":"395'3079","reported_seq":3115,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.742126+0000","last_change":"2024-03-24T12:45:43.234940+0000","last_active":"2024-03-24T12:46:32.742126+0000","last_peered":"2024-03-24T12:46:32.742126+0000","last_clean":"2024-03-24T12:46:32.742126+0000","last_became_active":"2024-03-24T12:45:43.234520+0000","last_became_peered":"2024-03-24T12:45:43.234520+0000","last_unstale":"2024-03-24T12:46:32.742126+0000","last_undegraded":"2024-03-24T12:46:32.742126+0000","last_fullsized":"2024-03-24T12:46:32.742126+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3079,"log_dups_size":0,"ondisk_log_size":3079,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:51.076919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201785344,"num_objects":3079,"num_object_clones":0,"num_object_copies":9237,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3079,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3079,"num_write_kb":197056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"390'30","reported_seq":843,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274825+0000","last_change":"2024-03-24T12:46:09.601895+0000","last_active":"2024-03-24T12:46:30.274825+0000","last_peered":"2024-03-24T12:46:30.274825+0000","last_clean":"2024-03-24T12:46:30.274825+0000","last_became_active":"2024-03-24T12:45:13.265666+0000","last_became_peered":"2024-03-24T12:45:13.265666+0000","last_unstale":"2024-03-24T12:46:30.274825+0000","last_undegraded":"2024-03-24T12:46:30.274825+0000","last_fullsized":"2024-03-24T12:46:30.274825+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:46:09.601792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:09.601792+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:21.073551+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":51,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279408+0000","last_change":"2024-03-24T12:45:57.127400+0000","last_active":"2024-03-24T12:46:30.279408+0000","last_peered":"2024-03-24T12:46:30.279408+0000","last_clean":"2024-03-24T12:46:30.279408+0000","last_became_active":"2024-03-24T12:45:40.250678+0000","last_became_peered":"2024-03-24T12:45:40.250678+0000","last_unstale":"2024-03-24T12:46:30.279408+0000","last_undegraded":"2024-03-24T12:46:30.279408+0000","last_fullsized":"2024-03-24T12:46:30.279408+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:57.127283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:57.127283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:04.494714+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":978,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.270012+0000","last_change":"2024-03-24T12:45:06.867846+0000","last_active":"2024-03-24T12:46:30.270012+0000","last_peered":"2024-03-24T12:46:30.270012+0000","last_clean":"2024-03-24T12:46:30.270012+0000","last_became_active":"2024-03-24T12:44:48.938189+0000","last_became_peered":"2024-03-24T12:44:48.938189+0000","last_unstale":"2024-03-24T12:46:30.270012+0000","last_undegraded":"2024-03-24T12:46:30.270012+0000","last_fullsized":"2024-03-24T12:46:30.270012+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:45:06.867746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:06.867746+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":736,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275997+0000","last_change":"2024-03-24T12:46:01.482058+0000","last_active":"2024-03-24T12:46:30.275997+0000","last_peered":"2024-03-24T12:46:30.275997+0000","last_clean":"2024-03-24T12:46:30.275997+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:46:30.275997+0000","last_undegraded":"2024-03-24T12:46:30.275997+0000","last_fullsized":"2024-03-24T12:46:30.275997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:01.481956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:42:45.013715+0000","last_clean_scrub_stamp":"2024-03-24T12:46:01.481956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.433495+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1049,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268209+0000","last_change":"2024-03-24T12:45:52.951527+0000","last_active":"2024-03-24T12:46:30.268209+0000","last_peered":"2024-03-24T12:46:30.268209+0000","last_clean":"2024-03-24T12:46:30.268209+0000","last_became_active":"2024-03-24T12:41:52.953388+0000","last_became_peered":"2024-03-24T12:41:52.953388+0000","last_unstale":"2024-03-24T12:46:30.268209+0000","last_undegraded":"2024-03-24T12:46:30.268209+0000","last_fullsized":"2024-03-24T12:46:30.268209+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:45:52.951428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:45:52.951428+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:22.598588+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"395'3332","reported_seq":3368,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:34.751637+0000","last_change":"2024-03-24T12:45:43.229931+0000","last_active":"2024-03-24T12:46:34.751637+0000","last_peered":"2024-03-24T12:46:34.751637+0000","last_clean":"2024-03-24T12:46:34.751637+0000","last_became_active":"2024-03-24T12:45:43.229611+0000","last_became_peered":"2024-03-24T12:45:43.229611+0000","last_unstale":"2024-03-24T12:46:34.751637+0000","last_undegraded":"2024-03-24T12:46:34.751637+0000","last_fullsized":"2024-03-24T12:46:34.751637+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3332,"log_dups_size":0,"ondisk_log_size":3332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:52.917940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218365952,"num_objects":3332,"num_object_clones":0,"num_object_copies":9996,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3332,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3332,"num_write_kb":213248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"390'53","reported_seq":822,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275215+0000","last_change":"2024-03-24T12:46:23.746624+0000","last_active":"2024-03-24T12:46:30.275215+0000","last_peered":"2024-03-24T12:46:30.275215+0000","last_clean":"2024-03-24T12:46:30.275215+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:46:30.275215+0000","last_undegraded":"2024-03-24T12:46:30.275215+0000","last_fullsized":"2024-03-24T12:46:30.275215+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'53","last_scrub_stamp":"2024-03-24T12:46:23.746506+0000","last_deep_scrub":"207'39","last_deep_scrub_stamp":"2024-03-24T12:36:22.669686+0000","last_clean_scrub_stamp":"2024-03-24T12:46:23.746506+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:41.502009+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":113,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274591+0000","last_change":"2024-03-24T12:45:13.258278+0000","last_active":"2024-03-24T12:46:30.274591+0000","last_peered":"2024-03-24T12:46:30.274591+0000","last_clean":"2024-03-24T12:46:30.274591+0000","last_became_active":"2024-03-24T12:45:13.257987+0000","last_became_peered":"2024-03-24T12:45:13.257987+0000","last_unstale":"2024-03-24T12:46:30.274591+0000","last_undegraded":"2024-03-24T12:46:30.274591+0000","last_fullsized":"2024-03-24T12:46:30.274591+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:38.805830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:44:38.805830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":333,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274796+0000","last_change":"2024-03-24T12:45:12.448648+0000","last_active":"2024-03-24T12:46:30.274796+0000","last_peered":"2024-03-24T12:46:30.274796+0000","last_clean":"2024-03-24T12:46:30.274796+0000","last_became_active":"2024-03-24T12:39:07.753551+0000","last_became_peered":"2024-03-24T12:39:07.753551+0000","last_unstale":"2024-03-24T12:46:30.274796+0000","last_undegraded":"2024-03-24T12:46:30.274796+0000","last_fullsized":"2024-03-24T12:46:30.274796+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:12.448516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:12.448516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:38.669833+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":750,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269477+0000","last_change":"2024-03-24T12:46:20.834095+0000","last_active":"2024-03-24T12:46:30.269477+0000","last_peered":"2024-03-24T12:46:30.269477+0000","last_clean":"2024-03-24T12:46:30.269477+0000","last_became_active":"2024-03-24T12:45:40.248552+0000","last_became_peered":"2024-03-24T12:45:40.248552+0000","last_unstale":"2024-03-24T12:46:30.269477+0000","last_undegraded":"2024-03-24T12:46:30.269477+0000","last_fullsized":"2024-03-24T12:46:30.269477+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:20.833956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:53.627010+0000","last_clean_scrub_stamp":"2024-03-24T12:46:20.833956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:46.599490+0000","scrub_duration":265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":564,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.265879+0000","last_change":"2024-03-24T12:46:12.525677+0000","last_active":"2024-03-24T12:46:30.265879+0000","last_peered":"2024-03-24T12:46:30.265879+0000","last_clean":"2024-03-24T12:46:30.265879+0000","last_became_active":"2024-03-24T12:45:43.243436+0000","last_became_peered":"2024-03-24T12:45:43.243436+0000","last_unstale":"2024-03-24T12:46:30.265879+0000","last_undegraded":"2024-03-24T12:46:30.265879+0000","last_fullsized":"2024-03-24T12:46:30.265879+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:46:12.525399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:46:12.525399+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:23.447639+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"395'3068","reported_seq":3104,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.925614+0000","last_change":"2024-03-24T12:45:43.236083+0000","last_active":"2024-03-24T12:46:32.925614+0000","last_peered":"2024-03-24T12:46:32.925614+0000","last_clean":"2024-03-24T12:46:32.925614+0000","last_became_active":"2024-03-24T12:45:43.235777+0000","last_became_peered":"2024-03-24T12:45:43.235777+0000","last_unstale":"2024-03-24T12:46:32.925614+0000","last_undegraded":"2024-03-24T12:46:32.925614+0000","last_fullsized":"2024-03-24T12:46:32.925614+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3068,"log_dups_size":0,"ondisk_log_size":3068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:43.844171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201064448,"num_objects":3068,"num_object_clones":0,"num_object_copies":9204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3068,"num_write_kb":196352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"390'27","reported_seq":851,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275375+0000","last_change":"2024-03-24T12:45:40.289168+0000","last_active":"2024-03-24T12:46:30.275375+0000","last_peered":"2024-03-24T12:46:30.275375+0000","last_clean":"2024-03-24T12:46:30.275375+0000","last_became_active":"2024-03-24T12:45:40.249367+0000","last_became_peered":"2024-03-24T12:45:40.249367+0000","last_unstale":"2024-03-24T12:46:30.275375+0000","last_undegraded":"2024-03-24T12:46:30.275375+0000","last_fullsized":"2024-03-24T12:46:30.275375+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:44:41.541329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:44:41.541329+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":740,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274224+0000","last_change":"2024-03-24T12:46:18.685653+0000","last_active":"2024-03-24T12:46:30.274224+0000","last_peered":"2024-03-24T12:46:30.274224+0000","last_clean":"2024-03-24T12:46:30.274224+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:46:30.274224+0000","last_undegraded":"2024-03-24T12:46:30.274224+0000","last_fullsized":"2024-03-24T12:46:30.274224+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:18.685541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:18.685541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:46.514317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":440,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275988+0000","last_change":"2024-03-24T12:45:13.311240+0000","last_active":"2024-03-24T12:46:30.275988+0000","last_peered":"2024-03-24T12:46:30.275988+0000","last_clean":"2024-03-24T12:46:30.275988+0000","last_became_active":"2024-03-24T12:45:13.310907+0000","last_became_peered":"2024-03-24T12:45:13.310907+0000","last_unstale":"2024-03-24T12:46:30.275988+0000","last_undegraded":"2024-03-24T12:46:30.275988+0000","last_fullsized":"2024-03-24T12:46:30.275988+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:36.501921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:36.501921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":711,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268583+0000","last_change":"2024-03-24T12:46:07.891080+0000","last_active":"2024-03-24T12:46:30.268583+0000","last_peered":"2024-03-24T12:46:30.268583+0000","last_clean":"2024-03-24T12:46:30.268583+0000","last_became_active":"2024-03-24T12:45:40.244433+0000","last_became_peered":"2024-03-24T12:45:40.244433+0000","last_unstale":"2024-03-24T12:46:30.268583+0000","last_undegraded":"2024-03-24T12:46:30.268583+0000","last_fullsized":"2024-03-24T12:46:30.268583+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:07.890930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:07.890930+0000","last_clean_scrub_stamp":"2024-03-24T12:46:07.890930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.515746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"395'3228","reported_seq":3264,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.828100+0000","last_change":"2024-03-24T12:45:43.242575+0000","last_active":"2024-03-24T12:46:32.828100+0000","last_peered":"2024-03-24T12:46:32.828100+0000","last_clean":"2024-03-24T12:46:32.828100+0000","last_became_active":"2024-03-24T12:45:43.242256+0000","last_became_peered":"2024-03-24T12:45:43.242256+0000","last_unstale":"2024-03-24T12:46:32.828100+0000","last_undegraded":"2024-03-24T12:46:32.828100+0000","last_fullsized":"2024-03-24T12:46:32.828100+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3228,"log_dups_size":0,"ondisk_log_size":3228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:00.929569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211550208,"num_objects":3228,"num_object_clones":0,"num_object_copies":9684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3228,"num_write_kb":206592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"391'46","reported_seq":794,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269793+0000","last_change":"2024-03-24T12:46:09.424418+0000","last_active":"2024-03-24T12:46:30.269793+0000","last_peered":"2024-03-24T12:46:30.269793+0000","last_clean":"2024-03-24T12:46:30.269793+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:46:30.269793+0000","last_undegraded":"2024-03-24T12:46:30.269793+0000","last_fullsized":"2024-03-24T12:46:30.269793+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-24T12:46:09.424281+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:46:09.424281+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:32.753447+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":51,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.277871+0000","last_change":"2024-03-24T12:45:58.406476+0000","last_active":"2024-03-24T12:46:30.277871+0000","last_peered":"2024-03-24T12:46:30.277871+0000","last_clean":"2024-03-24T12:46:30.277871+0000","last_became_active":"2024-03-24T12:45:40.244230+0000","last_became_peered":"2024-03-24T12:45:40.244230+0000","last_unstale":"2024-03-24T12:46:30.277871+0000","last_undegraded":"2024-03-24T12:46:30.277871+0000","last_fullsized":"2024-03-24T12:46:30.277871+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:58.406348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:58.406348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:14.203731+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'11","reported_seq":978,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275908+0000","last_change":"2024-03-24T12:44:56.632534+0000","last_active":"2024-03-24T12:46:30.275908+0000","last_peered":"2024-03-24T12:46:30.275908+0000","last_clean":"2024-03-24T12:46:30.275908+0000","last_became_active":"2024-03-24T12:44:48.870472+0000","last_became_peered":"2024-03-24T12:44:48.870472+0000","last_unstale":"2024-03-24T12:46:30.275908+0000","last_undegraded":"2024-03-24T12:46:30.275908+0000","last_fullsized":"2024-03-24T12:46:30.275908+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'11","last_scrub_stamp":"2024-03-24T12:44:56.632417+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:44:56.632417+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":415,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.281550+0000","last_change":"2024-03-24T12:46:28.074580+0000","last_active":"2024-03-24T12:46:30.281550+0000","last_peered":"2024-03-24T12:46:30.281550+0000","last_clean":"2024-03-24T12:46:30.281550+0000","last_became_active":"2024-03-24T12:44:48.876647+0000","last_became_peered":"2024-03-24T12:44:48.876647+0000","last_unstale":"2024-03-24T12:46:30.281550+0000","last_undegraded":"2024-03-24T12:46:30.281550+0000","last_fullsized":"2024-03-24T12:46:30.281550+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:28.074463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:00.198109+0000","last_clean_scrub_stamp":"2024-03-24T12:46:28.074463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:47.436641+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":536,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268483+0000","last_change":"2024-03-24T12:45:16.221400+0000","last_active":"2024-03-24T12:46:30.268483+0000","last_peered":"2024-03-24T12:46:30.268483+0000","last_clean":"2024-03-24T12:46:30.268483+0000","last_became_active":"2024-03-24T12:45:16.220608+0000","last_became_peered":"2024-03-24T12:45:16.220608+0000","last_unstale":"2024-03-24T12:46:30.268483+0000","last_undegraded":"2024-03-24T12:46:30.268483+0000","last_fullsized":"2024-03-24T12:46:30.268483+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:44:34.685258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:44:34.685258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"395'3056","reported_seq":3092,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.829558+0000","last_change":"2024-03-24T12:45:43.235757+0000","last_active":"2024-03-24T12:46:32.829558+0000","last_peered":"2024-03-24T12:46:32.829558+0000","last_clean":"2024-03-24T12:46:32.829558+0000","last_became_active":"2024-03-24T12:45:43.235407+0000","last_became_peered":"2024-03-24T12:45:43.235407+0000","last_unstale":"2024-03-24T12:46:32.829558+0000","last_undegraded":"2024-03-24T12:46:32.829558+0000","last_fullsized":"2024-03-24T12:46:32.829558+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3056,"log_dups_size":0,"ondisk_log_size":3056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:10.644056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":200278016,"num_objects":3056,"num_object_clones":0,"num_object_copies":9168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3056,"num_write_kb":195584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":752,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274134+0000","last_change":"2024-03-24T12:46:21.688315+0000","last_active":"2024-03-24T12:46:30.274134+0000","last_peered":"2024-03-24T12:46:30.274134+0000","last_clean":"2024-03-24T12:46:30.274134+0000","last_became_active":"2024-03-24T12:45:40.232411+0000","last_became_peered":"2024-03-24T12:45:40.232411+0000","last_unstale":"2024-03-24T12:46:30.274134+0000","last_undegraded":"2024-03-24T12:46:30.274134+0000","last_fullsized":"2024-03-24T12:46:30.274134+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:21.688184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:21.688184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:48.313281+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"390'42","reported_seq":449,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.280099+0000","last_change":"2024-03-24T12:45:53.036097+0000","last_active":"2024-03-24T12:46:30.280099+0000","last_peered":"2024-03-24T12:46:30.280099+0000","last_clean":"2024-03-24T12:46:30.280099+0000","last_became_active":"2024-03-24T12:45:40.256722+0000","last_became_peered":"2024-03-24T12:45:40.256722+0000","last_unstale":"2024-03-24T12:46:30.280099+0000","last_undegraded":"2024-03-24T12:46:30.280099+0000","last_fullsized":"2024-03-24T12:46:30.280099+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'30","last_scrub_stamp":"2024-03-24T12:45:53.035945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:53.035945+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:07.486150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":724,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.262964+0000","last_change":"2024-03-24T12:45:56.018473+0000","last_active":"2024-03-24T12:46:30.262964+0000","last_peered":"2024-03-24T12:46:30.262964+0000","last_clean":"2024-03-24T12:46:30.262964+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:46:30.262964+0000","last_undegraded":"2024-03-24T12:46:30.262964+0000","last_fullsized":"2024-03-24T12:46:30.262964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:56.018335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:56.018335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:24.758836+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":734,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274583+0000","last_change":"2024-03-24T12:46:10.560869+0000","last_active":"2024-03-24T12:46:30.274583+0000","last_peered":"2024-03-24T12:46:30.274583+0000","last_clean":"2024-03-24T12:46:30.274583+0000","last_became_active":"2024-03-24T12:45:40.227539+0000","last_became_peered":"2024-03-24T12:45:40.227539+0000","last_unstale":"2024-03-24T12:46:30.274583+0000","last_undegraded":"2024-03-24T12:46:30.274583+0000","last_fullsized":"2024-03-24T12:46:30.274583+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:10.560773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:10.560773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:23.548563+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":528,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274797+0000","last_change":"2024-03-24T12:45:16.221053+0000","last_active":"2024-03-24T12:46:30.274797+0000","last_peered":"2024-03-24T12:46:30.274797+0000","last_clean":"2024-03-24T12:46:30.274797+0000","last_became_active":"2024-03-24T12:45:16.220449+0000","last_became_peered":"2024-03-24T12:45:16.220449+0000","last_unstale":"2024-03-24T12:46:30.274797+0000","last_undegraded":"2024-03-24T12:46:30.274797+0000","last_fullsized":"2024-03-24T12:46:30.274797+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:44:21.566951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:44:21.566951+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"395'3109","reported_seq":3145,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.673136+0000","last_change":"2024-03-24T12:45:43.227598+0000","last_active":"2024-03-24T12:46:32.673136+0000","last_peered":"2024-03-24T12:46:32.673136+0000","last_clean":"2024-03-24T12:46:32.673136+0000","last_became_active":"2024-03-24T12:45:43.227183+0000","last_became_peered":"2024-03-24T12:45:43.227183+0000","last_unstale":"2024-03-24T12:46:32.673136+0000","last_undegraded":"2024-03-24T12:46:32.673136+0000","last_fullsized":"2024-03-24T12:46:32.673136+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3109,"log_dups_size":0,"ondisk_log_size":3109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:08.633858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203751424,"num_objects":3109,"num_object_clones":0,"num_object_copies":9327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3109,"num_write_kb":198976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"380'9","reported_seq":91,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.278696+0000","last_change":"2024-03-24T12:46:17.715550+0000","last_active":"2024-03-24T12:46:30.278696+0000","last_peered":"2024-03-24T12:46:30.278696+0000","last_clean":"2024-03-24T12:46:30.278696+0000","last_became_active":"2024-03-24T12:45:40.248941+0000","last_became_peered":"2024-03-24T12:45:40.248941+0000","last_unstale":"2024-03-24T12:46:30.278696+0000","last_undegraded":"2024-03-24T12:46:30.278696+0000","last_fullsized":"2024-03-24T12:46:30.278696+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'9","last_scrub_stamp":"2024-03-24T12:46:17.715400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:17.715400+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:43.926057+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"390'49","reported_seq":855,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274368+0000","last_change":"2024-03-24T12:45:13.322895+0000","last_active":"2024-03-24T12:46:30.274368+0000","last_peered":"2024-03-24T12:46:30.274368+0000","last_clean":"2024-03-24T12:46:30.274368+0000","last_became_active":"2024-03-24T12:45:13.309157+0000","last_became_peered":"2024-03-24T12:45:13.309157+0000","last_unstale":"2024-03-24T12:46:30.274368+0000","last_undegraded":"2024-03-24T12:46:30.274368+0000","last_fullsized":"2024-03-24T12:46:30.274368+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-24T12:44:34.419203+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:44:34.419203+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":749,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.466712+0000","last_change":"2024-03-24T12:46:32.466755+0000","last_active":"2024-03-24T12:46:32.466712+0000","last_peered":"2024-03-24T12:46:32.466712+0000","last_clean":"2024-03-24T12:46:32.466712+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:46:32.466712+0000","last_undegraded":"2024-03-24T12:46:32.466712+0000","last_fullsized":"2024-03-24T12:46:32.466712+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:46:32.466631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:32.466631+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:49.392397+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":746,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269098+0000","last_change":"2024-03-24T12:46:03.105883+0000","last_active":"2024-03-24T12:46:30.269098+0000","last_peered":"2024-03-24T12:46:30.269098+0000","last_clean":"2024-03-24T12:46:30.269098+0000","last_became_active":"2024-03-24T12:45:13.313559+0000","last_became_peered":"2024-03-24T12:45:13.313559+0000","last_unstale":"2024-03-24T12:46:30.269098+0000","last_undegraded":"2024-03-24T12:46:30.269098+0000","last_fullsized":"2024-03-24T12:46:30.269098+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:03.105774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:03.105774+0000","last_clean_scrub_stamp":"2024-03-24T12:46:03.105774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:13.587775+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":868,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268497+0000","last_change":"2024-03-24T12:45:48.880834+0000","last_active":"2024-03-24T12:46:30.268497+0000","last_peered":"2024-03-24T12:46:30.268497+0000","last_clean":"2024-03-24T12:46:30.268497+0000","last_became_active":"2024-03-24T12:45:43.234521+0000","last_became_peered":"2024-03-24T12:45:43.234521+0000","last_unstale":"2024-03-24T12:46:30.268497+0000","last_undegraded":"2024-03-24T12:46:30.268497+0000","last_fullsized":"2024-03-24T12:46:30.268497+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:45:48.880729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:45:48.880729+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:57.474353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"395'3311","reported_seq":3347,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:34.756073+0000","last_change":"2024-03-24T12:45:43.236122+0000","last_active":"2024-03-24T12:46:34.756073+0000","last_peered":"2024-03-24T12:46:34.756073+0000","last_clean":"2024-03-24T12:46:34.756073+0000","last_became_active":"2024-03-24T12:45:43.235816+0000","last_became_peered":"2024-03-24T12:45:43.235816+0000","last_unstale":"2024-03-24T12:46:34.756073+0000","last_undegraded":"2024-03-24T12:46:34.756073+0000","last_fullsized":"2024-03-24T12:46:34.756073+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3311,"log_dups_size":0,"ondisk_log_size":3311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:47.468430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216989696,"num_objects":3311,"num_object_clones":0,"num_object_copies":9933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3311,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3311,"num_write_kb":211904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"370'16","reported_seq":1016,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.276643+0000","last_change":"2024-03-24T12:45:54.070898+0000","last_active":"2024-03-24T12:46:30.276643+0000","last_peered":"2024-03-24T12:46:30.276643+0000","last_clean":"2024-03-24T12:46:30.276643+0000","last_became_active":"2024-03-24T12:45:13.320893+0000","last_became_peered":"2024-03-24T12:45:13.320893+0000","last_unstale":"2024-03-24T12:46:30.276643+0000","last_undegraded":"2024-03-24T12:46:30.276643+0000","last_fullsized":"2024-03-24T12:46:30.276643+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'16","last_scrub_stamp":"2024-03-24T12:45:54.070804+0000","last_deep_scrub":"318'14","last_deep_scrub_stamp":"2024-03-24T12:42:45.093908+0000","last_clean_scrub_stamp":"2024-03-24T12:45:54.070804+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:00.575976+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"390'35","reported_seq":791,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.277078+0000","last_change":"2024-03-24T12:46:19.770424+0000","last_active":"2024-03-24T12:46:30.277078+0000","last_peered":"2024-03-24T12:46:30.277078+0000","last_clean":"2024-03-24T12:46:30.277078+0000","last_became_active":"2024-03-24T12:44:48.870632+0000","last_became_peered":"2024-03-24T12:44:48.870632+0000","last_unstale":"2024-03-24T12:46:30.277078+0000","last_undegraded":"2024-03-24T12:46:30.277078+0000","last_fullsized":"2024-03-24T12:46:30.277078+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'35","last_scrub_stamp":"2024-03-24T12:46:19.770287+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:46:19.770287+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:37.559235+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":404,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269784+0000","last_change":"2024-03-24T12:46:21.642486+0000","last_active":"2024-03-24T12:46:30.269784+0000","last_peered":"2024-03-24T12:46:30.269784+0000","last_clean":"2024-03-24T12:46:30.269784+0000","last_became_active":"2024-03-24T12:45:40.245717+0000","last_became_peered":"2024-03-24T12:45:40.245717+0000","last_unstale":"2024-03-24T12:46:30.269784+0000","last_undegraded":"2024-03-24T12:46:30.269784+0000","last_fullsized":"2024-03-24T12:46:30.269784+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:21.642368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:21.642368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:23.741548+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":735,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279448+0000","last_change":"2024-03-24T12:45:55.103722+0000","last_active":"2024-03-24T12:46:30.279448+0000","last_peered":"2024-03-24T12:46:30.279448+0000","last_clean":"2024-03-24T12:46:30.279448+0000","last_became_active":"2024-03-24T12:45:40.257194+0000","last_became_peered":"2024-03-24T12:45:40.257194+0000","last_unstale":"2024-03-24T12:46:30.279448+0000","last_undegraded":"2024-03-24T12:46:30.279448+0000","last_fullsized":"2024-03-24T12:46:30.279448+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:55.103636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:45:55.103636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:20.360847+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":864,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269275+0000","last_change":"2024-03-24T12:45:49.875212+0000","last_active":"2024-03-24T12:46:30.269275+0000","last_peered":"2024-03-24T12:46:30.269275+0000","last_clean":"2024-03-24T12:46:30.269275+0000","last_became_active":"2024-03-24T12:45:43.235500+0000","last_became_peered":"2024-03-24T12:45:43.235500+0000","last_unstale":"2024-03-24T12:46:30.269275+0000","last_undegraded":"2024-03-24T12:46:30.269275+0000","last_fullsized":"2024-03-24T12:46:30.269275+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:45:49.875065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:45:49.875065+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:01.657197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.c","version":"395'3233","reported_seq":3269,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:34.978710+0000","last_change":"2024-03-24T12:45:43.234493+0000","last_active":"2024-03-24T12:46:34.978710+0000","last_peered":"2024-03-24T12:46:34.978710+0000","last_clean":"2024-03-24T12:46:34.978710+0000","last_became_active":"2024-03-24T12:45:43.234204+0000","last_became_peered":"2024-03-24T12:45:43.234204+0000","last_unstale":"2024-03-24T12:46:34.978710+0000","last_undegraded":"2024-03-24T12:46:34.978710+0000","last_fullsized":"2024-03-24T12:46:34.978710+0000","mapping_epoch":382,"log_start":"388'300","ondisk_log_start":"388'300","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":2933,"log_dups_size":300,"ondisk_log_size":2933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:04.799248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211877888,"num_objects":3233,"num_object_clones":0,"num_object_copies":9699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3233,"num_write_kb":206912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":663,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.282086+0000","last_change":"2024-03-24T12:45:13.324296+0000","last_active":"2024-03-24T12:46:30.282086+0000","last_peered":"2024-03-24T12:46:30.282086+0000","last_clean":"2024-03-24T12:46:30.282086+0000","last_became_active":"2024-03-24T12:45:13.323975+0000","last_became_peered":"2024-03-24T12:45:13.323975+0000","last_unstale":"2024-03-24T12:46:30.282086+0000","last_undegraded":"2024-03-24T12:46:30.282086+0000","last_fullsized":"2024-03-24T12:46:30.282086+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:43.893545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:44:43.893545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"390'47","reported_seq":805,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279600+0000","last_change":"2024-03-24T12:44:52.923795+0000","last_active":"2024-03-24T12:46:30.279600+0000","last_peered":"2024-03-24T12:46:30.279600+0000","last_clean":"2024-03-24T12:46:30.279600+0000","last_became_active":"2024-03-24T12:44:48.875042+0000","last_became_peered":"2024-03-24T12:44:48.875042+0000","last_unstale":"2024-03-24T12:46:30.279600+0000","last_undegraded":"2024-03-24T12:46:30.279600+0000","last_fullsized":"2024-03-24T12:46:30.279600+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-24T12:44:52.923646+0000","last_deep_scrub":"195'27","last_deep_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_clean_scrub_stamp":"2024-03-24T12:44:52.923646+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":640,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.277422+0000","last_change":"2024-03-24T12:44:57.808299+0000","last_active":"2024-03-24T12:46:30.277422+0000","last_peered":"2024-03-24T12:46:30.277422+0000","last_clean":"2024-03-24T12:46:30.277422+0000","last_became_active":"2024-03-24T12:44:48.887828+0000","last_became_peered":"2024-03-24T12:44:48.887828+0000","last_unstale":"2024-03-24T12:46:30.277422+0000","last_undegraded":"2024-03-24T12:46:30.277422+0000","last_fullsized":"2024-03-24T12:46:30.277422+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:57.808197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:44:57.808197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":762,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.276173+0000","last_change":"2024-03-24T12:46:21.792474+0000","last_active":"2024-03-24T12:46:30.276173+0000","last_peered":"2024-03-24T12:46:30.276173+0000","last_clean":"2024-03-24T12:46:30.276173+0000","last_became_active":"2024-03-24T12:45:13.300587+0000","last_became_peered":"2024-03-24T12:45:13.300587+0000","last_unstale":"2024-03-24T12:46:30.276173+0000","last_undegraded":"2024-03-24T12:46:30.276173+0000","last_fullsized":"2024-03-24T12:46:30.276173+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:21.792342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:21.792342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:43.285526+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":376,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279528+0000","last_change":"2024-03-24T12:45:49.950311+0000","last_active":"2024-03-24T12:46:30.279528+0000","last_peered":"2024-03-24T12:46:30.279528+0000","last_clean":"2024-03-24T12:46:30.279528+0000","last_became_active":"2024-03-24T12:45:43.231412+0000","last_became_peered":"2024-03-24T12:45:43.231412+0000","last_unstale":"2024-03-24T12:46:30.279528+0000","last_undegraded":"2024-03-24T12:46:30.279528+0000","last_fullsized":"2024-03-24T12:46:30.279528+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:45:49.950231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:45:49.950231+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:50.025030+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"395'3116","reported_seq":3152,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.924791+0000","last_change":"2024-03-24T12:45:43.241315+0000","last_active":"2024-03-24T12:46:32.924791+0000","last_peered":"2024-03-24T12:46:32.924791+0000","last_clean":"2024-03-24T12:46:32.924791+0000","last_became_active":"2024-03-24T12:45:43.240963+0000","last_became_peered":"2024-03-24T12:45:43.240963+0000","last_unstale":"2024-03-24T12:46:32.924791+0000","last_undegraded":"2024-03-24T12:46:32.924791+0000","last_fullsized":"2024-03-24T12:46:32.924791+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3116,"log_dups_size":0,"ondisk_log_size":3116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:11.798777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204210176,"num_objects":3116,"num_object_clones":0,"num_object_copies":9348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3116,"num_write_kb":199424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":408,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.283190+0000","last_change":"2024-03-24T12:45:11.355222+0000","last_active":"2024-03-24T12:46:30.283190+0000","last_peered":"2024-03-24T12:46:30.283190+0000","last_clean":"2024-03-24T12:46:30.283190+0000","last_became_active":"2024-03-24T12:44:48.869674+0000","last_became_peered":"2024-03-24T12:44:48.869674+0000","last_unstale":"2024-03-24T12:46:30.283190+0000","last_undegraded":"2024-03-24T12:46:30.283190+0000","last_fullsized":"2024-03-24T12:46:30.283190+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:11.355123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:11.355123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:30.535158+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"390'42","reported_seq":797,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279800+0000","last_change":"2024-03-24T12:46:05.173807+0000","last_active":"2024-03-24T12:46:30.279800+0000","last_peered":"2024-03-24T12:46:30.279800+0000","last_clean":"2024-03-24T12:46:30.279800+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:46:30.279800+0000","last_undegraded":"2024-03-24T12:46:30.279800+0000","last_fullsized":"2024-03-24T12:46:30.279800+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'30","last_scrub_stamp":"2024-03-24T12:46:05.173517+0000","last_deep_scrub":"260'30","last_deep_scrub_stamp":"2024-03-24T12:39:51.288276+0000","last_clean_scrub_stamp":"2024-03-24T12:46:05.173517+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:19.527185+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":1980,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:31.006145+0000","last_change":"2024-03-24T12:45:08.898913+0000","last_active":"2024-03-24T12:46:31.006145+0000","last_peered":"2024-03-24T12:46:31.006145+0000","last_clean":"2024-03-24T12:46:31.006145+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:46:31.006145+0000","last_undegraded":"2024-03-24T12:46:31.006145+0000","last_fullsized":"2024-03-24T12:46:31.006145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:45:08.898795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:08.898795+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1221,"num_read_kb":1216,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":754,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.281360+0000","last_change":"2024-03-24T12:46:14.118009+0000","last_active":"2024-03-24T12:46:30.281360+0000","last_peered":"2024-03-24T12:46:30.281360+0000","last_clean":"2024-03-24T12:46:30.281360+0000","last_became_active":"2024-03-24T12:45:40.239552+0000","last_became_peered":"2024-03-24T12:45:40.239552+0000","last_unstale":"2024-03-24T12:46:30.281360+0000","last_undegraded":"2024-03-24T12:46:30.281360+0000","last_fullsized":"2024-03-24T12:46:30.281360+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:14.117633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:13.629968+0000","last_clean_scrub_stamp":"2024-03-24T12:46:14.117633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:34.104347+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":733,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.277410+0000","last_change":"2024-03-24T12:45:40.253070+0000","last_active":"2024-03-24T12:46:30.277410+0000","last_peered":"2024-03-24T12:46:30.277410+0000","last_clean":"2024-03-24T12:46:30.277410+0000","last_became_active":"2024-03-24T12:45:40.252202+0000","last_became_peered":"2024-03-24T12:45:40.252202+0000","last_unstale":"2024-03-24T12:46:30.277410+0000","last_undegraded":"2024-03-24T12:46:30.277410+0000","last_fullsized":"2024-03-24T12:46:30.277410+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:03.186807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:03.186807+0000","last_clean_scrub_stamp":"2024-03-24T12:45:03.186807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":775,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274500+0000","last_change":"2024-03-24T12:45:50.557525+0000","last_active":"2024-03-24T12:46:30.274500+0000","last_peered":"2024-03-24T12:46:30.274500+0000","last_clean":"2024-03-24T12:46:30.274500+0000","last_became_active":"2024-03-24T12:45:16.221340+0000","last_became_peered":"2024-03-24T12:45:16.221340+0000","last_unstale":"2024-03-24T12:46:30.274500+0000","last_undegraded":"2024-03-24T12:46:30.274500+0000","last_fullsized":"2024-03-24T12:46:30.274500+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:45:50.557414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:45:50.557414+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:54.008104+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"395'3225","reported_seq":3261,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.914108+0000","last_change":"2024-03-24T12:45:43.228195+0000","last_active":"2024-03-24T12:46:32.914108+0000","last_peered":"2024-03-24T12:46:32.914108+0000","last_clean":"2024-03-24T12:46:32.914108+0000","last_became_active":"2024-03-24T12:45:43.227875+0000","last_became_peered":"2024-03-24T12:45:43.227875+0000","last_unstale":"2024-03-24T12:46:32.914108+0000","last_undegraded":"2024-03-24T12:46:32.914108+0000","last_fullsized":"2024-03-24T12:46:32.914108+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3225,"log_dups_size":0,"ondisk_log_size":3225,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:56.309020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211353600,"num_objects":3225,"num_object_clones":0,"num_object_copies":9675,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3225,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3225,"num_write_kb":206400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":709,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269083+0000","last_change":"2024-03-24T12:44:52.778468+0000","last_active":"2024-03-24T12:46:30.269083+0000","last_peered":"2024-03-24T12:46:30.269083+0000","last_clean":"2024-03-24T12:46:30.269083+0000","last_became_active":"2024-03-24T12:44:48.868097+0000","last_became_peered":"2024-03-24T12:44:48.868097+0000","last_unstale":"2024-03-24T12:46:30.269083+0000","last_undegraded":"2024-03-24T12:46:30.269083+0000","last_fullsized":"2024-03-24T12:46:30.269083+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:52.778341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:44:52.778341+0000","last_clean_scrub_stamp":"2024-03-24T12:44:52.778341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"390'43","reported_seq":683,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275109+0000","last_change":"2024-03-24T12:45:13.327526+0000","last_active":"2024-03-24T12:46:30.275109+0000","last_peered":"2024-03-24T12:46:30.275109+0000","last_clean":"2024-03-24T12:46:30.275109+0000","last_became_active":"2024-03-24T12:45:13.308927+0000","last_became_peered":"2024-03-24T12:45:13.308927+0000","last_unstale":"2024-03-24T12:46:30.275109+0000","last_undegraded":"2024-03-24T12:46:30.275109+0000","last_fullsized":"2024-03-24T12:46:30.275109+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'30","last_scrub_stamp":"2024-03-24T12:44:44.924886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:44:44.924886+0000","objects_scrubbed":8,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":732,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274109+0000","last_change":"2024-03-24T12:46:22.722964+0000","last_active":"2024-03-24T12:46:30.274109+0000","last_peered":"2024-03-24T12:46:30.274109+0000","last_clean":"2024-03-24T12:46:30.274109+0000","last_became_active":"2024-03-24T12:45:40.249404+0000","last_became_peered":"2024-03-24T12:45:40.249404+0000","last_unstale":"2024-03-24T12:46:30.274109+0000","last_undegraded":"2024-03-24T12:46:30.274109+0000","last_fullsized":"2024-03-24T12:46:30.274109+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:22.722795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:22.722795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:33.201103+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1051,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268074+0000","last_change":"2024-03-24T12:46:11.041441+0000","last_active":"2024-03-24T12:46:30.268074+0000","last_peered":"2024-03-24T12:46:30.268074+0000","last_clean":"2024-03-24T12:46:30.268074+0000","last_became_active":"2024-03-24T12:45:16.218785+0000","last_became_peered":"2024-03-24T12:45:16.218785+0000","last_unstale":"2024-03-24T12:46:30.268074+0000","last_undegraded":"2024-03-24T12:46:30.268074+0000","last_fullsized":"2024-03-24T12:46:30.268074+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:46:11.041313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:46:11.041313+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:16.166960+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":407,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.263108+0000","last_change":"2024-03-24T12:44:59.352953+0000","last_active":"2024-03-24T12:46:30.263108+0000","last_peered":"2024-03-24T12:46:30.263108+0000","last_clean":"2024-03-24T12:46:30.263108+0000","last_became_active":"2024-03-24T12:44:48.874876+0000","last_became_peered":"2024-03-24T12:44:48.874876+0000","last_unstale":"2024-03-24T12:46:30.263108+0000","last_undegraded":"2024-03-24T12:46:30.263108+0000","last_fullsized":"2024-03-24T12:46:30.263108+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:59.352835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:59.352835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"395'3136","reported_seq":3170,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.917544+0000","last_change":"2024-03-24T12:45:43.236500+0000","last_active":"2024-03-24T12:46:32.917544+0000","last_peered":"2024-03-24T12:46:32.917544+0000","last_clean":"2024-03-24T12:46:32.917544+0000","last_became_active":"2024-03-24T12:45:43.231774+0000","last_became_peered":"2024-03-24T12:45:43.231774+0000","last_unstale":"2024-03-24T12:46:32.917544+0000","last_undegraded":"2024-03-24T12:46:32.917544+0000","last_fullsized":"2024-03-24T12:46:32.917544+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3136,"log_dups_size":0,"ondisk_log_size":3136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:01.699360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205520896,"num_objects":3136,"num_object_clones":0,"num_object_copies":9408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3136,"num_write_kb":200704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":717,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269395+0000","last_change":"2024-03-24T12:45:00.789166+0000","last_active":"2024-03-24T12:46:30.269395+0000","last_peered":"2024-03-24T12:46:30.269395+0000","last_clean":"2024-03-24T12:46:30.269395+0000","last_became_active":"2024-03-24T12:44:48.874944+0000","last_became_peered":"2024-03-24T12:44:48.874944+0000","last_unstale":"2024-03-24T12:46:30.269395+0000","last_undegraded":"2024-03-24T12:46:30.269395+0000","last_fullsized":"2024-03-24T12:46:30.269395+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:00.788798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:00.788798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"390'26","reported_seq":785,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.282841+0000","last_change":"2024-03-24T12:46:28.873246+0000","last_active":"2024-03-24T12:46:30.282841+0000","last_peered":"2024-03-24T12:46:30.282841+0000","last_clean":"2024-03-24T12:46:30.282841+0000","last_became_active":"2024-03-24T12:44:48.870820+0000","last_became_peered":"2024-03-24T12:44:48.870820+0000","last_unstale":"2024-03-24T12:46:30.282841+0000","last_undegraded":"2024-03-24T12:46:30.282841+0000","last_fullsized":"2024-03-24T12:46:30.282841+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'26","last_scrub_stamp":"2024-03-24T12:46:28.873107+0000","last_deep_scrub":"207'22","last_deep_scrub_stamp":"2024-03-24T12:44:50.024489+0000","last_clean_scrub_stamp":"2024-03-24T12:46:28.873107+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:33.847919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":769,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.272307+0000","last_change":"2024-03-24T12:45:48.379164+0000","last_active":"2024-03-24T12:46:30.272307+0000","last_peered":"2024-03-24T12:46:30.272307+0000","last_clean":"2024-03-24T12:46:30.272307+0000","last_became_active":"2024-03-24T12:45:40.246825+0000","last_became_peered":"2024-03-24T12:45:40.246825+0000","last_unstale":"2024-03-24T12:46:30.272307+0000","last_undegraded":"2024-03-24T12:46:30.272307+0000","last_fullsized":"2024-03-24T12:46:30.272307+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:48.379081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:48.379081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:06.547545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":747,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.266321+0000","last_change":"2024-03-24T12:46:24.909961+0000","last_active":"2024-03-24T12:46:30.266321+0000","last_peered":"2024-03-24T12:46:30.266321+0000","last_clean":"2024-03-24T12:46:30.266321+0000","last_became_active":"2024-03-24T12:45:13.313392+0000","last_became_peered":"2024-03-24T12:45:13.313392+0000","last_unstale":"2024-03-24T12:46:30.266321+0000","last_undegraded":"2024-03-24T12:46:30.266321+0000","last_fullsized":"2024-03-24T12:46:30.266321+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:24.909653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:24.909653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:45.786341+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":754,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268917+0000","last_change":"2024-03-24T12:45:13.312221+0000","last_active":"2024-03-24T12:46:30.268917+0000","last_peered":"2024-03-24T12:46:30.268917+0000","last_clean":"2024-03-24T12:46:30.268917+0000","last_became_active":"2024-03-24T12:45:13.311960+0000","last_became_peered":"2024-03-24T12:45:13.311960+0000","last_unstale":"2024-03-24T12:46:30.268917+0000","last_undegraded":"2024-03-24T12:46:30.268917+0000","last_fullsized":"2024-03-24T12:46:30.268917+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:58.882258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:15.087115+0000","last_clean_scrub_stamp":"2024-03-24T12:44:58.882258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":496,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.281986+0000","last_change":"2024-03-24T12:46:16.236685+0000","last_active":"2024-03-24T12:46:30.281986+0000","last_peered":"2024-03-24T12:46:30.281986+0000","last_clean":"2024-03-24T12:46:30.281986+0000","last_became_active":"2024-03-24T12:42:40.200767+0000","last_became_peered":"2024-03-24T12:42:40.200767+0000","last_unstale":"2024-03-24T12:46:30.281986+0000","last_undegraded":"2024-03-24T12:46:30.281986+0000","last_fullsized":"2024-03-24T12:46:30.281986+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:46:16.236536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:46:16.236536+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:30.198700+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"395'3112","reported_seq":3148,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.908750+0000","last_change":"2024-03-24T12:45:43.242202+0000","last_active":"2024-03-24T12:46:32.908750+0000","last_peered":"2024-03-24T12:46:32.908750+0000","last_clean":"2024-03-24T12:46:32.908750+0000","last_became_active":"2024-03-24T12:45:43.241842+0000","last_became_peered":"2024-03-24T12:45:43.241842+0000","last_unstale":"2024-03-24T12:46:32.908750+0000","last_undegraded":"2024-03-24T12:46:32.908750+0000","last_fullsized":"2024-03-24T12:46:32.908750+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3112,"log_dups_size":0,"ondisk_log_size":3112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:54.404807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203948032,"num_objects":3112,"num_object_clones":0,"num_object_copies":9336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3112,"num_write_kb":199168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":732,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.280726+0000","last_change":"2024-03-24T12:46:08.171021+0000","last_active":"2024-03-24T12:46:30.280726+0000","last_peered":"2024-03-24T12:46:30.280726+0000","last_clean":"2024-03-24T12:46:30.280726+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:46:30.280726+0000","last_undegraded":"2024-03-24T12:46:30.280726+0000","last_fullsized":"2024-03-24T12:46:30.280726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:08.170846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:08.170846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:25.504004+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"390'64","reported_seq":997,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.277669+0000","last_change":"2024-03-24T12:46:03.462041+0000","last_active":"2024-03-24T12:46:30.277669+0000","last_peered":"2024-03-24T12:46:30.277669+0000","last_clean":"2024-03-24T12:46:30.277669+0000","last_became_active":"2024-03-24T12:45:40.246159+0000","last_became_peered":"2024-03-24T12:45:40.246159+0000","last_unstale":"2024-03-24T12:46:30.277669+0000","last_undegraded":"2024-03-24T12:46:30.277669+0000","last_fullsized":"2024-03-24T12:46:30.277669+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'52","last_scrub_stamp":"2024-03-24T12:46:03.461888+0000","last_deep_scrub":"248'51","last_deep_scrub_stamp":"2024-03-24T12:39:47.606538+0000","last_clean_scrub_stamp":"2024-03-24T12:46:03.461888+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:30.129732+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":758,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274923+0000","last_change":"2024-03-24T12:45:47.497683+0000","last_active":"2024-03-24T12:46:30.274923+0000","last_peered":"2024-03-24T12:46:30.274923+0000","last_clean":"2024-03-24T12:46:30.274923+0000","last_became_active":"2024-03-24T12:45:13.300234+0000","last_became_peered":"2024-03-24T12:45:13.300234+0000","last_unstale":"2024-03-24T12:46:30.274923+0000","last_undegraded":"2024-03-24T12:46:30.274923+0000","last_fullsized":"2024-03-24T12:46:30.274923+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:47.497596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:47.497596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:53.810993+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":492,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.276710+0000","last_change":"2024-03-24T12:46:20.764983+0000","last_active":"2024-03-24T12:46:30.276710+0000","last_peered":"2024-03-24T12:46:30.276710+0000","last_clean":"2024-03-24T12:46:30.276710+0000","last_became_active":"2024-03-24T12:45:40.258591+0000","last_became_peered":"2024-03-24T12:45:40.258591+0000","last_unstale":"2024-03-24T12:46:30.276710+0000","last_undegraded":"2024-03-24T12:46:30.276710+0000","last_fullsized":"2024-03-24T12:46:30.276710+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:46:20.764841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:20.764841+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:43.644952+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":50,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275023+0000","last_change":"2024-03-24T12:45:46.510121+0000","last_active":"2024-03-24T12:46:30.275023+0000","last_peered":"2024-03-24T12:46:30.275023+0000","last_clean":"2024-03-24T12:46:30.275023+0000","last_became_active":"2024-03-24T12:45:40.249680+0000","last_became_peered":"2024-03-24T12:45:40.249680+0000","last_unstale":"2024-03-24T12:46:30.275023+0000","last_undegraded":"2024-03-24T12:46:30.275023+0000","last_fullsized":"2024-03-24T12:46:30.275023+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:46.509996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:45:46.509996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:06.784540+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":542,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.281005+0000","last_change":"2024-03-24T12:46:09.166237+0000","last_active":"2024-03-24T12:46:30.281005+0000","last_peered":"2024-03-24T12:46:30.281005+0000","last_clean":"2024-03-24T12:46:30.281005+0000","last_became_active":"2024-03-24T12:44:51.223311+0000","last_became_peered":"2024-03-24T12:44:51.223311+0000","last_unstale":"2024-03-24T12:46:30.281005+0000","last_undegraded":"2024-03-24T12:46:30.281005+0000","last_fullsized":"2024-03-24T12:46:30.281005+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:46:09.165580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:46:09.165580+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:09.634684+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"395'3135","reported_seq":3171,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.812903+0000","last_change":"2024-03-24T12:45:43.235974+0000","last_active":"2024-03-24T12:46:32.812903+0000","last_peered":"2024-03-24T12:46:32.812903+0000","last_clean":"2024-03-24T12:46:32.812903+0000","last_became_active":"2024-03-24T12:45:43.235557+0000","last_became_peered":"2024-03-24T12:45:43.235557+0000","last_unstale":"2024-03-24T12:46:32.812903+0000","last_undegraded":"2024-03-24T12:46:32.812903+0000","last_fullsized":"2024-03-24T12:46:32.812903+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3135,"log_dups_size":0,"ondisk_log_size":3135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:49.283063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205455360,"num_objects":3135,"num_object_clones":0,"num_object_copies":9405,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3135,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3135,"num_write_kb":200640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"390'21","reported_seq":786,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279499+0000","last_change":"2024-03-24T12:45:59.161020+0000","last_active":"2024-03-24T12:46:30.279499+0000","last_peered":"2024-03-24T12:46:30.279499+0000","last_clean":"2024-03-24T12:46:30.279499+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:46:30.279499+0000","last_undegraded":"2024-03-24T12:46:30.279499+0000","last_fullsized":"2024-03-24T12:46:30.279499+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:45:59.160871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:59.160871+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.082106+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":175,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.277306+0000","last_change":"2024-03-24T12:46:12.516626+0000","last_active":"2024-03-24T12:46:30.277306+0000","last_peered":"2024-03-24T12:46:30.277306+0000","last_clean":"2024-03-24T12:46:30.277306+0000","last_became_active":"2024-03-24T12:45:40.252667+0000","last_became_peered":"2024-03-24T12:45:40.252667+0000","last_unstale":"2024-03-24T12:46:30.277306+0000","last_undegraded":"2024-03-24T12:46:30.277306+0000","last_fullsized":"2024-03-24T12:46:30.277306+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:46:12.516495+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:46:12.516495+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:32.355920+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":746,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268812+0000","last_change":"2024-03-24T12:45:40.240864+0000","last_active":"2024-03-24T12:46:30.268812+0000","last_peered":"2024-03-24T12:46:30.268812+0000","last_clean":"2024-03-24T12:46:30.268812+0000","last_became_active":"2024-03-24T12:45:40.240504+0000","last_became_peered":"2024-03-24T12:45:40.240504+0000","last_unstale":"2024-03-24T12:46:30.268812+0000","last_undegraded":"2024-03-24T12:46:30.268812+0000","last_fullsized":"2024-03-24T12:46:30.268812+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:11.890544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:11.890544+0000","last_clean_scrub_stamp":"2024-03-24T12:45:11.890544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":720,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.281575+0000","last_change":"2024-03-24T12:44:42.124563+0000","last_active":"2024-03-24T12:46:30.281575+0000","last_peered":"2024-03-24T12:46:30.281575+0000","last_clean":"2024-03-24T12:46:30.281575+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:46:30.281575+0000","last_undegraded":"2024-03-24T12:46:30.281575+0000","last_fullsized":"2024-03-24T12:46:30.281575+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:42.124327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:42.124327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":169,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"395'3202","reported_seq":3238,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.821611+0000","last_change":"2024-03-24T12:45:43.242582+0000","last_active":"2024-03-24T12:46:32.821611+0000","last_peered":"2024-03-24T12:46:32.821611+0000","last_clean":"2024-03-24T12:46:32.821611+0000","last_became_active":"2024-03-24T12:45:43.242256+0000","last_became_peered":"2024-03-24T12:45:43.242256+0000","last_unstale":"2024-03-24T12:46:32.821611+0000","last_undegraded":"2024-03-24T12:46:32.821611+0000","last_fullsized":"2024-03-24T12:46:32.821611+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3202,"log_dups_size":0,"ondisk_log_size":3202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:09.780701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":209846272,"num_objects":3202,"num_object_clones":0,"num_object_copies":9606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3202,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3202,"num_write_kb":204928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"391'51","reported_seq":549,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.276845+0000","last_change":"2024-03-24T12:46:00.353017+0000","last_active":"2024-03-24T12:46:30.276845+0000","last_peered":"2024-03-24T12:46:30.276845+0000","last_clean":"2024-03-24T12:46:30.276845+0000","last_became_active":"2024-03-24T12:45:13.319445+0000","last_became_peered":"2024-03-24T12:45:13.319445+0000","last_unstale":"2024-03-24T12:46:30.276845+0000","last_undegraded":"2024-03-24T12:46:30.276845+0000","last_fullsized":"2024-03-24T12:46:30.276845+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-24T12:46:00.352851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:00.352851+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:25.389136+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":763,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268790+0000","last_change":"2024-03-24T12:46:16.768092+0000","last_active":"2024-03-24T12:46:30.268790+0000","last_peered":"2024-03-24T12:46:30.268790+0000","last_clean":"2024-03-24T12:46:30.268790+0000","last_became_active":"2024-03-24T12:45:40.233640+0000","last_became_peered":"2024-03-24T12:45:40.233640+0000","last_unstale":"2024-03-24T12:46:30.268790+0000","last_undegraded":"2024-03-24T12:46:30.268790+0000","last_fullsized":"2024-03-24T12:46:30.268790+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:16.767979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:46:16.767979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:36.721481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":717,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269745+0000","last_change":"2024-03-24T12:45:08.897316+0000","last_active":"2024-03-24T12:46:30.269745+0000","last_peered":"2024-03-24T12:46:30.269745+0000","last_clean":"2024-03-24T12:46:30.269745+0000","last_became_active":"2024-03-24T12:44:48.888020+0000","last_became_peered":"2024-03-24T12:44:48.888020+0000","last_unstale":"2024-03-24T12:46:30.269745+0000","last_undegraded":"2024-03-24T12:46:30.269745+0000","last_fullsized":"2024-03-24T12:46:30.269745+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:08.897184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:08.897184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":728,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.267957+0000","last_change":"2024-03-24T12:44:46.705056+0000","last_active":"2024-03-24T12:46:30.267957+0000","last_peered":"2024-03-24T12:46:30.267957+0000","last_clean":"2024-03-24T12:46:30.267957+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:46:30.267957+0000","last_undegraded":"2024-03-24T12:46:30.267957+0000","last_fullsized":"2024-03-24T12:46:30.267957+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:46.704942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:46.704942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"395'3221","reported_seq":3257,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:32.913598+0000","last_change":"2024-03-24T12:45:43.241533+0000","last_active":"2024-03-24T12:46:32.913598+0000","last_peered":"2024-03-24T12:46:32.913598+0000","last_clean":"2024-03-24T12:46:32.913598+0000","last_became_active":"2024-03-24T12:45:43.241257+0000","last_became_peered":"2024-03-24T12:45:43.241257+0000","last_unstale":"2024-03-24T12:46:32.913598+0000","last_undegraded":"2024-03-24T12:46:32.913598+0000","last_fullsized":"2024-03-24T12:46:32.913598+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3221,"log_dups_size":0,"ondisk_log_size":3221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:43.036810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":211091456,"num_objects":3221,"num_object_clones":0,"num_object_copies":9663,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3221,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3221,"num_write_kb":206144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":740,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.266046+0000","last_change":"2024-03-24T12:45:12.924498+0000","last_active":"2024-03-24T12:46:30.266046+0000","last_peered":"2024-03-24T12:46:30.266046+0000","last_clean":"2024-03-24T12:46:30.266046+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:46:30.266046+0000","last_undegraded":"2024-03-24T12:46:30.266046+0000","last_fullsized":"2024-03-24T12:46:30.266046+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:12.924420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:12.924420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"390'29","reported_seq":808,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274478+0000","last_change":"2024-03-24T12:46:11.591668+0000","last_active":"2024-03-24T12:46:30.274478+0000","last_peered":"2024-03-24T12:46:30.274478+0000","last_clean":"2024-03-24T12:46:30.274478+0000","last_became_active":"2024-03-24T12:45:40.235146+0000","last_became_peered":"2024-03-24T12:45:40.235146+0000","last_unstale":"2024-03-24T12:46:30.274478+0000","last_undegraded":"2024-03-24T12:46:30.274478+0000","last_fullsized":"2024-03-24T12:46:30.274478+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'25","last_scrub_stamp":"2024-03-24T12:46:11.591573+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:46:11.591573+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:16.313518+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":740,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269334+0000","last_change":"2024-03-24T12:46:11.814739+0000","last_active":"2024-03-24T12:46:30.269334+0000","last_peered":"2024-03-24T12:46:30.269334+0000","last_clean":"2024-03-24T12:46:30.269334+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:46:30.269334+0000","last_undegraded":"2024-03-24T12:46:30.269334+0000","last_fullsized":"2024-03-24T12:46:30.269334+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:11.814638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:11.814638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.771958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":724,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.262284+0000","last_change":"2024-03-24T12:46:00.248579+0000","last_active":"2024-03-24T12:46:30.262284+0000","last_peered":"2024-03-24T12:46:30.262284+0000","last_clean":"2024-03-24T12:46:30.262284+0000","last_became_active":"2024-03-24T12:45:13.264047+0000","last_became_peered":"2024-03-24T12:45:13.264047+0000","last_unstale":"2024-03-24T12:46:30.262284+0000","last_undegraded":"2024-03-24T12:46:30.262284+0000","last_fullsized":"2024-03-24T12:46:30.262284+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:00.248216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:00.248216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:14.231101+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"395'3506","reported_seq":3542,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:34.976900+0000","last_change":"2024-03-24T12:45:43.234010+0000","last_active":"2024-03-24T12:46:34.976900+0000","last_peered":"2024-03-24T12:46:34.976900+0000","last_clean":"2024-03-24T12:46:34.976900+0000","last_became_active":"2024-03-24T12:45:43.233265+0000","last_became_peered":"2024-03-24T12:45:43.233265+0000","last_unstale":"2024-03-24T12:46:34.976900+0000","last_undegraded":"2024-03-24T12:46:34.976900+0000","last_fullsized":"2024-03-24T12:46:34.976900+0000","mapping_epoch":382,"log_start":"388'600","ondisk_log_start":"388'600","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":2906,"log_dups_size":600,"ondisk_log_size":2906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:48.408507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":229769216,"num_objects":3506,"num_object_clones":0,"num_object_copies":10518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3506,"num_write_kb":224384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"370'11","reported_seq":1027,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268819+0000","last_change":"2024-03-24T12:46:01.921661+0000","last_active":"2024-03-24T12:46:30.268819+0000","last_peered":"2024-03-24T12:46:30.268819+0000","last_clean":"2024-03-24T12:46:30.268819+0000","last_became_active":"2024-03-24T12:45:13.259309+0000","last_became_peered":"2024-03-24T12:45:13.259309+0000","last_unstale":"2024-03-24T12:46:30.268819+0000","last_undegraded":"2024-03-24T12:46:30.268819+0000","last_fullsized":"2024-03-24T12:46:30.268819+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'11","last_scrub_stamp":"2024-03-24T12:46:01.921579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:01.921579+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:02.553229+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"390'39","reported_seq":243,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274366+0000","last_change":"2024-03-24T12:45:40.265367+0000","last_active":"2024-03-24T12:46:30.274366+0000","last_peered":"2024-03-24T12:46:30.274366+0000","last_clean":"2024-03-24T12:46:30.274366+0000","last_became_active":"2024-03-24T12:45:40.229387+0000","last_became_peered":"2024-03-24T12:45:40.229387+0000","last_unstale":"2024-03-24T12:46:30.274366+0000","last_undegraded":"2024-03-24T12:46:30.274366+0000","last_fullsized":"2024-03-24T12:46:30.274366+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-24T12:44:39.791402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:44:39.791402+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":752,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269437+0000","last_change":"2024-03-24T12:46:17.778655+0000","last_active":"2024-03-24T12:46:30.269437+0000","last_peered":"2024-03-24T12:46:30.269437+0000","last_clean":"2024-03-24T12:46:30.269437+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:46:30.269437+0000","last_undegraded":"2024-03-24T12:46:30.269437+0000","last_fullsized":"2024-03-24T12:46:30.269437+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:46:17.778517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:17.778517+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:23.225291+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":361,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.282251+0000","last_change":"2024-03-24T12:46:07.137940+0000","last_active":"2024-03-24T12:46:30.282251+0000","last_peered":"2024-03-24T12:46:30.282251+0000","last_clean":"2024-03-24T12:46:30.282251+0000","last_became_active":"2024-03-24T12:45:13.325330+0000","last_became_peered":"2024-03-24T12:45:13.325330+0000","last_unstale":"2024-03-24T12:46:30.282251+0000","last_undegraded":"2024-03-24T12:46:30.282251+0000","last_fullsized":"2024-03-24T12:46:30.282251+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:07.137717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:07.137717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:31.609482+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"390'18","reported_seq":629,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269311+0000","last_change":"2024-03-24T12:45:47.380906+0000","last_active":"2024-03-24T12:46:30.269311+0000","last_peered":"2024-03-24T12:46:30.269311+0000","last_clean":"2024-03-24T12:46:30.269311+0000","last_became_active":"2024-03-24T12:45:40.245027+0000","last_became_peered":"2024-03-24T12:45:40.245027+0000","last_unstale":"2024-03-24T12:46:30.269311+0000","last_undegraded":"2024-03-24T12:46:30.269311+0000","last_fullsized":"2024-03-24T12:46:30.269311+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-24T12:45:47.380748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:47.380748+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:48.194932+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":685,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279862+0000","last_change":"2024-03-24T12:46:12.200721+0000","last_active":"2024-03-24T12:46:30.279862+0000","last_peered":"2024-03-24T12:46:30.279862+0000","last_clean":"2024-03-24T12:46:30.279862+0000","last_became_active":"2024-03-24T12:45:40.255835+0000","last_became_peered":"2024-03-24T12:45:40.255835+0000","last_unstale":"2024-03-24T12:46:30.279862+0000","last_undegraded":"2024-03-24T12:46:30.279862+0000","last_fullsized":"2024-03-24T12:46:30.279862+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:12.200446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:12.200446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:33.879810+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":396,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269242+0000","last_change":"2024-03-24T12:45:40.243760+0000","last_active":"2024-03-24T12:46:30.269242+0000","last_peered":"2024-03-24T12:46:30.269242+0000","last_clean":"2024-03-24T12:46:30.269242+0000","last_became_active":"2024-03-24T12:45:40.243393+0000","last_became_peered":"2024-03-24T12:45:40.243393+0000","last_unstale":"2024-03-24T12:46:30.269242+0000","last_undegraded":"2024-03-24T12:46:30.269242+0000","last_fullsized":"2024-03-24T12:46:30.269242+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:57.075121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:57.075121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":743,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268314+0000","last_change":"2024-03-24T12:46:18.888939+0000","last_active":"2024-03-24T12:46:30.268314+0000","last_peered":"2024-03-24T12:46:30.268314+0000","last_clean":"2024-03-24T12:46:30.268314+0000","last_became_active":"2024-03-24T12:45:13.312958+0000","last_became_peered":"2024-03-24T12:45:13.312958+0000","last_unstale":"2024-03-24T12:46:30.268314+0000","last_undegraded":"2024-03-24T12:46:30.268314+0000","last_fullsized":"2024-03-24T12:46:30.268314+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:18.888854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:55.271679+0000","last_clean_scrub_stamp":"2024-03-24T12:46:18.888854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:31.649819+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"390'21","reported_seq":801,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.280147+0000","last_change":"2024-03-24T12:45:40.290819+0000","last_active":"2024-03-24T12:46:30.280147+0000","last_peered":"2024-03-24T12:46:30.280147+0000","last_clean":"2024-03-24T12:46:30.280147+0000","last_became_active":"2024-03-24T12:45:40.256168+0000","last_became_peered":"2024-03-24T12:45:40.256168+0000","last_unstale":"2024-03-24T12:46:30.280147+0000","last_undegraded":"2024-03-24T12:46:30.280147+0000","last_fullsized":"2024-03-24T12:46:30.280147+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:45:10.860208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:10.860208+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:36.019748+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":766,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268659+0000","last_change":"2024-03-24T12:46:26.643534+0000","last_active":"2024-03-24T12:46:30.268659+0000","last_peered":"2024-03-24T12:46:30.268659+0000","last_clean":"2024-03-24T12:46:30.268659+0000","last_became_active":"2024-03-24T12:45:13.324149+0000","last_became_peered":"2024-03-24T12:45:13.324149+0000","last_unstale":"2024-03-24T12:46:30.268659+0000","last_undegraded":"2024-03-24T12:46:30.268659+0000","last_fullsized":"2024-03-24T12:46:30.268659+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:26.643413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:26.643413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.064715+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":50,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.282549+0000","last_change":"2024-03-24T12:45:51.305650+0000","last_active":"2024-03-24T12:46:30.282549+0000","last_peered":"2024-03-24T12:46:30.282549+0000","last_clean":"2024-03-24T12:46:30.282549+0000","last_became_active":"2024-03-24T12:45:40.249711+0000","last_became_peered":"2024-03-24T12:45:40.249711+0000","last_unstale":"2024-03-24T12:46:30.282549+0000","last_undegraded":"2024-03-24T12:46:30.282549+0000","last_fullsized":"2024-03-24T12:46:30.282549+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:51.305535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:51.305535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:52.414120+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":743,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.277838+0000","last_change":"2024-03-24T12:46:11.562073+0000","last_active":"2024-03-24T12:46:30.277838+0000","last_peered":"2024-03-24T12:46:30.277838+0000","last_clean":"2024-03-24T12:46:30.277838+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:46:30.277838+0000","last_undegraded":"2024-03-24T12:46:30.277838+0000","last_fullsized":"2024-03-24T12:46:30.277838+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:11.561956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:46:11.561956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:12.031927+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"390'25","reported_seq":519,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268894+0000","last_change":"2024-03-24T12:45:13.336981+0000","last_active":"2024-03-24T12:46:30.268894+0000","last_peered":"2024-03-24T12:46:30.268894+0000","last_clean":"2024-03-24T12:46:30.268894+0000","last_became_active":"2024-03-24T12:45:13.312997+0000","last_became_peered":"2024-03-24T12:45:13.312997+0000","last_unstale":"2024-03-24T12:46:30.268894+0000","last_undegraded":"2024-03-24T12:46:30.268894+0000","last_fullsized":"2024-03-24T12:46:30.268894+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:44:54.781997+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-24T12:44:54.781997+0000","last_clean_scrub_stamp":"2024-03-24T12:44:54.781997+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":745,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.282137+0000","last_change":"2024-03-24T12:45:40.251507+0000","last_active":"2024-03-24T12:46:30.282137+0000","last_peered":"2024-03-24T12:46:30.282137+0000","last_clean":"2024-03-24T12:46:30.282137+0000","last_became_active":"2024-03-24T12:45:40.251051+0000","last_became_peered":"2024-03-24T12:45:40.251051+0000","last_unstale":"2024-03-24T12:46:30.282137+0000","last_undegraded":"2024-03-24T12:46:30.282137+0000","last_fullsized":"2024-03-24T12:46:30.282137+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:12.900147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:12.900147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":153,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268897+0000","last_change":"2024-03-24T12:45:51.954271+0000","last_active":"2024-03-24T12:46:30.268897+0000","last_peered":"2024-03-24T12:46:30.268897+0000","last_clean":"2024-03-24T12:46:30.268897+0000","last_became_active":"2024-03-24T12:45:40.243508+0000","last_became_peered":"2024-03-24T12:45:40.243508+0000","last_unstale":"2024-03-24T12:46:30.268897+0000","last_undegraded":"2024-03-24T12:46:30.268897+0000","last_fullsized":"2024-03-24T12:46:30.268897+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-03-24T12:45:51.954162+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:45:51.954162+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:00.849993+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":712,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275919+0000","last_change":"2024-03-24T12:44:57.736897+0000","last_active":"2024-03-24T12:46:30.275919+0000","last_peered":"2024-03-24T12:46:30.275919+0000","last_clean":"2024-03-24T12:46:30.275919+0000","last_became_active":"2024-03-24T12:44:48.872901+0000","last_became_peered":"2024-03-24T12:44:48.872901+0000","last_unstale":"2024-03-24T12:46:30.275919+0000","last_undegraded":"2024-03-24T12:46:30.275919+0000","last_fullsized":"2024-03-24T12:46:30.275919+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:57.736450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:57.736450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"390'30","reported_seq":772,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.262493+0000","last_change":"2024-03-24T12:45:53.015966+0000","last_active":"2024-03-24T12:46:30.262493+0000","last_peered":"2024-03-24T12:46:30.262493+0000","last_clean":"2024-03-24T12:46:30.262493+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:46:30.262493+0000","last_undegraded":"2024-03-24T12:46:30.262493+0000","last_fullsized":"2024-03-24T12:46:30.262493+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:45:53.015813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:53.015813+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:19.152842+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":742,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268387+0000","last_change":"2024-03-24T12:45:58.894456+0000","last_active":"2024-03-24T12:46:30.268387+0000","last_peered":"2024-03-24T12:46:30.268387+0000","last_clean":"2024-03-24T12:46:30.268387+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:46:30.268387+0000","last_undegraded":"2024-03-24T12:46:30.268387+0000","last_fullsized":"2024-03-24T12:46:30.268387+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:58.894376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:58.894376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:07.945657+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"354'11","reported_seq":153,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.280714+0000","last_change":"2024-03-24T12:45:08.811457+0000","last_active":"2024-03-24T12:46:30.280714+0000","last_peered":"2024-03-24T12:46:30.280714+0000","last_clean":"2024-03-24T12:46:30.280714+0000","last_became_active":"2024-03-24T12:44:48.881388+0000","last_became_peered":"2024-03-24T12:44:48.881388+0000","last_unstale":"2024-03-24T12:46:30.280714+0000","last_undegraded":"2024-03-24T12:46:30.280714+0000","last_fullsized":"2024-03-24T12:46:30.280714+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'11","last_scrub_stamp":"2024-03-24T12:45:08.811350+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:08.811350+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":738,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279627+0000","last_change":"2024-03-24T12:46:01.182095+0000","last_active":"2024-03-24T12:46:30.279627+0000","last_peered":"2024-03-24T12:46:30.279627+0000","last_clean":"2024-03-24T12:46:30.279627+0000","last_became_active":"2024-03-24T12:45:40.242801+0000","last_became_peered":"2024-03-24T12:45:40.242801+0000","last_unstale":"2024-03-24T12:46:30.279627+0000","last_undegraded":"2024-03-24T12:46:30.279627+0000","last_fullsized":"2024-03-24T12:46:30.279627+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:46:01.181962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:01.181962+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:22.329684+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"390'32","reported_seq":827,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.819858+0000","last_change":"2024-03-24T12:46:30.819919+0000","last_active":"2024-03-24T12:46:30.819858+0000","last_peered":"2024-03-24T12:46:30.819858+0000","last_clean":"2024-03-24T12:46:30.819858+0000","last_became_active":"2024-03-24T12:45:13.305687+0000","last_became_peered":"2024-03-24T12:45:13.305687+0000","last_unstale":"2024-03-24T12:46:30.819858+0000","last_undegraded":"2024-03-24T12:46:30.819858+0000","last_fullsized":"2024-03-24T12:46:30.819858+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'32","last_scrub_stamp":"2024-03-24T12:46:30.819781+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:46:30.819781+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:55.708488+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":757,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274695+0000","last_change":"2024-03-24T12:45:13.306175+0000","last_active":"2024-03-24T12:46:30.274695+0000","last_peered":"2024-03-24T12:46:30.274695+0000","last_clean":"2024-03-24T12:46:30.274695+0000","last_became_active":"2024-03-24T12:45:13.305719+0000","last_became_peered":"2024-03-24T12:45:13.305719+0000","last_unstale":"2024-03-24T12:46:30.274695+0000","last_undegraded":"2024-03-24T12:46:30.274695+0000","last_fullsized":"2024-03-24T12:46:30.274695+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:26.482979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:44:26.482979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":730,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:31.878693+0000","last_change":"2024-03-24T12:46:31.878736+0000","last_active":"2024-03-24T12:46:31.878693+0000","last_peered":"2024-03-24T12:46:31.878693+0000","last_clean":"2024-03-24T12:46:31.878693+0000","last_became_active":"2024-03-24T12:45:40.244668+0000","last_became_peered":"2024-03-24T12:45:40.244668+0000","last_unstale":"2024-03-24T12:46:31.878693+0000","last_undegraded":"2024-03-24T12:46:31.878693+0000","last_fullsized":"2024-03-24T12:46:31.878693+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:31.878576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:46:31.878576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:59.407040+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":654,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.281402+0000","last_change":"2024-03-24T12:46:19.086662+0000","last_active":"2024-03-24T12:46:30.281402+0000","last_peered":"2024-03-24T12:46:30.281402+0000","last_clean":"2024-03-24T12:46:30.281402+0000","last_became_active":"2024-03-24T12:45:40.240370+0000","last_became_peered":"2024-03-24T12:45:40.240370+0000","last_unstale":"2024-03-24T12:46:30.281402+0000","last_undegraded":"2024-03-24T12:46:30.281402+0000","last_fullsized":"2024-03-24T12:46:30.281402+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:19.086515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:46:19.086515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:45.346160+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"390'21","reported_seq":134,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.276409+0000","last_change":"2024-03-24T12:46:02.430642+0000","last_active":"2024-03-24T12:46:30.276409+0000","last_peered":"2024-03-24T12:46:30.276409+0000","last_clean":"2024-03-24T12:46:30.276409+0000","last_became_active":"2024-03-24T12:45:13.318491+0000","last_became_peered":"2024-03-24T12:45:13.318491+0000","last_unstale":"2024-03-24T12:46:30.276409+0000","last_undegraded":"2024-03-24T12:46:30.276409+0000","last_fullsized":"2024-03-24T12:46:30.276409+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:46:02.430495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:02.430495+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:02.583663+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":739,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268425+0000","last_change":"2024-03-24T12:45:07.883531+0000","last_active":"2024-03-24T12:46:30.268425+0000","last_peered":"2024-03-24T12:46:30.268425+0000","last_clean":"2024-03-24T12:46:30.268425+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:46:30.268425+0000","last_undegraded":"2024-03-24T12:46:30.268425+0000","last_fullsized":"2024-03-24T12:46:30.268425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:07.883424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:07.883424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":723,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.262187+0000","last_change":"2024-03-24T12:46:28.070294+0000","last_active":"2024-03-24T12:46:30.262187+0000","last_peered":"2024-03-24T12:46:30.262187+0000","last_clean":"2024-03-24T12:46:30.262187+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:46:30.262187+0000","last_undegraded":"2024-03-24T12:46:30.262187+0000","last_fullsized":"2024-03-24T12:46:30.262187+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:28.070148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:28.070148+0000","last_clean_scrub_stamp":"2024-03-24T12:46:28.070148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:30.014917+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":710,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279349+0000","last_change":"2024-03-24T12:45:13.306238+0000","last_active":"2024-03-24T12:46:30.279349+0000","last_peered":"2024-03-24T12:46:30.279349+0000","last_clean":"2024-03-24T12:46:30.279349+0000","last_became_active":"2024-03-24T12:45:13.305903+0000","last_became_peered":"2024-03-24T12:45:13.305903+0000","last_unstale":"2024-03-24T12:46:30.279349+0000","last_undegraded":"2024-03-24T12:46:30.279349+0000","last_fullsized":"2024-03-24T12:46:30.279349+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:50.038907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:44:50.038907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"390'27","reported_seq":802,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.262041+0000","last_change":"2024-03-24T12:46:29.088327+0000","last_active":"2024-03-24T12:46:30.262041+0000","last_peered":"2024-03-24T12:46:30.262041+0000","last_clean":"2024-03-24T12:46:30.262041+0000","last_became_active":"2024-03-24T12:45:13.326567+0000","last_became_peered":"2024-03-24T12:45:13.326567+0000","last_unstale":"2024-03-24T12:46:30.262041+0000","last_undegraded":"2024-03-24T12:46:30.262041+0000","last_fullsized":"2024-03-24T12:46:30.262041+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'27","last_scrub_stamp":"2024-03-24T12:46:29.088176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:29.088176+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:50.975223+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":437,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269307+0000","last_change":"2024-03-24T12:46:22.643662+0000","last_active":"2024-03-24T12:46:30.269307+0000","last_peered":"2024-03-24T12:46:30.269307+0000","last_clean":"2024-03-24T12:46:30.269307+0000","last_became_active":"2024-03-24T12:45:40.243344+0000","last_became_peered":"2024-03-24T12:45:40.243344+0000","last_unstale":"2024-03-24T12:46:30.269307+0000","last_undegraded":"2024-03-24T12:46:30.269307+0000","last_fullsized":"2024-03-24T12:46:30.269307+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:46:22.643568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:22.643568+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:23.465204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":749,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268977+0000","last_change":"2024-03-24T12:45:13.307703+0000","last_active":"2024-03-24T12:46:30.268977+0000","last_peered":"2024-03-24T12:46:30.268977+0000","last_clean":"2024-03-24T12:46:30.268977+0000","last_became_active":"2024-03-24T12:45:13.307347+0000","last_became_peered":"2024-03-24T12:45:13.307347+0000","last_unstale":"2024-03-24T12:46:30.268977+0000","last_undegraded":"2024-03-24T12:46:30.268977+0000","last_fullsized":"2024-03-24T12:46:30.268977+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:59.763462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:59.763462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":712,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.276307+0000","last_change":"2024-03-24T12:45:49.068473+0000","last_active":"2024-03-24T12:46:30.276307+0000","last_peered":"2024-03-24T12:46:30.276307+0000","last_clean":"2024-03-24T12:46:30.276307+0000","last_became_active":"2024-03-24T12:45:40.236267+0000","last_became_peered":"2024-03-24T12:45:40.236267+0000","last_unstale":"2024-03-24T12:46:30.276307+0000","last_undegraded":"2024-03-24T12:46:30.276307+0000","last_fullsized":"2024-03-24T12:46:30.276307+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:49.068372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:45:49.068372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:17.365313+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"390'12","reported_seq":756,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.273925+0000","last_change":"2024-03-24T12:45:40.256168+0000","last_active":"2024-03-24T12:46:30.273925+0000","last_peered":"2024-03-24T12:46:30.273925+0000","last_clean":"2024-03-24T12:46:30.273925+0000","last_became_active":"2024-03-24T12:45:40.228239+0000","last_became_peered":"2024-03-24T12:45:40.228239+0000","last_unstale":"2024-03-24T12:46:30.273925+0000","last_undegraded":"2024-03-24T12:46:30.273925+0000","last_fullsized":"2024-03-24T12:46:30.273925+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-24T12:44:31.116754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:44:31.116754+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":728,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.281358+0000","last_change":"2024-03-24T12:45:40.250012+0000","last_active":"2024-03-24T12:46:30.281358+0000","last_peered":"2024-03-24T12:46:30.281358+0000","last_clean":"2024-03-24T12:46:30.281358+0000","last_became_active":"2024-03-24T12:45:40.249671+0000","last_became_peered":"2024-03-24T12:45:40.249671+0000","last_unstale":"2024-03-24T12:46:30.281358+0000","last_undegraded":"2024-03-24T12:46:30.281358+0000","last_fullsized":"2024-03-24T12:46:30.281358+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:01.948746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:01.948746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":114,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274124+0000","last_change":"2024-03-24T12:45:13.309572+0000","last_active":"2024-03-24T12:46:30.274124+0000","last_peered":"2024-03-24T12:46:30.274124+0000","last_clean":"2024-03-24T12:46:30.274124+0000","last_became_active":"2024-03-24T12:45:13.308636+0000","last_became_peered":"2024-03-24T12:45:13.308636+0000","last_unstale":"2024-03-24T12:46:30.274124+0000","last_undegraded":"2024-03-24T12:46:30.274124+0000","last_fullsized":"2024-03-24T12:46:30.274124+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:40.785287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:40.785287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":727,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268758+0000","last_change":"2024-03-24T12:44:57.823516+0000","last_active":"2024-03-24T12:46:30.268758+0000","last_peered":"2024-03-24T12:46:30.268758+0000","last_clean":"2024-03-24T12:46:30.268758+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:46:30.268758+0000","last_undegraded":"2024-03-24T12:46:30.268758+0000","last_fullsized":"2024-03-24T12:46:30.268758+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:57.823399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:57.823399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"390'27","reported_seq":530,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.283943+0000","last_change":"2024-03-24T12:46:01.333390+0000","last_active":"2024-03-24T12:46:30.283943+0000","last_peered":"2024-03-24T12:46:30.283943+0000","last_clean":"2024-03-24T12:46:30.283943+0000","last_became_active":"2024-03-24T12:45:40.252179+0000","last_became_peered":"2024-03-24T12:45:40.252179+0000","last_unstale":"2024-03-24T12:46:30.283943+0000","last_undegraded":"2024-03-24T12:46:30.283943+0000","last_fullsized":"2024-03-24T12:46:30.283943+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:46:01.333187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:01.333187+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:25.456289+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":51,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279542+0000","last_change":"2024-03-24T12:45:52.018351+0000","last_active":"2024-03-24T12:46:30.279542+0000","last_peered":"2024-03-24T12:46:30.279542+0000","last_clean":"2024-03-24T12:46:30.279542+0000","last_became_active":"2024-03-24T12:45:40.256265+0000","last_became_peered":"2024-03-24T12:45:40.256265+0000","last_unstale":"2024-03-24T12:46:30.279542+0000","last_undegraded":"2024-03-24T12:46:30.279542+0000","last_fullsized":"2024-03-24T12:46:30.279542+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:52.018268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:45:52.018268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:58.067532+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":733,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.280318+0000","last_change":"2024-03-24T12:46:13.115943+0000","last_active":"2024-03-24T12:46:30.280318+0000","last_peered":"2024-03-24T12:46:30.280318+0000","last_clean":"2024-03-24T12:46:30.280318+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:46:30.280318+0000","last_undegraded":"2024-03-24T12:46:30.280318+0000","last_fullsized":"2024-03-24T12:46:30.280318+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:13.115801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:13.115801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:24.278595+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":626,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274846+0000","last_change":"2024-03-24T12:44:54.658333+0000","last_active":"2024-03-24T12:46:30.274846+0000","last_peered":"2024-03-24T12:46:30.274846+0000","last_clean":"2024-03-24T12:46:30.274846+0000","last_became_active":"2024-03-24T12:44:48.870417+0000","last_became_peered":"2024-03-24T12:44:48.870417+0000","last_unstale":"2024-03-24T12:46:30.274846+0000","last_undegraded":"2024-03-24T12:46:30.274846+0000","last_fullsized":"2024-03-24T12:46:30.274846+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:54.658192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:54.658192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"390'30","reported_seq":826,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.280034+0000","last_change":"2024-03-24T12:46:11.145946+0000","last_active":"2024-03-24T12:46:30.280034+0000","last_peered":"2024-03-24T12:46:30.280034+0000","last_clean":"2024-03-24T12:46:30.280034+0000","last_became_active":"2024-03-24T12:45:40.255480+0000","last_became_peered":"2024-03-24T12:45:40.255480+0000","last_unstale":"2024-03-24T12:46:30.280034+0000","last_undegraded":"2024-03-24T12:46:30.280034+0000","last_fullsized":"2024-03-24T12:46:30.280034+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:46:11.145131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:11.145131+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:19.639438+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"392'2","reported_seq":757,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.276016+0000","last_change":"2024-03-24T12:46:18.735937+0000","last_active":"2024-03-24T12:46:30.276016+0000","last_peered":"2024-03-24T12:46:30.276016+0000","last_clean":"2024-03-24T12:46:30.276016+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:46:30.276016+0000","last_undegraded":"2024-03-24T12:46:30.276016+0000","last_fullsized":"2024-03-24T12:46:30.276016+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'2","last_scrub_stamp":"2024-03-24T12:46:18.735806+0000","last_deep_scrub":"392'2","last_deep_scrub_stamp":"2024-03-24T12:46:18.735806+0000","last_clean_scrub_stamp":"2024-03-24T12:46:18.735806+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:36.258237+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"370'12","reported_seq":182,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275399+0000","last_change":"2024-03-24T12:45:13.326830+0000","last_active":"2024-03-24T12:46:30.275399+0000","last_peered":"2024-03-24T12:46:30.275399+0000","last_clean":"2024-03-24T12:46:30.275399+0000","last_became_active":"2024-03-24T12:45:13.310827+0000","last_became_peered":"2024-03-24T12:45:13.310827+0000","last_unstale":"2024-03-24T12:46:30.275399+0000","last_undegraded":"2024-03-24T12:46:30.275399+0000","last_fullsized":"2024-03-24T12:46:30.275399+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-24T12:44:42.835854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:42.835854+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":719,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268993+0000","last_change":"2024-03-24T12:45:40.232224+0000","last_active":"2024-03-24T12:46:30.268993+0000","last_peered":"2024-03-24T12:46:30.268993+0000","last_clean":"2024-03-24T12:46:30.268993+0000","last_became_active":"2024-03-24T12:45:40.227973+0000","last_became_peered":"2024-03-24T12:45:40.227973+0000","last_unstale":"2024-03-24T12:46:30.268993+0000","last_undegraded":"2024-03-24T12:46:30.268993+0000","last_fullsized":"2024-03-24T12:46:30.268993+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:44:51.808307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:51.808307+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"391'24","reported_seq":416,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.281648+0000","last_change":"2024-03-24T12:45:56.341335+0000","last_active":"2024-03-24T12:46:30.281648+0000","last_peered":"2024-03-24T12:46:30.281648+0000","last_clean":"2024-03-24T12:46:30.281648+0000","last_became_active":"2024-03-24T12:45:40.253908+0000","last_became_peered":"2024-03-24T12:45:40.253908+0000","last_unstale":"2024-03-24T12:46:30.281648+0000","last_undegraded":"2024-03-24T12:46:30.281648+0000","last_fullsized":"2024-03-24T12:46:30.281648+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-24T12:45:56.341162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:56.341162+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:57.022327+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":739,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274703+0000","last_change":"2024-03-24T12:45:56.529812+0000","last_active":"2024-03-24T12:46:30.274703+0000","last_peered":"2024-03-24T12:46:30.274703+0000","last_clean":"2024-03-24T12:46:30.274703+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:46:30.274703+0000","last_undegraded":"2024-03-24T12:46:30.274703+0000","last_fullsized":"2024-03-24T12:46:30.274703+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:56.529721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:56.529721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:56.982975+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":731,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274902+0000","last_change":"2024-03-24T12:44:39.546725+0000","last_active":"2024-03-24T12:46:30.274902+0000","last_peered":"2024-03-24T12:46:30.274902+0000","last_clean":"2024-03-24T12:46:30.274902+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:46:30.274902+0000","last_undegraded":"2024-03-24T12:46:30.274902+0000","last_fullsized":"2024-03-24T12:46:30.274902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:39.546649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:44:39.546649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":737,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.263210+0000","last_change":"2024-03-24T12:46:02.238230+0000","last_active":"2024-03-24T12:46:30.263210+0000","last_peered":"2024-03-24T12:46:30.263210+0000","last_clean":"2024-03-24T12:46:30.263210+0000","last_became_active":"2024-03-24T12:45:13.313299+0000","last_became_peered":"2024-03-24T12:45:13.313299+0000","last_unstale":"2024-03-24T12:46:30.263210+0000","last_undegraded":"2024-03-24T12:46:30.263210+0000","last_fullsized":"2024-03-24T12:46:30.263210+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:02.237746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:02.237746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:23.102747+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"390'39","reported_seq":826,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.277522+0000","last_change":"2024-03-24T12:45:49.393129+0000","last_active":"2024-03-24T12:46:30.277522+0000","last_peered":"2024-03-24T12:46:30.277522+0000","last_clean":"2024-03-24T12:46:30.277522+0000","last_became_active":"2024-03-24T12:45:40.245335+0000","last_became_peered":"2024-03-24T12:45:40.245335+0000","last_unstale":"2024-03-24T12:46:30.277522+0000","last_undegraded":"2024-03-24T12:46:30.277522+0000","last_fullsized":"2024-03-24T12:46:30.277522+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-24T12:45:49.393029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:49.393029+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:15.980341+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":739,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279698+0000","last_change":"2024-03-24T12:45:56.143367+0000","last_active":"2024-03-24T12:46:30.279698+0000","last_peered":"2024-03-24T12:46:30.279698+0000","last_clean":"2024-03-24T12:46:30.279698+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:46:30.279698+0000","last_undegraded":"2024-03-24T12:46:30.279698+0000","last_fullsized":"2024-03-24T12:46:30.279698+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:56.143236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:45:56.143236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:08.519482+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":755,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.282087+0000","last_change":"2024-03-24T12:46:15.640190+0000","last_active":"2024-03-24T12:46:30.282087+0000","last_peered":"2024-03-24T12:46:30.282087+0000","last_clean":"2024-03-24T12:46:30.282087+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:46:30.282087+0000","last_undegraded":"2024-03-24T12:46:30.282087+0000","last_fullsized":"2024-03-24T12:46:30.282087+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:15.638516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:15.638516+0000","last_clean_scrub_stamp":"2024-03-24T12:46:15.638516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:34.548383+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":760,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275898+0000","last_change":"2024-03-24T12:46:26.731340+0000","last_active":"2024-03-24T12:46:30.275898+0000","last_peered":"2024-03-24T12:46:30.275898+0000","last_clean":"2024-03-24T12:46:30.275898+0000","last_became_active":"2024-03-24T12:45:40.234852+0000","last_became_peered":"2024-03-24T12:45:40.234852+0000","last_unstale":"2024-03-24T12:46:30.275898+0000","last_undegraded":"2024-03-24T12:46:30.275898+0000","last_fullsized":"2024-03-24T12:46:30.275898+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:26.731239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:26.731239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:44.116348+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"390'9","reported_seq":751,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274665+0000","last_change":"2024-03-24T12:45:52.553551+0000","last_active":"2024-03-24T12:46:30.274665+0000","last_peered":"2024-03-24T12:46:30.274665+0000","last_clean":"2024-03-24T12:46:30.274665+0000","last_became_active":"2024-03-24T12:26:05.322480+0000","last_became_peered":"2024-03-24T12:26:05.322480+0000","last_unstale":"2024-03-24T12:46:30.274665+0000","last_undegraded":"2024-03-24T12:46:30.274665+0000","last_fullsized":"2024-03-24T12:46:30.274665+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-24T12:45:52.553430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:52.553430+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:22.183036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":413,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274639+0000","last_change":"2024-03-24T12:45:13.260482+0000","last_active":"2024-03-24T12:46:30.274639+0000","last_peered":"2024-03-24T12:46:30.274639+0000","last_clean":"2024-03-24T12:46:30.274639+0000","last_became_active":"2024-03-24T12:45:13.259915+0000","last_became_peered":"2024-03-24T12:45:13.259915+0000","last_unstale":"2024-03-24T12:46:30.274639+0000","last_undegraded":"2024-03-24T12:46:30.274639+0000","last_fullsized":"2024-03-24T12:46:30.274639+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:55.611389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:44:55.611389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":731,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268660+0000","last_change":"2024-03-24T12:45:06.855502+0000","last_active":"2024-03-24T12:46:30.268660+0000","last_peered":"2024-03-24T12:46:30.268660+0000","last_clean":"2024-03-24T12:46:30.268660+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:46:30.268660+0000","last_undegraded":"2024-03-24T12:46:30.268660+0000","last_fullsized":"2024-03-24T12:46:30.268660+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:06.855388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:45:06.855388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":421,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:33.991002+0000","last_change":"2024-03-24T12:46:33.991040+0000","last_active":"2024-03-24T12:46:33.991002+0000","last_peered":"2024-03-24T12:46:33.991002+0000","last_clean":"2024-03-24T12:46:33.991002+0000","last_became_active":"2024-03-24T12:45:40.252467+0000","last_became_peered":"2024-03-24T12:45:40.252467+0000","last_unstale":"2024-03-24T12:46:33.991002+0000","last_undegraded":"2024-03-24T12:46:33.991002+0000","last_fullsized":"2024-03-24T12:46:33.991002+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:33.990925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:46:33.990925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:46.948320+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"390'21","reported_seq":769,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274858+0000","last_change":"2024-03-24T12:46:29.784711+0000","last_active":"2024-03-24T12:46:30.274858+0000","last_peered":"2024-03-24T12:46:30.274858+0000","last_clean":"2024-03-24T12:46:30.274858+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:46:30.274858+0000","last_undegraded":"2024-03-24T12:46:30.274858+0000","last_fullsized":"2024-03-24T12:46:30.274858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:46:29.784586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:29.784586+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:32.849608+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":445,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.275139+0000","last_change":"2024-03-24T12:45:40.249900+0000","last_active":"2024-03-24T12:46:30.275139+0000","last_peered":"2024-03-24T12:46:30.275139+0000","last_clean":"2024-03-24T12:46:30.275139+0000","last_became_active":"2024-03-24T12:45:40.249430+0000","last_became_peered":"2024-03-24T12:45:40.249430+0000","last_unstale":"2024-03-24T12:46:30.275139+0000","last_undegraded":"2024-03-24T12:46:30.275139+0000","last_fullsized":"2024-03-24T12:46:30.275139+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:35.453832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:44:35.453832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":440,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.276549+0000","last_change":"2024-03-24T12:46:18.200110+0000","last_active":"2024-03-24T12:46:30.276549+0000","last_peered":"2024-03-24T12:46:30.276549+0000","last_clean":"2024-03-24T12:46:30.276549+0000","last_became_active":"2024-03-24T12:45:40.250171+0000","last_became_peered":"2024-03-24T12:45:40.250171+0000","last_unstale":"2024-03-24T12:46:30.276549+0000","last_undegraded":"2024-03-24T12:46:30.276549+0000","last_fullsized":"2024-03-24T12:46:30.276549+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:18.199684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:18.199684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:44.538094+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":735,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.279641+0000","last_change":"2024-03-24T12:46:06.136390+0000","last_active":"2024-03-24T12:46:30.279641+0000","last_peered":"2024-03-24T12:46:30.279641+0000","last_clean":"2024-03-24T12:46:30.279641+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:46:30.279641+0000","last_undegraded":"2024-03-24T12:46:30.279641+0000","last_fullsized":"2024-03-24T12:46:30.279641+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:06.135893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:46:06.135893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:18.153539+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"391'46","reported_seq":439,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.270974+0000","last_change":"2024-03-24T12:45:40.299662+0000","last_active":"2024-03-24T12:46:30.270974+0000","last_peered":"2024-03-24T12:46:30.270974+0000","last_clean":"2024-03-24T12:46:30.270974+0000","last_became_active":"2024-03-24T12:45:40.247792+0000","last_became_peered":"2024-03-24T12:45:40.247792+0000","last_unstale":"2024-03-24T12:46:30.270974+0000","last_undegraded":"2024-03-24T12:46:30.270974+0000","last_fullsized":"2024-03-24T12:46:30.270974+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-24T12:45:07.975149+0000","last_deep_scrub":"207'34","last_deep_scrub_stamp":"2024-03-24T12:45:07.975149+0000","last_clean_scrub_stamp":"2024-03-24T12:45:07.975149+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":733,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.269236+0000","last_change":"2024-03-24T12:46:13.616359+0000","last_active":"2024-03-24T12:46:30.269236+0000","last_peered":"2024-03-24T12:46:30.269236+0000","last_clean":"2024-03-24T12:46:30.269236+0000","last_became_active":"2024-03-24T12:45:40.244546+0000","last_became_peered":"2024-03-24T12:45:40.244546+0000","last_unstale":"2024-03-24T12:46:30.269236+0000","last_undegraded":"2024-03-24T12:46:30.269236+0000","last_fullsized":"2024-03-24T12:46:30.269236+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:13.616208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:13.616208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:26.870863+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":749,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274242+0000","last_change":"2024-03-24T12:45:13.311076+0000","last_active":"2024-03-24T12:46:30.274242+0000","last_peered":"2024-03-24T12:46:30.274242+0000","last_clean":"2024-03-24T12:46:30.274242+0000","last_became_active":"2024-03-24T12:45:13.310787+0000","last_became_peered":"2024-03-24T12:45:13.310787+0000","last_unstale":"2024-03-24T12:46:30.274242+0000","last_undegraded":"2024-03-24T12:46:30.274242+0000","last_fullsized":"2024-03-24T12:46:30.274242+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:46.508221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:17.828354+0000","last_clean_scrub_stamp":"2024-03-24T12:44:46.508221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":711,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268349+0000","last_change":"2024-03-24T12:45:47.024216+0000","last_active":"2024-03-24T12:46:30.268349+0000","last_peered":"2024-03-24T12:46:30.268349+0000","last_clean":"2024-03-24T12:46:30.268349+0000","last_became_active":"2024-03-24T12:45:40.243426+0000","last_became_peered":"2024-03-24T12:45:40.243426+0000","last_unstale":"2024-03-24T12:46:30.268349+0000","last_undegraded":"2024-03-24T12:46:30.268349+0000","last_fullsized":"2024-03-24T12:46:30.268349+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:47.024115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:45:47.024115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:46:50.069837+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"390'21","reported_seq":833,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274102+0000","last_change":"2024-03-24T12:45:40.261157+0000","last_active":"2024-03-24T12:46:30.274102+0000","last_peered":"2024-03-24T12:46:30.274102+0000","last_clean":"2024-03-24T12:46:30.274102+0000","last_became_active":"2024-03-24T12:45:40.229470+0000","last_became_peered":"2024-03-24T12:45:40.229470+0000","last_unstale":"2024-03-24T12:46:30.274102+0000","last_undegraded":"2024-03-24T12:46:30.274102+0000","last_fullsized":"2024-03-24T12:46:30.274102+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:44:43.532996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:44:43.532996+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":723,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.281137+0000","last_change":"2024-03-24T12:45:13.322851+0000","last_active":"2024-03-24T12:46:30.281137+0000","last_peered":"2024-03-24T12:46:30.281137+0000","last_clean":"2024-03-24T12:46:30.281137+0000","last_became_active":"2024-03-24T12:45:13.322473+0000","last_became_peered":"2024-03-24T12:45:13.322473+0000","last_unstale":"2024-03-24T12:46:30.281137+0000","last_undegraded":"2024-03-24T12:46:30.281137+0000","last_fullsized":"2024-03-24T12:46:30.281137+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:05.825276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:05.825276+0000","last_clean_scrub_stamp":"2024-03-24T12:45:05.825276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":632,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.268508+0000","last_change":"2024-03-24T12:44:53.810778+0000","last_active":"2024-03-24T12:46:30.268508+0000","last_peered":"2024-03-24T12:46:30.268508+0000","last_clean":"2024-03-24T12:46:30.268508+0000","last_became_active":"2024-03-24T12:44:48.869617+0000","last_became_peered":"2024-03-24T12:44:48.869617+0000","last_unstale":"2024-03-24T12:46:30.268508+0000","last_undegraded":"2024-03-24T12:46:30.268508+0000","last_fullsized":"2024-03-24T12:46:30.268508+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:53.810528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:53.810528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":712,"reported_epoch":395,"state":"active+clean","last_fresh":"2024-03-24T12:46:30.274468+0000","last_change":"2024-03-24T12:45:03.590860+0000","last_active":"2024-03-24T12:46:30.274468+0000","last_peered":"2024-03-24T12:46:30.274468+0000","last_clean":"2024-03-24T12:46:30.274468+0000","last_became_active":"2024-03-24T12:44:48.860436+0000","last_became_peered":"2024-03-24T12:44:48.860436+0000","last_unstale":"2024-03-24T12:46:30.274468+0000","last_undegraded":"2024-03-24T12:46:30.274468+0000","last_fullsized":"2024-03-24T12:46:30.274468+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:03.590750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:45:03.590750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":3346857984,"num_objects":51069,"num_object_clones":0,"num_object_copies":153207,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51069,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":51069,"num_write_kb":3268416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10145300480,"data_stored":10145300480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":50169,"ondisk_log_size":50169,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1221,"num_read_kb":1216,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1671,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":6519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1546,"num_read_kb":1303,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":243,"num_bytes_recovered":4846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1047,"ondisk_log_size":1047,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038010,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2299796,"kb_used_data":1600844,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":91445356,"statfs":{"total":95995035648,"available":93640044544,"internally_reserved":0,"allocated":1639264256,"data_stored":1638156762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":715717066},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":219043332352,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575131,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3514884,"kb_used_data":2753468,"kb_used_omap":1,"kb_used_meta":761406,"kb_avail":90230268,"statfs":{"total":95995035648,"available":92395794432,"internally_reserved":0,"allocated":2819551232,"data_stored":2818396464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":779680197},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,4,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738655,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2715120,"kb_used_data":1813480,"kb_used_omap":1,"kb_used_meta":901630,"kb_avail":91030032,"statfs":{"total":95995035648,"available":93214752768,"internally_reserved":0,"allocated":1857003520,"data_stored":1855868793,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":923269573},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411605,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2857744,"kb_used_data":2064392,"kb_used_omap":1,"kb_used_meta":793342,"kb_avail":90887408,"statfs":{"total":95995035648,"available":93068705792,"internally_reserved":0,"allocated":2113937408,"data_stored":2112835092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":812382661},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117262,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":943700,"kb_used_data":204492,"kb_used_omap":1,"kb_used_meta":739198,"kb_avail":92801452,"statfs":{"total":95995035648,"available":95028686848,"internally_reserved":0,"allocated":209399808,"data_stored":208355553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":756939207},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921034,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2023900,"kb_used_data":1374928,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":91721252,"statfs":{"total":95995035648,"available":93922562048,"internally_reserved":0,"allocated":1407926272,"data_stored":1406814704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":664533445},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":2998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1876688896,"data_stored":1876688896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2826043392,"data_stored":2826043392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2150170624,"data_stored":2150170624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":211419136,"data_stored":211419136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1435435008,"data_stored":1435435008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1645543424,"data_stored":1645543424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:46:37.044 INFO:tasks.thrashosds.thrasher:Setting 5.f to [3, 4, 7] 2024-03-24T12:46:37.044 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '5.f', '3', '4', '7'] 2024-03-24T12:46:37.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pg-upmap 5.f 3 4 7 2024-03-24T12:46:37.064 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:37.090 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:37.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:37.195 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:37.221 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:37.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:37.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:37.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:37.387 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:37.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:37.489 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:37.514 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:37.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:37.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:37.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:37 smithi029 ceph-mon[29919]: 6.10 scrub ok 2024-03-24T12:46:37.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:37 smithi029 ceph-mon[29919]: pgmap v911: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 834 op/s 2024-03-24T12:46:37.534 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:37 smithi029 ceph-mon[29919]: 6.7 deep-scrub starts 2024-03-24T12:46:37.535 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:37 smithi029 ceph-mon[41500]: 6.10 scrub ok 2024-03-24T12:46:37.535 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:37 smithi029 ceph-mon[41500]: pgmap v911: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 834 op/s 2024-03-24T12:46:37.535 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:37 smithi029 ceph-mon[41500]: 6.7 deep-scrub starts 2024-03-24T12:46:37.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:37 smithi190 ceph-mon[33748]: 6.10 scrub ok 2024-03-24T12:46:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:37 smithi190 ceph-mon[33748]: pgmap v911: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 834 op/s 2024-03-24T12:46:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:37 smithi190 ceph-mon[33748]: 6.7 deep-scrub starts 2024-03-24T12:46:37.644 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:37.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:37.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:37.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:37.792 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:37.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:37.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:37.894 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:37.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 42 16 56364 56348 83.8342 64.8125 0.0117195 0.0119235 2024-03-24T12:46:37.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:46:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:46:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:46:37.935 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:37.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:37.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:38.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:38.082 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:38.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:38.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:38.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:38.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:38.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:38.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:38.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:38.354 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:46:38.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:38.472 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:38.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:38.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:38 smithi029 ceph-mon[41500]: 6.7 deep-scrub ok 2024-03-24T12:46:38.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:38 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/90838090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:46:38.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:38 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:46:38.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:38 smithi029 ceph-mon[41500]: 3.1d scrub starts 2024-03-24T12:46:38.563 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:38 smithi029 ceph-mon[41500]: 3.1d scrub ok 2024-03-24T12:46:38.563 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:38 smithi029 ceph-mon[29919]: 6.7 deep-scrub ok 2024-03-24T12:46:38.563 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:38 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/90838090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:46:38.563 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:38 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:46:38.564 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:38 smithi029 ceph-mon[29919]: 3.1d scrub starts 2024-03-24T12:46:38.564 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:38 smithi029 ceph-mon[29919]: 3.1d scrub ok 2024-03-24T12:46:38.574 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:38.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:38.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:38.632 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:46:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:46:38.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:38 smithi190 ceph-mon[33748]: 6.7 deep-scrub ok 2024-03-24T12:46:38.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:38 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/90838090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:46:38.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:38 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:46:38.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:38 smithi190 ceph-mon[33748]: 3.1d scrub starts 2024-03-24T12:46:38.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:38 smithi190 ceph-mon[33748]: 3.1d scrub ok 2024-03-24T12:46:38.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:38.751 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:38.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:38.852 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:38.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:38.880 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:38.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:38.916 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 43 16 57579 57563 83.6504 75.9375 0.0131811 0.0119486 2024-03-24T12:46:38.948 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:46:38.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:39.028 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:39.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:39.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:39.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:39.169 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:39.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:39.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:39.271 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:39.298 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:39.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:39.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:39.400 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:39.446 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:39.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:39.531 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:46:39.547 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:39.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:39.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:39.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:39 smithi190 ceph-mon[33748]: 5.15 scrub starts 2024-03-24T12:46:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:39 smithi190 ceph-mon[33748]: 5.15 scrub ok 2024-03-24T12:46:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:39 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:46:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:39 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:46:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:39 smithi190 ceph-mon[33748]: osdmap e397: 8 total, 7 up, 6 in 2024-03-24T12:46:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:39.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:39 smithi190 ceph-mon[33748]: pgmap v913: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:46:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[29919]: 5.15 scrub starts 2024-03-24T12:46:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[29919]: 5.15 scrub ok 2024-03-24T12:46:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:46:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:46:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[29919]: osdmap e397: 8 total, 7 up, 6 in 2024-03-24T12:46:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[29919]: pgmap v913: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:46:39.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[41500]: 5.15 scrub starts 2024-03-24T12:46:39.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[41500]: 5.15 scrub ok 2024-03-24T12:46:39.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:46:39.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:46:39.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[41500]: osdmap e397: 8 total, 7 up, 6 in 2024-03-24T12:46:39.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:39.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:39 smithi029 ceph-mon[41500]: pgmap v913: 158 pgs: 158 active+clean; 3.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-24T12:46:39.677 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:39.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:39.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:39.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:39.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:39.872 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:39.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:39.917 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 44 16 58857 58841 83.563 79.875 0.00634783 0.0119615 2024-03-24T12:46:39.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:40.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:40.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:40.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:40.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:40.148 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:40.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:40.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:40.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:40.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:40.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:40.397 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:40.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:40.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:40.425 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:40.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:40.554 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:40.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:40.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:40.656 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:40.683 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:40.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:40.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:40.821 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:46:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:46:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-03-24T12:46:40.821 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-03-24T12:46:40.821 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:46:40.826 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:40.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:40.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:40.917 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 45 16 60089 60073 83.417 77 0.0153227 0.0119823 2024-03-24T12:46:40.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:40.955 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:40.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:40.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:41.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:41.098 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:41.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:41.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:41.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:41.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:41.242 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:41.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:41.344 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:41.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:41.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:41.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:41.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:41 smithi029 ceph-mon[29919]: osdmap e398: 8 total, 7 up, 6 in 2024-03-24T12:46:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:41 smithi029 ceph-mon[41500]: osdmap e398: 8 total, 7 up, 6 in 2024-03-24T12:46:41.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:41.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:41.494 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:41.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:41.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:46:41.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:41 smithi190 ceph-mon[33748]: osdmap e398: 8 total, 7 up, 6 in 2024-03-24T12:46:41.684 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:41.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:41.786 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:41.838 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:41.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:41.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:41.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 46 16 61114 61098 82.9957 64.0625 0.0124026 0.0120432 2024-03-24T12:46:41.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:41.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:41.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:41.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:42.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:42.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:42.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:42.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:42.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:42.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:42.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:42.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:42.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:42.366 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:42.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:42.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:42 smithi029 ceph-mon[29919]: pgmap v915: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T12:46:42.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:42 smithi029 ceph-mon[41500]: pgmap v915: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T12:46:42.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:42.512 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:42.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:42.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:42.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:42 smithi190 ceph-mon[33748]: pgmap v915: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 853 B/s rd, 69 MiB/s wr, 1.11k op/s 2024-03-24T12:46:42.652 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:42.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:42.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:42.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:42.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T12:46:42.781 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:42.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:42.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:42.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 47 16 62376 62360 82.9079 78.875 0.0132774 0.0120552 2024-03-24T12:46:42.941 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:42.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:42.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:43.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:43.111 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:43.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:43.215 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:43.269 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:43.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:43.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:43.366 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:43 smithi029 ceph-mon[41500]: pgmap v916: 158 pgs: 158 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-24T12:46:43.366 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:43 smithi029 ceph-mon[29919]: pgmap v916: 158 pgs: 158 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-24T12:46:43.371 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:43.397 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:43.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:43.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:43.499 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:43.548 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:43.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:43 smithi190 ceph-mon[33748]: pgmap v916: 158 pgs: 158 active+clean; 3.6 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-24T12:46:43.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:43.689 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:43.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:43.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:43.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:43.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 48 16 63560 63544 82.7222 74 0.00621076 0.0120835 2024-03-24T12:46:43.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:43.980 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:43.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:43.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:44.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:44.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:44.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:44.268 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:44.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:44.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:44.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:44.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:44.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:44.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:44.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:44.547 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:44.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:44.649 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:44.677 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:44.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:44.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:44.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:44.808 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:44.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:44.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:44.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:44.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 49 16 64787 64771 82.5987 76.6875 0.0181777 0.0121007 2024-03-24T12:46:44.956 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:44.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:44.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:45.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:45.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:45.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:45.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:45.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:45.231 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:45.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:45.332 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:45.359 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:45.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:45.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:45.461 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:45.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:45.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:46:45.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:45.634 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:45.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:45.735 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:45.759 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:45.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:45.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:45.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:45.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:45.902 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:45.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:45.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 50 16 65882 65866 82.3154 68.4375 0.0212931 0.0121427 2024-03-24T12:46:45.930 INFO:teuthology.orchestra.run.smithi029.stderr:set 5.f pg_upmap mapping to [3,4,7] 2024-03-24T12:46:46.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:46.049 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:46.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:46.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:46.139 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:45 smithi029 ceph-mon[29919]: pgmap v917: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:46:46.140 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:45 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/516801957' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [3, 4, 7]}]: dispatch 2024-03-24T12:46:46.140 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:45 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3608494698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:46:46.140 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:45 smithi029 ceph-mon[41500]: pgmap v917: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:46:46.140 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:45 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/516801957' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [3, 4, 7]}]: dispatch 2024-03-24T12:46:46.140 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:45 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3608494698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:46:46.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:46:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:46:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:46:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:46:46.187 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:46:46.188 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:46:46.188 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:46:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:46:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:46:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:46:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:46:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:46:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:46:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:46:46.188 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:46:46.192 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:46.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:46.294 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:46.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:46.322 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:46.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:46.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:45 smithi190 ceph-mon[33748]: pgmap v917: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-24T12:46:46.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:45 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/516801957' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [3, 4, 7]}]: dispatch 2024-03-24T12:46:46.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:45 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3608494698' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:46:46.424 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:46.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:46.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:46.554 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:46.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:46.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:46.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:46.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:46.717 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:46:46.719 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:46.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:46.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:46.821 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:46.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:46.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:46.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 51 16 66964 66948 82.0273 67.625 0.00850578 0.012186 2024-03-24T12:46:46.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:46.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:46.987 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:46.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:47.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:47.130 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:47.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:47.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:47.274 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:47.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:47.376 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:46 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/516801957' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [3, 4, 7]}]': finished 2024-03-24T12:46:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:46 smithi190 ceph-mon[33748]: osdmap e399: 8 total, 7 up, 6 in 2024-03-24T12:46:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [1, 7]}]: dispatch 2024-03-24T12:46:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]: dispatch 2024-03-24T12:46:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:46 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [1, 7]}]': finished 2024-03-24T12:46:47.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:46 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/516801957' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [3, 4, 7]}]': finished 2024-03-24T12:46:47.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:46 smithi029 ceph-mon[29919]: osdmap e399: 8 total, 7 up, 6 in 2024-03-24T12:46:47.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [1, 7]}]: dispatch 2024-03-24T12:46:47.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]: dispatch 2024-03-24T12:46:47.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:46 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [1, 7]}]': finished 2024-03-24T12:46:47.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:46 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/516801957' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "5.f", "id": [3, 4, 7]}]': finished 2024-03-24T12:46:47.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:46 smithi029 ceph-mon[41500]: osdmap e399: 8 total, 7 up, 6 in 2024-03-24T12:46:47.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [1, 7]}]: dispatch 2024-03-24T12:46:47.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]: dispatch 2024-03-24T12:46:47.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:46 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [1, 7]}]': finished 2024-03-24T12:46:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:47.420 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:47.420 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:47.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:47.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:46:47.569 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:46:47.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:47.705 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:47.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:47.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:47.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:47.848 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:46:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:46:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:46:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:47.850 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:47.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:47.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 52 16 68190 68174 81.9232 76.625 0.0122861 0.0122001 2024-03-24T12:46:47.951 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:47.996 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:47.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:47.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:48.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:48.137 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:48.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:48.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:47 smithi029 ceph-mon[41500]: pgmap v919: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 59 MiB/s wr, 937 op/s 2024-03-24T12:46:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]': finished 2024-03-24T12:46:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:47 smithi029 ceph-mon[41500]: osdmap e400: 8 total, 7 up, 6 in 2024-03-24T12:46:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:47 smithi029 ceph-mon[41500]: osdmap e401: 8 total, 7 up, 6 in 2024-03-24T12:46:48.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:47 smithi029 ceph-mon[29919]: pgmap v919: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 59 MiB/s wr, 937 op/s 2024-03-24T12:46:48.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]': finished 2024-03-24T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:47 smithi029 ceph-mon[29919]: osdmap e400: 8 total, 7 up, 6 in 2024-03-24T12:46:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:47 smithi029 ceph-mon[29919]: osdmap e401: 8 total, 7 up, 6 in 2024-03-24T12:46:48.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:48.278 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:48.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:48.279 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:48.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:47 smithi190 ceph-mon[33748]: pgmap v919: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 59 MiB/s wr, 937 op/s 2024-03-24T12:46:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 7]}]': finished 2024-03-24T12:46:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:47 smithi190 ceph-mon[33748]: osdmap e400: 8 total, 7 up, 6 in 2024-03-24T12:46:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:47 smithi190 ceph-mon[33748]: osdmap e401: 8 total, 7 up, 6 in 2024-03-24T12:46:48.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:48.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:48.408 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:48.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:48.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:48.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:48.631 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:46:48 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:46:48.407+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:46:48.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:48.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:48.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:48.805 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:48.834 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:48.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:48.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:48.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:46:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:46:48.918 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 53 16 69428 69412 81.8373 77.375 0.00736555 0.0122137 2024-03-24T12:46:48.937 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:48.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:48.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:48.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:49.070 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:49.098 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:49.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:49.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:49.243 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:49.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:49.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:49.345 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:49.372 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:49.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:49.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:49.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:49.518 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:49.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:49.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:46:49.615 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:46:49.615 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:46:49.615 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:46:49.615 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:49.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:49.616 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[29919]: pgmap v922: 158 pgs: 1 active+recovering, 1 peering, 156 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.35k op/s; 3/202425 objects misplaced (0.001%); 18 B/s, 0 objects/s recovering 2024-03-24T12:46:49.616 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:46:49.616 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:46:49.616 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:46:49.616 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:49.616 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:49.616 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:49 smithi029 ceph-mon[41500]: pgmap v922: 158 pgs: 1 active+recovering, 1 peering, 156 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.35k op/s; 3/202425 objects misplaced (0.001%); 18 B/s, 0 objects/s recovering 2024-03-24T12:46:49.620 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:49.647 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:49.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:49.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:49.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:49.797 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:49.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:49.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:49 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:46:49.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:49 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:46:49.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:46:49.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:46:49.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:49.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:49 smithi190 ceph-mon[33748]: pgmap v922: 158 pgs: 1 active+recovering, 1 peering, 156 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 852 B/s rd, 84 MiB/s wr, 1.35k op/s; 3/202425 objects misplaced (0.001%); 18 B/s, 0 objects/s recovering 2024-03-24T12:46:49.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:49.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 54 16 70648 70632 81.7326 76.25 0.0169465 0.0122296 2024-03-24T12:46:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:49.953 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:49.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:50.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:50.099 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:50.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:50.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:50.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:50.252 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:50.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:50.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:50.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:50.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:50.379 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:50.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:50.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:50.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:50.524 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:50.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:50.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:50.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:50.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:50.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:50.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:50.784 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:50.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:50.886 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:50.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:50.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:50.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:50.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 55 16 71633 71617 81.3659 61.5625 0.0118489 0.0122848 2024-03-24T12:46:51.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:51.054 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:51.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:51.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:51.195 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:51.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:51.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:51.297 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:51.323 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:51.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:51.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:51 smithi029 ceph-mon[29919]: osdmap e402: 8 total, 7 up, 6 in 2024-03-24T12:46:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:51 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:46:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:51 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:46:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:51 smithi029 ceph-mon[41500]: osdmap e402: 8 total, 7 up, 6 in 2024-03-24T12:46:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:51 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:46:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:51 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:46:51.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:51.448 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:51.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:51.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:51.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:51.591 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:51.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:51 smithi190 ceph-mon[33748]: osdmap e402: 8 total, 7 up, 6 in 2024-03-24T12:46:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:51 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:46:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:51 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:46:51.692 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:51.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:51.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:51.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:51.867 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:51.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:51.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:51.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 56 16 72726 72710 81.1327 68.3125 0.0130627 0.0123203 2024-03-24T12:46:51.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:52.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:52.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:52.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:52.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:52.130 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:52.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:52.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:52.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:52.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:52.357 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:52.381 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:52.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:52.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:52.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:52.520 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:52.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:52.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:52 smithi190 ceph-mon[33748]: pgmap v924: 158 pgs: 1 active+recovering, 1 peering, 156 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 3/206892 objects misplaced (0.001%); 22 B/s, 0 objects/s recovering 2024-03-24T12:46:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:52 smithi190 ceph-mon[33748]: osdmap e403: 8 total, 7 up, 6 in 2024-03-24T12:46:52.648 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:52.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:52.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:52 smithi029 ceph-mon[41500]: pgmap v924: 158 pgs: 1 active+recovering, 1 peering, 156 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 3/206892 objects misplaced (0.001%); 22 B/s, 0 objects/s recovering 2024-03-24T12:46:52.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:52 smithi029 ceph-mon[41500]: osdmap e403: 8 total, 7 up, 6 in 2024-03-24T12:46:52.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:52 smithi029 ceph-mon[29919]: pgmap v924: 158 pgs: 1 active+recovering, 1 peering, 156 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.0 KiB/s rd, 74 MiB/s wr, 1.18k op/s; 3/206892 objects misplaced (0.001%); 22 B/s, 0 objects/s recovering 2024-03-24T12:46:52.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:52 smithi029 ceph-mon[29919]: osdmap e403: 8 total, 7 up, 6 in 2024-03-24T12:46:52.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:52.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:52.775 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:52.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:52.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:52.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:52.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:52.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:52.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 57 16 73383 73367 80.4297 41.0625 0.00232407 0.0124048 2024-03-24T12:46:53.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:53.040 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:53.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:53.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:53.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:53.172 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:53.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:53.274 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:53.297 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:53.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:53.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:53.399 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:53.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:53.424 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:53.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:53.525 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:53.549 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:53.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:53.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:53 smithi190 ceph-mon[33748]: pgmap v926: 158 pgs: 1 active+recovering, 157 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 3/212985 objects misplaced (0.001%); 37 B/s, 0 objects/s recovering 2024-03-24T12:46:53.651 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:53.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:53 smithi029 ceph-mon[29919]: pgmap v926: 158 pgs: 1 active+recovering, 157 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 3/212985 objects misplaced (0.001%); 37 B/s, 0 objects/s recovering 2024-03-24T12:46:53.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:53 smithi029 ceph-mon[41500]: pgmap v926: 158 pgs: 1 active+recovering, 157 active+clean; 4.3 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.7 KiB/s rd, 83 MiB/s wr, 1.33k op/s; 3/212985 objects misplaced (0.001%); 37 B/s, 0 objects/s recovering 2024-03-24T12:46:53.682 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7] out_osds: [5, 6] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:46:53.682 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:46:53.683 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-24T12:46:53.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-24T12:46:53.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:53.684 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:53.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:53.785 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:53.814 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:53.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:53.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:53.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:53.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 58 16 74058 74042 79.7704 42.1875 0.00819645 0.0125306 2024-03-24T12:46:53.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:53.961 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:53.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:54.062 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:54.105 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:54.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:54.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:54.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:54.230 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:54.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:54.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:54.356 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:54.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:54.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:54.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:54.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:54.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:54.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:54.629 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:54.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:54.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:54.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:54.755 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:54.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:54.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:54.887 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:54.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:54.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:54.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 59 16 74872 74856 79.2806 50.875 0.00547323 0.0125914 2024-03-24T12:46:54.989 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:55.032 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:55.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:55.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:55.133 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:55.160 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:55.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:55.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:55.262 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:55.285 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:55.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:55.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:55.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:55.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:55.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:55.562 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:55.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:55.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:55.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:55.688 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:55.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:55.789 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:55.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:55.813 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:55.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:55.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:55.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:46:55.925430+0000 min lat: 0.00142775 max lat: 0.499653 avg lat: 0.0126825 2024-03-24T12:46:55.920 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:46:55.920 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 60 15 75696 75681 78.8186 51.5625 0.0147741 0.0126825 2024-03-24T12:46:55.942 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:55.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:56.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:56.083 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:56.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:56.185 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:56.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:56.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:56.310 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:56.338 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:56.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:56.440 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:56.467 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:56.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:56.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:56.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:56.592 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:56.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:56.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:56 smithi190 ceph-mon[33748]: pgmap v927: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 737 B/s rd, 73 MiB/s wr, 1.17k op/s; 31 B/s, 0 objects/s recovering 2024-03-24T12:46:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:56 smithi190 ceph-mon[33748]: osdmap e404: 8 total, 7 up, 6 in 2024-03-24T12:46:56.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:56 smithi029 ceph-mon[29919]: pgmap v927: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 737 B/s rd, 73 MiB/s wr, 1.17k op/s; 31 B/s, 0 objects/s recovering 2024-03-24T12:46:56.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:56 smithi029 ceph-mon[29919]: osdmap e404: 8 total, 7 up, 6 in 2024-03-24T12:46:56.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:56 smithi029 ceph-mon[41500]: pgmap v927: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 737 B/s rd, 73 MiB/s wr, 1.17k op/s; 31 B/s, 0 objects/s recovering 2024-03-24T12:46:56.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:56 smithi029 ceph-mon[41500]: osdmap e404: 8 total, 7 up, 6 in 2024-03-24T12:46:56.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:56.716 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:56.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:56.818 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:56.846 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:56.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:56.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:56.919 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 61 16 76507 76491 78.3564 50.625 0.0100198 0.0127565 2024-03-24T12:46:56.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:46:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:56.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:56.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:46:57.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:57.129 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:57.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:57.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:57.231 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:57.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:57.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:57.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:57.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:46:57.385 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:57.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:46:57.487 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:57.526 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:57.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:57.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:57 smithi029 ceph-mon[41500]: pgmap v929: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 771 B/s rd, 47 MiB/s wr, 752 op/s; 16 B/s, 0 objects/s recovering 2024-03-24T12:46:57.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:57 smithi029 ceph-mon[29919]: pgmap v929: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 771 B/s rd, 47 MiB/s wr, 752 op/s; 16 B/s, 0 objects/s recovering 2024-03-24T12:46:57.628 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:57.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:57 smithi190 ceph-mon[33748]: pgmap v929: 158 pgs: 158 active+clean; 4.5 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 771 B/s rd, 47 MiB/s wr, 752 op/s; 16 B/s, 0 objects/s recovering 2024-03-24T12:46:57.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:57.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:57.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:57.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:57.781 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:57.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:57.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:57.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:57.911 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:57.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:57.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:46:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:46:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:46:57.920 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 62 16 77521 77505 78.1147 63.375 0.0173968 0.0127968 2024-03-24T12:46:58.013 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:58.061 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:58.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:58.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:58.189 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:58.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:58.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:58.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:58.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:58.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:58.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:46:58.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:58.441 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:58.441 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:46:58.542 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:58.545 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:58 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:46:58.545 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:58 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:46:58.560 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:58 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:46:58.560 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:58 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:46:58.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:58.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:58.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:58.696 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:58.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:58.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:58.798 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:58.824 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:58.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:58.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:58.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:46:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:46:58.892 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:58 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:46:58.893 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:58 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:46:58.920 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 63 15 78344 78329 77.6922 51.5 0.0129522 0.0128652 2024-03-24T12:46:58.926 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:46:58.952 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:58.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:46:59.054 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:59.082 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:59.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:59.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:59.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:59.209 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:59.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:59.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:59.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:46:59.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:59.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:59.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:46:59.439 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:46:59.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:59.485 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:46:59.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:46:59.586 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:59.609 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:59.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:59.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:59 smithi190 ceph-mon[33748]: osdmap e405: 8 total, 7 up, 6 in 2024-03-24T12:46:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:46:59 smithi190 ceph-mon[33748]: pgmap v931: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 14 B/s, 0 objects/s recovering 2024-03-24T12:46:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:59 smithi029 ceph-mon[29919]: osdmap e405: 8 total, 7 up, 6 in 2024-03-24T12:46:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:46:59 smithi029 ceph-mon[29919]: pgmap v931: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 14 B/s, 0 objects/s recovering 2024-03-24T12:46:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:59 smithi029 ceph-mon[41500]: osdmap e405: 8 total, 7 up, 6 in 2024-03-24T12:46:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:46:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:46:59 smithi029 ceph-mon[41500]: pgmap v931: 158 pgs: 158 active+clean; 4.7 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 66 MiB/s wr, 1.06k op/s; 14 B/s, 0 objects/s recovering 2024-03-24T12:46:59.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:59.736 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:59.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:59.837 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:46:59.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:46:59.866 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:46:59.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:46:59.920 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 64 16 79256 79240 77.3679 56.9375 0.0123539 0.0129204 2024-03-24T12:46:59.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:00.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:00.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:00.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:00.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:00.151 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:00.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:00.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:00.253 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:00.277 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:00.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:00.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:00.403 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:00.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:00.404 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:00.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:00.549 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:00.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:00.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:00.678 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:00.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:00.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:00.822 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:00.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:00.920 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 65 16 80203 80187 77.0882 59.1875 0.00731559 0.0129674 2024-03-24T12:47:00.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:00.952 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:00.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:00.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:01.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:01.096 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:01.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:01.198 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:01.225 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:01.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:01.327 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:01.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:01.351 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:01.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:01.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:01.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:01.477 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:01.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:01.578 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:01.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:01.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:01.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:01 smithi190 ceph-mon[33748]: osdmap e406: 8 total, 7 up, 6 in 2024-03-24T12:47:01.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:01 smithi029 ceph-mon[41500]: osdmap e406: 8 total, 7 up, 6 in 2024-03-24T12:47:01.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:01 smithi029 ceph-mon[29919]: osdmap e406: 8 total, 7 up, 6 in 2024-03-24T12:47:01.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:01.744 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:01.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:01.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:01.905 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:01.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:01.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 66 16 81099 81083 76.7678 56 0.0104989 0.0130206 2024-03-24T12:47:02.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:02.058 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:02.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:02.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:02.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:02.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:02.219 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:02.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:02.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:02.344 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:02.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:02.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:02.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:02.469 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:02.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:02.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:02.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:02.613 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:02.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:02 smithi190 ceph-mon[33748]: pgmap v933: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 49 MiB/s wr, 781 op/s 2024-03-24T12:47:02.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:02 smithi029 ceph-mon[41500]: pgmap v933: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 49 MiB/s wr, 781 op/s 2024-03-24T12:47:02.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:02 smithi029 ceph-mon[29919]: pgmap v933: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 49 MiB/s wr, 781 op/s 2024-03-24T12:47:02.715 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:02.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:02.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:02.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:02.868 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:02.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:02.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 67 16 81777 81761 76.2544 42.375 0.0031882 0.0130937 2024-03-24T12:47:02.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:02.994 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:02.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:03.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:03.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:03.121 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:03.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:03.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:03.259 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:03.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:03.361 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:03.404 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:03.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:03.506 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:03.533 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:03.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:03.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:03 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1947045418' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:47:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:03 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:47:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:03 smithi190 ceph-mon[33748]: pgmap v934: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 47 MiB/s wr, 751 op/s 2024-03-24T12:47:03.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:03.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:03.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:03 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1947045418' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:47:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:03 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:47:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:03 smithi029 ceph-mon[29919]: pgmap v934: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 47 MiB/s wr, 751 op/s 2024-03-24T12:47:03.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:47:03 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:47:03.427+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:47:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:03 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1947045418' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:47:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:03 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:47:03.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:03 smithi029 ceph-mon[41500]: pgmap v934: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 47 MiB/s wr, 751 op/s 2024-03-24T12:47:03.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:03.808 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:03.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:03.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:03.910 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:03.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 68 15 82606 82591 75.8959 51.875 0.0213604 0.0131707 2024-03-24T12:47:03.937 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:03.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:04.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:04.074 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:04.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:04.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:04.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:04.199 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:04.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:04.301 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:04.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:04.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:04.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:04.475 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:04.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:04.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:04.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:04.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:04.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:04.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:04 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:47:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:04 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:47:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:04 smithi190 ceph-mon[33748]: osdmap e407: 8 total, 7 up, 6 in 2024-03-24T12:47:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:04 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:47:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:04 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:47:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:04 smithi029 ceph-mon[41500]: osdmap e407: 8 total, 7 up, 6 in 2024-03-24T12:47:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:04 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:47:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:04 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:47:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:04 smithi029 ceph-mon[29919]: osdmap e407: 8 total, 7 up, 6 in 2024-03-24T12:47:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:04.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:04.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:04.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:04.839 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:04.868 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:04.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:04.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:04.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 69 16 83557 83541 75.6564 59.375 0.00992471 0.0132125 2024-03-24T12:47:04.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:05.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:05.011 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:05.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:05.065 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:47:05.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:05.135 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:05.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:05.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:05.262 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:05.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:05.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:05.387 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:05.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:05.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:05.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:05.529 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:05.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:05.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:05.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:05.669 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:05.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:05.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:05.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:05.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:05.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:05.895 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:05.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:05.920 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:05.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:05.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 70 16 84218 84202 75.1658 41.3125 0.0764277 0.0132725 2024-03-24T12:47:06.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:06.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:06.062 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:06.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:06.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:06.204 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:06.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:06.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:06.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:06.330 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:06.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:06.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:06.459 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:06.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:06.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:06.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:06.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:06.602 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:06.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:06 smithi190 ceph-mon[33748]: pgmap v936: 158 pgs: 158 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 777 B/s rd, 46 MiB/s wr, 732 op/s 2024-03-24T12:47:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:06 smithi190 ceph-mon[33748]: osdmap e408: 8 total, 7 up, 6 in 2024-03-24T12:47:06.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:06 smithi029 ceph-mon[41500]: pgmap v936: 158 pgs: 158 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 777 B/s rd, 46 MiB/s wr, 732 op/s 2024-03-24T12:47:06.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:06 smithi029 ceph-mon[41500]: osdmap e408: 8 total, 7 up, 6 in 2024-03-24T12:47:06.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:06 smithi029 ceph-mon[29919]: pgmap v936: 158 pgs: 158 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 777 B/s rd, 46 MiB/s wr, 732 op/s 2024-03-24T12:47:06.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:06 smithi029 ceph-mon[29919]: osdmap e408: 8 total, 7 up, 6 in 2024-03-24T12:47:06.702 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:06.726 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:06.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:06.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:06.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:06.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:06.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:06.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 71 16 85119 85103 74.9002 56.3125 0.00247791 0.0133214 2024-03-24T12:47:06.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:06.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:06.995 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:06.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:07.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:07.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:07.136 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:07.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:07.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:07.260 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:07.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:07.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:07.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:07.386 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:07.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:07.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:07.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:07.523 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:07.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:07.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:07 smithi029 ceph-mon[41500]: pgmap v938: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 771 B/s rd, 40 MiB/s wr, 644 op/s 2024-03-24T12:47:07.605 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:07 smithi029 ceph-mon[29919]: pgmap v938: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 771 B/s rd, 40 MiB/s wr, 644 op/s 2024-03-24T12:47:07.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:07.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:07 smithi190 ceph-mon[33748]: pgmap v938: 158 pgs: 158 active+clean; 5.0 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 771 B/s rd, 40 MiB/s wr, 644 op/s 2024-03-24T12:47:07.652 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:07.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:07.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:07.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:07.795 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:07.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:07.796 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:07.896 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:07.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:47:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:47:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:47:07.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 72 16 85979 85963 74.6064 53.75 0.036246 0.0133974 2024-03-24T12:47:07.926 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:07.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:07.927 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:08.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:08.071 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:08.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:08.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:08.200 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:08.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:08.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:08.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:08.344 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:08.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:08.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:08.472 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:08.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:08.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:08.574 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:08.614 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:08.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:08.716 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:08.739 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:08.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:08.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:08.841 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:08.870 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:08.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:08.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:47:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:47:08.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 73 16 87054 87038 74.5047 67.1875 0.0118987 0.0134175 2024-03-24T12:47:08.971 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:08.999 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:09.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:09.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:09.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:09.130 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:09.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:09.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:09.232 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:09.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:09.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:09.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:09.363 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:09.391 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:09.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:09.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:09.517 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:09.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:09.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:09.657 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:09.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:09.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:09.801 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:09.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:09.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:09.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:09.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 74 16 88002 87986 74.2985 59.25 0.0101284 0.0134543 2024-03-24T12:47:09.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:09.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:10.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:10.073 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:10.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:10.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:10.216 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:10.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:10.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:10.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:10.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:09 smithi190 ceph-mon[33748]: pgmap v939: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 930 op/s 2024-03-24T12:47:10.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:10.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:09 smithi029 ceph-mon[41500]: pgmap v939: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 930 op/s 2024-03-24T12:47:10.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:10.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:09 smithi029 ceph-mon[29919]: pgmap v939: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 930 op/s 2024-03-24T12:47:10.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:10.499 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:10.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:10.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:10.639 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:10.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:10.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:10.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:10.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:10.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:10.915 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:10.921 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 75 15 89026 89011 74.1619 64.0625 0.0109712 0.0134791 2024-03-24T12:47:10.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:10.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:11.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:11.095 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:11.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:11.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:11.114 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:47:11.182 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:47:11.182 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:47:11.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:11.253 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:11.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:11.355 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:11.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:11.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:11.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:11.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:11 smithi029 ceph-mon[29919]: pgmap v940: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 54 MiB/s wr, 870 op/s 2024-03-24T12:47:11.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:11 smithi029 ceph-mon[41500]: pgmap v940: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 54 MiB/s wr, 870 op/s 2024-03-24T12:47:11.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:11.530 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:11.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:11.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:11.631 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:11.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:11 smithi190 ceph-mon[33748]: pgmap v940: 158 pgs: 158 active+clean; 5.3 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 54 MiB/s wr, 870 op/s 2024-03-24T12:47:11.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:11.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:11.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:11.801 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:11.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:11.903 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:11.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 76 16 90089 90073 74.0594 66.375 0.0255069 0.0134975 2024-03-24T12:47:11.929 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:11.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:11.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:12.004 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:47:12.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:12.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:12.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:12.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:12.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:12.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:12.331 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:12.358 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:12.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:12.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:12.460 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:12.501 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:12.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:12.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:12.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:12.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:12.641 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:12.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:12.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:12.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:12.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:12.876 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:12.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:12.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:12.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:12.923 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 77 16 91129 91113 73.9417 65 0.0138462 0.013519 2024-03-24T12:47:13.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:13.045 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:13.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:13.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:13.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:13.184 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:13.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:13.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:13.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:13.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:13.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:13.414 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:13.440 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:13.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:13.541 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:13.565 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:13.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:13.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:13.709 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:13.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:13.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:13.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:47:13.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:13.838 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:13.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:13.922 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 78 15 92155 92140 73.8166 64.1875 0.0100487 0.0135421 2024-03-24T12:47:13.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:13.968 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:13.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:14.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:14.117 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:14.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:14.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:13 smithi029 ceph-mon[41500]: pgmap v941: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 752 op/s 2024-03-24T12:47:14.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:13 smithi029 ceph-mon[29919]: pgmap v941: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 752 op/s 2024-03-24T12:47:14.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:14.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:14.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:14.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:13 smithi190 ceph-mon[33748]: pgmap v941: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.0 KiB/s rd, 47 MiB/s wr, 752 op/s 2024-03-24T12:47:14.390 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:14.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:14.492 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:14.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:14.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:14.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:14.682 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:14.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:14.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:14.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T12:47:14.784 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:14.812 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:14.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:14.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:14.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 79 16 92999 92983 73.5473 52.6875 0.0109847 0.0135918 2024-03-24T12:47:14.940 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:14.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:15.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:15.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:15.102 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:15.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:15.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:15.264 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:15.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:15.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:15.366 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:15.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:15.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:15.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:15.495 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:15.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:15.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:15.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:15.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:15.651 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:15.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:15.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:15.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:15.795 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:15.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:15.896 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:15.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:2024-03-24T12:47:15.929877+0000 min lat: 0.00142775 max lat: 0.499653 avg lat: 0.0135878 2024-03-24T12:47:15.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-24T12:47:15.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 80 16 94204 94188 73.5693 75.3125 0.0159827 0.0135878 2024-03-24T12:47:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:15.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:16.040 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:16.085 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:16.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:16.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:16.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:16.190 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:15 smithi029 ceph-mon[29919]: pgmap v942: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 58 MiB/s wr, 928 op/s 2024-03-24T12:47:16.191 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:15 smithi029 ceph-mon[41500]: pgmap v942: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 58 MiB/s wr, 928 op/s 2024-03-24T12:47:16.246 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:16.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:16.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:16.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:15 smithi190 ceph-mon[33748]: pgmap v942: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 58 MiB/s wr, 928 op/s 2024-03-24T12:47:16.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:16.386 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:16.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:16.487 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:16.515 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:16.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:16.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:16.555 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:47:16.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:16.665 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:16.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:16.767 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:16.794 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:16.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:16.896 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:16.924 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:16.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 81 16 95430 95414 73.6069 76.625 0.00968629 0.0135806 2024-03-24T12:47:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:16.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:17.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:17.052 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:17.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:17.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:17.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:17.181 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:17.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:17.283 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:17.309 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:17.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:17.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:17.411 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:17.460 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:17.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:17.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:17.561 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:17.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:17.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:17.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:17.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:17.736 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:17.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:17.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:17.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:17.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:17.909 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:47:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:47:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:47:17.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 82 16 96634 96618 73.6268 75.25 0.0129277 0.0135777 2024-03-24T12:47:17.985 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:18.012 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:18.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:18.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:18.156 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:18.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:18.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:17 smithi029 ceph-mon[29919]: pgmap v943: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 879 B/s rd, 50 MiB/s wr, 808 op/s 2024-03-24T12:47:18.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:17 smithi029 ceph-mon[41500]: pgmap v943: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 879 B/s rd, 50 MiB/s wr, 808 op/s 2024-03-24T12:47:18.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:18.299 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:18.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:18.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:18.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:17 smithi190 ceph-mon[33748]: pgmap v943: 158 pgs: 158 active+clean; 5.6 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 879 B/s rd, 50 MiB/s wr, 808 op/s 2024-03-24T12:47:18.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:18.444 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:18.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:18.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:18.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:18.591 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:18.691 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:18.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:18.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:18.832 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:18.848 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:47:18 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:47:18.457+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:47:18.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:18.860 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:18.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:18.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:47:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:47:18.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 83 16 97785 97769 73.6064 71.9375 0.011853 0.0135806 2024-03-24T12:47:18.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:19.013 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:19.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:19.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:19.140 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:19.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:19.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:18 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1590086846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:47:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:18 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1590086846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:47:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:19.243 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:19.285 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:19.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:18 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1590086846' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:47:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:19.387 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:19.413 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:19.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:19.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:19.515 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:19.542 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:19.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:19.647 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:19.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:19.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:19.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:19.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:19.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:19.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:19.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:19.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 84 16 98814 98798 73.4957 64.3125 0.00391387 0.0135911 2024-03-24T12:47:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:19.943 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:19.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:20.044 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:20.067 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:20.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:20.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:20.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:20.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:20.208 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:20.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:20.309 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:20.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:20.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:19 smithi190 ceph-mon[33748]: pgmap v944: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:47:20.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:20.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:20 smithi029 ceph-mon[29919]: pgmap v944: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:47:20.399 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:20.399 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:20 smithi029 ceph-mon[41500]: pgmap v944: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-24T12:47:20.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:20.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:20.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:20.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:20.624 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:20.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:20.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:20.751 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:20.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:20.853 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:20.879 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:20.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:20.924 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 85 16 99496 99480 73.1325 42.625 0.0188926 0.0136683 2024-03-24T12:47:20.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:21.007 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:21.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:21.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:21.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:21.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:21.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:21.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:21.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:20 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2400465393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:47:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:20 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2400465393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:47:21.237 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:21.266 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:21.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:21.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:21.368 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:20 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2400465393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-24T12:47:21.391 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:21.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:21.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:21.493 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:21.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:21.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:21.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:21.613 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout: "epoch": 408, 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout: "fsid": "bc017092-e9d8-11ee-95cd-87774f69a715", 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout: "created": "2024-03-24T12:20:18.712998+0000", 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout: "modified": "2024-03-24T12:47:05.239358+0000", 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout: "last_up_change": "2024-03-24T12:35:57.677873+0000", 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout: "last_in_change": "2024-03-24T12:45:12.212720+0000", 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_num": 5806080, 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_set": [ 2024-03-24T12:47:21.614 INFO:teuthology.orchestra.run.smithi029.stdout: "nodeep-scrub", 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "noscrub", 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "pglog_hardlimit", 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "purged_snapdirs", 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "recovery_deletes", 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "sortbitwise" 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_version": 18, 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "full_ratio": 0.94999998807907104, 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_snapshot": "", 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_max": 11, 2024-03-24T12:47:21.615 INFO:teuthology.orchestra.run.smithi029.stdout: "max_osd": 8, 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "require_min_compat_client": "luminous", 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "min_compat_client": "luminous", 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "require_osd_release": "squid", 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "allow_crimson": false, 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "pools": [ 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 1, 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".mgr", 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:23:41.491597+0000", 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:47:21.616 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 11, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 11, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 11, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 11, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 11, 2024-03-24T12:47:21.617 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "295", 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "293", 2024-03-24T12:47:21.618 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:47:21.619 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:47:21.620 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_max": 32, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_min": 1 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:47:21.621 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": {} 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.9100000858306885, 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.9100000858306885, 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.1800000667572021, 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2.1800000667572021, 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:47:21.622 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 2, 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "datapool", 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:01.465080+0000", 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 8193, 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:47:21.623 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 3, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 3, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 3, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 3, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 3, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:47:21.624 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "71", 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 2, 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 71, 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:47:21.625 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:47:21.626 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:47:21.627 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "rbd": {} 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.6600000858306885, 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.6600000858306885, 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.5, 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.3300000429153442, 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.3300000429153442, 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:47:21.628 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 3, 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": ".rgw.root", 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:02.100021+0000", 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:47:21.629 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:47:21.630 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "68", 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:47:21.631 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:47:21.632 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:47:21.633 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.25, 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.25, 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.690000057220459, 2024-03-24T12:47:21.634 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.690000057220459, 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 4, 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.log", 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:03.473862+0000", 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:47:21.635 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:47:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "70", 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:47:21.637 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:47:21.638 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:47:21.639 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.640 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 3, 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 3, 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 2.25, 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 2.25, 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 5, 2024-03-24T12:47:21.641 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.control", 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:05.573828+0000", 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:47:21.642 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.643 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "72", 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:47:21.644 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:47:21.645 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:47:21.646 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 1.75, 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 1.75, 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.309999942779541, 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.309999942779541, 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.647 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 6, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "default.rgw.meta", 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:26:07.645083+0000", 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:47:21.648 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 32, 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 32, 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 32, 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 32, 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 32, 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.649 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "74", 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:47:21.650 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:47:21.651 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:47:21.652 INFO:teuthology.orchestra.run.smithi029.stdout: "options": { 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_bias": 4 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "rgw": {} 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2, 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2, 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.5, 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.5, 2024-03-24T12:47:21.653 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: "pool": 11, 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_name": "unique_pool_4", 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: "create_time": "2024-03-24T12:45:41.907152+0000", 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: "flags": 1, 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: "flags_names": "hashpspool", 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: "type": 1, 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 3, 2024-03-24T12:47:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: "min_size": 2, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_rule": 0, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_count": 0, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_target": 0, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_barrier": 0, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "object_hash": 2, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_autoscale_mode": "off", 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num": 16, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num": 16, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_placement_num_target": 16, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_target": 16, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_num_pending": 16, 2024-03-24T12:47:21.655 INFO:teuthology.orchestra.run.smithi029.stdout: "last_pg_merge_meta": { 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "source_pgid": "0.0", 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "ready_epoch": 0, 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_started": 0, 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "last_epoch_clean": 0, 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "source_version": "0'0", 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "target_version": "0'0" 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "last_change": "387", 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend": "0", 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "last_force_op_resend_preluminous": "0", 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "auid": 0, 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_mode": "selfmanaged", 2024-03-24T12:47:21.656 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_seq": 0, 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "snap_epoch": 0, 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "pool_snaps": [], 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps": "[]", 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_bytes": 0, 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "quota_max_objects": 0, 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "tiers": [], 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "tier_of": -1, 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "read_tier": -1, 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "write_tier": -1, 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_mode": "none", 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_bytes": 0, 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "target_max_objects": 0, 2024-03-24T12:47:21.657 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_flush_age": 0, 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "cache_min_evict_age": 0, 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profile": "", 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_params": { 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "none" 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_period": 0, 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_count": 0, 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "use_gmt_hitset": true, 2024-03-24T12:47:21.658 INFO:teuthology.orchestra.run.smithi029.stdout: "min_read_recency_for_promote": 0, 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "min_write_recency_for_promote": 0, 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_grade_decay_rate": 0, 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "hit_set_search_last_n": 0, 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "grade_table": [], 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "stripe_width": 0, 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "expected_num_objects": 0, 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "fast_read": false, 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "options": {}, 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "application_metadata": { 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "rados": {} 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "read_balance": { 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "score_type": "Fair distribution", 2024-03-24T12:47:21.659 INFO:teuthology.orchestra.run.smithi029.stdout: "score_acting": 2.5099999904632568, 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: "score_stable": 2.5099999904632568, 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: "optimal_score": 0.75, 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity_weighted": 0.94697189331054688, 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity": 0.94999998807907104, 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: "average_primary_affinity_weighted": 0.94999998807907104 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:47:21.660 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0aa1c49b-7887-4f74-b03c-08cc0adec90b", 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 8, 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 400, 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.661 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6801", 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6802", 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.662 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6804", 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.663 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6803", 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2514818464 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6801/2514818464", 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6802/2514818464", 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6804/2514818464", 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6803/2514818464", 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:47:21.664 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "99d8ef8f-eb59-4882-ab51-b398e6b53bcc", 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 0.575775146484375, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 13, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 382, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:47:21.665 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6805", 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.666 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6806", 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6808", 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.667 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6807", 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 2923396572 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6805/2923396572", 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6806/2923396572", 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6808/2923396572", 2024-03-24T12:47:21.668 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6807/2923396572", 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "c3f83505-25ee-4217-b60a-081c36092e92", 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:47:21.669 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 18, 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 382, 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6809", 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.670 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6810", 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.671 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6812", 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6811", 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 126032915 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.672 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6809/126032915", 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6810/126032915", 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6812/126032915", 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6811/126032915", 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "f5e479a8-459a-4315-a808-711a7b84bb96", 2024-03-24T12:47:21.673 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 0.3369293212890625, 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 27, 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 399, 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.674 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6813", 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6814", 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.675 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6816", 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.676 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.29:6815", 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1080171824 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.29:6813/1080171824", 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.29:6814/1080171824", 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.29:6816/1080171824", 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.29:6815/1080171824", 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.677 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "5e420304-2798-4d11-9688-d0d89482008c", 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 1, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 33, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 400, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:47:21.678 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6800", 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.679 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6801", 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6803", 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.680 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6802", 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 3690258869 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6800/3690258869", 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6801/3690258869", 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6803/3690258869", 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6802/3690258869", 2024-03-24T12:47:21.681 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "1213431a-652c-4cbf-b32a-36335430c666", 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 0, 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 0, 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 0, 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:47:21.682 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 42, 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 158, 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 204, 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6804", 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.683 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6805", 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.684 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6807", 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6806", 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1327220488 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6804/1327220488", 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6805/1327220488", 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6807/1327220488", 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6806/1327220488", 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: "exists" 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "0712e511-0257-44f3-a82f-ee09ab957c0c", 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 0, 2024-03-24T12:47:21.686 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 0, 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 51, 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 369, 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6808", 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:47:21.687 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6809", 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.688 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6811", 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6810", 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 1295064976 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.689 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6808/1295064976", 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6809/1295064976", 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6811/1295064976", 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6810/1295064976", 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:47:21.690 INFO:teuthology.orchestra.run.smithi029.stdout: "uuid": "7e36bde4-14d0-4e7e-9a82-2bb97859dbda", 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "up": 1, 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "in": 1, 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "weight": 0.828521728515625, 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_affinity": 1, 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_begin": 0, 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "last_clean_end": 0, 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "up_from": 60, 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "up_thru": 382, 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "down_at": 0, 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "lost_at": 0, 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addrs": { 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.691 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6812", 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addrs": { 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6813", 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.692 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addrs": { 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6815", 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addrs": { 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: "addrvec": [ 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.693 INFO:teuthology.orchestra.run.smithi029.stdout: "type": "v2", 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: "addr": "172.21.15.190:6814", 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: "nonce": 110963949 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: "public_addr": "172.21.15.190:6812/110963949", 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_addr": "172.21.15.190:6813/110963949", 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_back_addr": "172.21.15.190:6815/110963949", 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: "heartbeat_front_addr": "172.21.15.190:6814/110963949", 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: "state": [ 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: "exists", 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: "up" 2024-03-24T12:47:21.694 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: "osd_xinfo": [ 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 0, 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:46:19.874752+0000", 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:47:21.695 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 1, 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:46:47.624746+0000", 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 2, 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:47:21.696 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:46:02.405235+0000", 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 3, 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:47:21.697 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:46:57.995370+0000", 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 4, 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:46:12.574938+0000", 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.698 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 5, 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 65536, 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:35:04.518484+0000", 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 203 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 6, 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:47:21.699 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 65536, 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:46:50.892481+0000", 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": 7, 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "down_stamp": "0.000000", 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_probability": 0, 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "laggy_interval": 0, 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "features": 4540701547738038271, 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "old_weight": 0, 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "last_purged_snaps_scrub": "2024-03-24T12:46:29.441066+0000", 2024-03-24T12:47:21.700 INFO:teuthology.orchestra.run.smithi029.stdout: "dead_epoch": 0 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap": [ 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "5.f", 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: "osds": [ 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: 3, 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: 4, 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: 7 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_items": [ 2024-03-24T12:47:21.701 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.b", 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.f", 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.702 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 2 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.16", 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 7 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.703 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: "pgid": "4.1b", 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: "mappings": [ 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: "from": 1, 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: "to": 7 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_upmap_primaries": [], 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: "pg_temp": [], 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: "primary_temp": [], 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: "blocklist": { 2024-03-24T12:47:21.704 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2633022206": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2612078930": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2365028933": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4113609697": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/4130171883": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3697529912": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/2929430696": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/327626505": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/3715448582": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/861146182": "2024-03-25T12:26:46.364982+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/2109454792": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1631806715": "2024-03-25T12:20:46.948976+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:6800/852212806": "2024-03-25T12:21:15.151901+0000", 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: "172.21.15.29:0/1823252077": "2024-03-25T12:21:15.151901+0000" 2024-03-24T12:47:21.705 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "range_blocklist": {}, 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "erasure_code_profiles": { 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "default": { 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "crush-failure-domain": "osd", 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "k": "2", 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "m": "1", 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "plugin": "jerasure", 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "technique": "reed_sol_van" 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "removed_snaps_queue": [], 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "new_removed_snaps": [], 2024-03-24T12:47:21.706 INFO:teuthology.orchestra.run.smithi029.stdout: "new_purged_snaps": [], 2024-03-24T12:47:21.707 INFO:teuthology.orchestra.run.smithi029.stdout: "crush_node_flags": {}, 2024-03-24T12:47:21.707 INFO:teuthology.orchestra.run.smithi029.stdout: "device_class_flags": {}, 2024-03-24T12:47:21.707 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode": { 2024-03-24T12:47:21.707 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_enabled": false, 2024-03-24T12:47:21.707 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_bucket_count": 0, 2024-03-24T12:47:21.707 INFO:teuthology.orchestra.run.smithi029.stdout: "degraded_stretch_mode": 0, 2024-03-24T12:47:21.707 INFO:teuthology.orchestra.run.smithi029.stdout: "recovering_stretch_mode": 0, 2024-03-24T12:47:21.707 INFO:teuthology.orchestra.run.smithi029.stdout: "stretch_mode_bucket": 0 2024-03-24T12:47:21.707 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:21.707 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:47:21.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:21.748 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:21.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:21.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:21.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:21.891 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:21.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:21.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 86 16 100531 100515 73.0343 64.6875 0.0149962 0.0136871 2024-03-24T12:47:21.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:22.033 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:22.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:22.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:22.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:22.177 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:22.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:22.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:22.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:22.307 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:22.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:22.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:21 smithi190 ceph-mon[33748]: pgmap v945: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 61 MiB/s wr, 981 op/s 2024-03-24T12:47:22.392 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:21 smithi029 ceph-mon[41500]: pgmap v945: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 61 MiB/s wr, 981 op/s 2024-03-24T12:47:22.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:21 smithi029 ceph-mon[29919]: pgmap v945: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 61 MiB/s wr, 981 op/s 2024-03-24T12:47:22.409 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:22.439 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:22.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:22.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:22.541 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:22.586 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:22.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:22.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:22.688 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:22.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:22.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:22.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:22.901 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:22.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:22.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:22.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 87 15 101731 101716 73.0575 75.0625 0.02084 0.0136824 2024-03-24T12:47:22.957 INFO:tasks.thrashosds.thrasher:j is {'epoch': 408, 'fsid': 'bc017092-e9d8-11ee-95cd-87774f69a715', 'created': '2024-03-24T12:20:18.712998+0000', 'modified': '2024-03-24T12:47:05.239358+0000', 'last_up_change': '2024-03-24T12:35:57.677873+0000', 'last_in_change': '2024-03-24T12:45:12.212720+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T12:23:41.491597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '295', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '293', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9100000858306885, 'score_stable': 2.9100000858306885, 'optimal_score': 0.75, 'raw_score_acting': 2.180000066757202, 'raw_score_stable': 2.180000066757202, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-03-24T12:26:01.465080+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6600000858306885, 'score_stable': 2.6600000858306885, 'optimal_score': 0.5, 'raw_score_acting': 1.3300000429153442, 'raw_score_stable': 1.3300000429153442, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-03-24T12:26:02.100021+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-24T12:26:03.473862+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3, 'score_stable': 3, 'optimal_score': 0.75, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-24T12:26:05.573828+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 0.75, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-24T12:26:07.645083+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 0.75, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-24T12:45:41.907152+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '387', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9469718933105469, 'average_primary_affinity': 0.949999988079071, 'average_primary_affinity_weighted': 0.949999988079071}}], 'osds': [{'osd': 0, 'uuid': '0aa1c49b-7887-4f74-b03c-08cc0adec90b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 400, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6801', 'nonce': 2514818464}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6802', 'nonce': 2514818464}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6804', 'nonce': 2514818464}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6803', 'nonce': 2514818464}]}, 'public_addr': '172.21.15.29:6801/2514818464', 'cluster_addr': '172.21.15.29:6802/2514818464', 'heartbeat_back_addr': '172.21.15.29:6804/2514818464', 'heartbeat_front_addr': '172.21.15.29:6803/2514818464', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '99d8ef8f-eb59-4882-ab51-b398e6b53bcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.575775146484375, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6805', 'nonce': 2923396572}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6806', 'nonce': 2923396572}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6808', 'nonce': 2923396572}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6807', 'nonce': 2923396572}]}, 'public_addr': '172.21.15.29:6805/2923396572', 'cluster_addr': '172.21.15.29:6806/2923396572', 'heartbeat_back_addr': '172.21.15.29:6808/2923396572', 'heartbeat_front_addr': '172.21.15.29:6807/2923396572', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'c3f83505-25ee-4217-b60a-081c36092e92', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6809', 'nonce': 126032915}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6810', 'nonce': 126032915}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6812', 'nonce': 126032915}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6811', 'nonce': 126032915}]}, 'public_addr': '172.21.15.29:6809/126032915', 'cluster_addr': '172.21.15.29:6810/126032915', 'heartbeat_back_addr': '172.21.15.29:6812/126032915', 'heartbeat_front_addr': '172.21.15.29:6811/126032915', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f5e479a8-459a-4315-a808-711a7b84bb96', 'up': 1, 'in': 1, 'weight': 0.3369293212890625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 27, 'up_thru': 399, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6813', 'nonce': 1080171824}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6814', 'nonce': 1080171824}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6816', 'nonce': 1080171824}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6815', 'nonce': 1080171824}]}, 'public_addr': '172.21.15.29:6813/1080171824', 'cluster_addr': '172.21.15.29:6814/1080171824', 'heartbeat_back_addr': '172.21.15.29:6816/1080171824', 'heartbeat_front_addr': '172.21.15.29:6815/1080171824', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '5e420304-2798-4d11-9688-d0d89482008c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 33, 'up_thru': 400, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6800', 'nonce': 3690258869}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6801', 'nonce': 3690258869}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6803', 'nonce': 3690258869}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6802', 'nonce': 3690258869}]}, 'public_addr': '172.21.15.190:6800/3690258869', 'cluster_addr': '172.21.15.190:6801/3690258869', 'heartbeat_back_addr': '172.21.15.190:6803/3690258869', 'heartbeat_front_addr': '172.21.15.190:6802/3690258869', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '1213431a-652c-4cbf-b32a-36335430c666', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 42, 'up_thru': 158, 'down_at': 204, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6804', 'nonce': 1327220488}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6805', 'nonce': 1327220488}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6807', 'nonce': 1327220488}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6806', 'nonce': 1327220488}]}, 'public_addr': '172.21.15.190:6804/1327220488', 'cluster_addr': '172.21.15.190:6805/1327220488', 'heartbeat_back_addr': '172.21.15.190:6807/1327220488', 'heartbeat_front_addr': '172.21.15.190:6806/1327220488', 'state': ['exists']}, {'osd': 6, 'uuid': '0712e511-0257-44f3-a82f-ee09ab957c0c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 51, 'up_thru': 369, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6808', 'nonce': 1295064976}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6809', 'nonce': 1295064976}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6811', 'nonce': 1295064976}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6810', 'nonce': 1295064976}]}, 'public_addr': '172.21.15.190:6808/1295064976', 'cluster_addr': '172.21.15.190:6809/1295064976', 'heartbeat_back_addr': '172.21.15.190:6811/1295064976', 'heartbeat_front_addr': '172.21.15.190:6810/1295064976', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '7e36bde4-14d0-4e7e-9a82-2bb97859dbda', 'up': 1, 'in': 1, 'weight': 0.828521728515625, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 382, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6812', 'nonce': 110963949}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6813', 'nonce': 110963949}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6815', 'nonce': 110963949}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.190:6814', 'nonce': 110963949}]}, 'public_addr': '172.21.15.190:6812/110963949', 'cluster_addr': '172.21.15.190:6813/110963949', 'heartbeat_back_addr': '172.21.15.190:6815/110963949', 'heartbeat_front_addr': '172.21.15.190:6814/110963949', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:46:19.874752+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:46:47.624746+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:46:02.405235+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:46:57.995370+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:46:12.574938+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-24T12:35:04.518484+0000', 'dead_epoch': 203}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-24T12:46:50.892481+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-24T12:46:29.441066+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '5.f', 'osds': [3, 4, 7]}], 'pg_upmap_items': [{'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.16', 'mappings': [{'from': 1, 'to': 7}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 7}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.29:0/2633022206': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2612078930': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2365028933': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4113609697': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/4130171883': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/3697529912': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:0/2929430696': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/327626505': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/3715448582': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/861146182': '2024-03-25T12:26:46.364982+0000', '172.21.15.29:6800/2109454792': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:0/1631806715': '2024-03-25T12:20:46.948976+0000', '172.21.15.29:6800/852212806': '2024-03-25T12:21:15.151901+0000', '172.21.15.29:0/1823252077': '2024-03-25T12:21:15.151901+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-24T12:47:22.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:47:23.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:23.054 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:47:23.054 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:47:23.054 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:47:23.054 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:47:23.054 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:47:23.054 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:47:23.054 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:47:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:47:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:47:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:47:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:47:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:47:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:47:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:47:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:47:23.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:23.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:23.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:23.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:23.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:23.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:23.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:23.425 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:23.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:23.526 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:23.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:23.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:23.655 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:23.709 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:23.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:23.811 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:23.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:23.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:23.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:23.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 88 16 102787 102771 72.9766 65.9375 0.0112792 0.0136979 2024-03-24T12:47:23.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:23.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:24.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:24.086 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:23 smithi029 ceph-mon[29919]: pgmap v946: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-24T12:47:24.086 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:23 smithi029 ceph-mon[41500]: pgmap v946: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-24T12:47:24.102 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:47:24.149 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:24.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:24.150 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:47:24.158 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:47:24.200 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:47:24.251 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:24.279 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:24.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:24.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:24.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:23 smithi190 ceph-mon[33748]: pgmap v946: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-24T12:47:24.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:24.423 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:24.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:24.524 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:24.551 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:24.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:24.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:24.694 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:24.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:24.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:24.822 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:24.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:24.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:24.924 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:24.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout: 89 16 103914 103898 72.948 70.4375 0.00838939 0.0137037 2024-03-24T12:47:24.954 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:24.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:47:25.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:25.099 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:25.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:25.200 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:25.241 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:25.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:25.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:25.349 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:24 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1127594471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:47:25.349 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:24 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:47:25.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:24 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1127594471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:47:25.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:24 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:47:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:24 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1127594471' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:47:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:24 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:47:25.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:25.384 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:25.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:25.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:25.529 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:25.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:25.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:25.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:25.658 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:25.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:25.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:25.784 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:25.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:25.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:25.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Total time run: 90.0054 2024-03-24T12:47:25.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Total writes made: 104932 2024-03-24T12:47:25.925 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Write size: 65536 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Object size: 65536 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Bandwidth (MB/sec): 72.8651 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev Bandwidth: 28.4475 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max bandwidth (MB/sec): 185.875 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min bandwidth (MB/sec): 31 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Average IOPS: 1165 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev IOPS: 455.16 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max IOPS: 2974 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min IOPS: 496 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Average Latency(s): 0.0137199 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Stddev Latency(s): 0.0245118 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Max latency(s): 0.499653 2024-03-24T12:47:25.926 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Min latency(s): 0.00142775 2024-03-24T12:47:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:25.927 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:25.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:25.928 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Cleaning up (deleting benchmark objects) 2024-03-24T12:47:26.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:26.061 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:26.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:26.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:47:26.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:26.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:26.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:26.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:26.292 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:26.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:25 smithi190 ceph-mon[33748]: pgmap v947: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T12:47:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:25 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:47:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:25 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:47:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:25 smithi190 ceph-mon[33748]: osdmap e409: 8 total, 7 up, 6 in 2024-03-24T12:47:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:25 smithi190 ceph-mon[33748]: 6.4 scrub starts 2024-03-24T12:47:26.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:25 smithi190 ceph-mon[33748]: 6.4 scrub ok 2024-03-24T12:47:26.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[41500]: pgmap v947: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T12:47:26.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:47:26.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:47:26.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[41500]: osdmap e409: 8 total, 7 up, 6 in 2024-03-24T12:47:26.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[41500]: 6.4 scrub starts 2024-03-24T12:47:26.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[41500]: 6.4 scrub ok 2024-03-24T12:47:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[29919]: pgmap v947: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 852 B/s rd, 69 MiB/s wr, 1.10k op/s 2024-03-24T12:47:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:47:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:47:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[29919]: osdmap e409: 8 total, 7 up, 6 in 2024-03-24T12:47:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[29919]: 6.4 scrub starts 2024-03-24T12:47:26.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:25 smithi029 ceph-mon[29919]: 6.4 scrub ok 2024-03-24T12:47:26.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:26.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:26.468 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:26.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:26.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:26.609 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:26.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:26.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:26.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:26.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:26.751 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:26.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:26.852 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:26.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:26.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:26.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:27.027 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:27.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:27.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:27.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:27.175 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:27.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:27.276 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:27.316 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:27.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:27.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:27.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:26 smithi029 ceph-mon[41500]: 11.f scrub starts 2024-03-24T12:47:27.353 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:26 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:47:27.353 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:26 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:47:27.353 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:26 smithi029 ceph-mon[41500]: osdmap e410: 8 total, 7 up, 6 in 2024-03-24T12:47:27.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:26 smithi029 ceph-mon[29919]: 11.f scrub starts 2024-03-24T12:47:27.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:26 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:47:27.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:26 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:47:27.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:26 smithi029 ceph-mon[29919]: osdmap e410: 8 total, 7 up, 6 in 2024-03-24T12:47:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:26 smithi190 ceph-mon[33748]: 11.f scrub starts 2024-03-24T12:47:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:26 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:47:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:26 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:47:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:26 smithi190 ceph-mon[33748]: osdmap e410: 8 total, 7 up, 6 in 2024-03-24T12:47:27.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:27.466 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:27.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:27.567 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:27.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:27.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:27.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:27.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:27.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:27.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:27.865 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:47:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:47:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:47:27.867 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:27.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:27.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:27.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:28.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:28.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:28.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:28.135 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:28.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:28.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:28 smithi029 ceph-mon[41500]: 3.16 scrub starts 2024-03-24T12:47:28.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:28 smithi029 ceph-mon[41500]: 3.16 scrub ok 2024-03-24T12:47:28.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:28 smithi029 ceph-mon[41500]: pgmap v950: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-24T12:47:28.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:28 smithi029 ceph-mon[41500]: 4.14 scrub starts 2024-03-24T12:47:28.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:28 smithi029 ceph-mon[41500]: 4.14 scrub ok 2024-03-24T12:47:28.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:28 smithi029 ceph-mon[41500]: 5.2 scrub starts 2024-03-24T12:47:28.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:28 smithi029 ceph-mon[41500]: 5.2 scrub ok 2024-03-24T12:47:28.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:27 smithi029 ceph-mon[29919]: 3.16 scrub starts 2024-03-24T12:47:28.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:27 smithi029 ceph-mon[29919]: 3.16 scrub ok 2024-03-24T12:47:28.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:27 smithi029 ceph-mon[29919]: pgmap v950: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-24T12:47:28.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:27 smithi029 ceph-mon[29919]: 4.14 scrub starts 2024-03-24T12:47:28.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:27 smithi029 ceph-mon[29919]: 4.14 scrub ok 2024-03-24T12:47:28.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:27 smithi029 ceph-mon[29919]: 5.2 scrub starts 2024-03-24T12:47:28.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:27 smithi029 ceph-mon[29919]: 5.2 scrub ok 2024-03-24T12:47:28.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:28.282 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:27 smithi190 ceph-mon[33748]: 3.16 scrub starts 2024-03-24T12:47:28.282 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:27 smithi190 ceph-mon[33748]: 3.16 scrub ok 2024-03-24T12:47:28.282 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:27 smithi190 ceph-mon[33748]: pgmap v950: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 51 MiB/s wr, 820 op/s 2024-03-24T12:47:28.282 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:27 smithi190 ceph-mon[33748]: 4.14 scrub starts 2024-03-24T12:47:28.282 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:27 smithi190 ceph-mon[33748]: 4.14 scrub ok 2024-03-24T12:47:28.282 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:27 smithi190 ceph-mon[33748]: 5.2 scrub starts 2024-03-24T12:47:28.283 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:27 smithi190 ceph-mon[33748]: 5.2 scrub ok 2024-03-24T12:47:28.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:28.284 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:28.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:28.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:28.431 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:28.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:28.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:28.533 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:28.560 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:28.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:28.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:28.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:28.698 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:28.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:28.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:28.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:28.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:28.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:47:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:47:28.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:28.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:28.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:29.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:29.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:29.095 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:29.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:29.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:29 smithi029 ceph-mon[41500]: 11.f scrub ok 2024-03-24T12:47:29.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:29 smithi029 ceph-mon[41500]: 5.7 scrub starts 2024-03-24T12:47:29.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:29 smithi029 ceph-mon[41500]: 5.7 scrub ok 2024-03-24T12:47:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:29 smithi029 ceph-mon[29919]: 11.f scrub ok 2024-03-24T12:47:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:29 smithi029 ceph-mon[29919]: 5.7 scrub starts 2024-03-24T12:47:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:29 smithi029 ceph-mon[29919]: 5.7 scrub ok 2024-03-24T12:47:29.196 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:29.224 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:29.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:29.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:29.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:29.367 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:29.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:29.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:29 smithi190 ceph-mon[33748]: 11.f scrub ok 2024-03-24T12:47:29.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:29 smithi190 ceph-mon[33748]: 5.7 scrub starts 2024-03-24T12:47:29.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:29 smithi190 ceph-mon[33748]: 5.7 scrub ok 2024-03-24T12:47:29.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:29.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:29.517 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:29.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:29.618 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:29.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:29.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:29.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:109289 11.a 11:592be3c8:::benchmark_data_smithi029_192894_object4354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:47:29.663441+0000", 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.002397789, 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0024329299999999998, 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "waiting for sub ops", 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:47:29.662 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 109289 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.663441+0000", 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.663441+0000", 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:47:29.663 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.663442+0000", 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.668e-06 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.663450+0000", 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5870000000000004e-06 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.663471+0000", 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0599e-05 2024-03-24T12:47:29.664 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.663482+0000", 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1572e-05 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.663617+0000", 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013439600000000001 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:47:29.665 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.663757+0000", 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014039099999999999 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.663872+0000", 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000114839 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:109292 11.a 11.daf9a3ba (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e410)", 2024-03-24T12:47:29.666 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:47:29.664563+0000", 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0012752110000000001, 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0014012709999999999, 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 109292 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.664563+0000", 2024-03-24T12:47:29.667 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.664563+0000", 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.664564+0000", 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6400000000000001e-07 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.668 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.664569+0000", 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6689999999999997e-06 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.664579+0000", 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0182e-05 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:29.664593+0000", 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3867e-05 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:47:29.669 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:29.670 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:47:29.670 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:47:29.670 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 2 2024-03-24T12:47:29.670 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:47:29.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:29.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:29.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:29.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:29.911 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:29.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:30.013 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:30.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:30.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:30.154 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:30.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[41500]: 3.7 scrub starts 2024-03-24T12:47:30.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[41500]: 3.7 scrub ok 2024-03-24T12:47:30.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[41500]: 5.16 scrub starts 2024-03-24T12:47:30.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[41500]: 5.16 scrub ok 2024-03-24T12:47:30.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:30.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[41500]: pgmap v951: 158 pgs: 158 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 1.15k op/s 2024-03-24T12:47:30.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[41500]: 3.17 scrub starts 2024-03-24T12:47:30.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[41500]: 3.17 scrub ok 2024-03-24T12:47:30.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[29919]: 3.7 scrub starts 2024-03-24T12:47:30.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[29919]: 3.7 scrub ok 2024-03-24T12:47:30.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[29919]: 5.16 scrub starts 2024-03-24T12:47:30.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[29919]: 5.16 scrub ok 2024-03-24T12:47:30.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:30.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[29919]: pgmap v951: 158 pgs: 158 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 1.15k op/s 2024-03-24T12:47:30.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[29919]: 3.17 scrub starts 2024-03-24T12:47:30.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:30 smithi029 ceph-mon[29919]: 3.17 scrub ok 2024-03-24T12:47:30.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:30.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:30.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:30.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:30.326 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:30.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:30.356 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:47:30.369 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:47:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:30 smithi190 ceph-mon[33748]: 3.7 scrub starts 2024-03-24T12:47:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:30 smithi190 ceph-mon[33748]: 3.7 scrub ok 2024-03-24T12:47:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:30 smithi190 ceph-mon[33748]: 5.16 scrub starts 2024-03-24T12:47:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:30 smithi190 ceph-mon[33748]: 5.16 scrub ok 2024-03-24T12:47:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:30 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:30 smithi190 ceph-mon[33748]: pgmap v951: 158 pgs: 158 active+clean; 6.3 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 1.15k op/s 2024-03-24T12:47:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:30 smithi190 ceph-mon[33748]: 3.17 scrub starts 2024-03-24T12:47:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:30 smithi190 ceph-mon[33748]: 3.17 scrub ok 2024-03-24T12:47:30.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:30.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:30.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:30.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:30.557 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:30.584 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:30.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:30.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:30.723 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:30.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:30.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:30.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T12:47:30.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:30.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:30.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:30.989 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:31.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:31.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:31.118 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:31.184 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:31.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:31.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:31.312 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:31.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:31.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:47:31.413 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:31.443 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:31.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:31.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:31.545 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:31.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:31.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:31 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:47:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:31 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:47:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:31 smithi190 ceph-mon[33748]: 11.7 scrub starts 2024-03-24T12:47:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:31 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:47:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:31 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:47:31.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:31 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:47:31.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:31 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:47:31.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:31 smithi029 ceph-mon[29919]: 11.7 scrub starts 2024-03-24T12:47:31.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:31 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:47:31.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:31 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:47:31.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:31 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:47:31.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:31 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:47:31.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:31 smithi029 ceph-mon[41500]: 11.7 scrub starts 2024-03-24T12:47:31.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:31 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:47:31.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:31 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:47:31.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:31.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:31.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:31.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:31.845 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:31.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:31.946 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:31.969 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:31.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:32.071 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:32.095 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:32.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:32.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:32.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:32.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:32.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:32.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:32.387 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:32.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:32.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:32.538 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:32.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:47:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: 5.6 scrub starts 2024-03-24T12:47:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: 5.6 scrub ok 2024-03-24T12:47:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:47:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:47:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: pgmap v952: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 767 B/s rd, 46 MiB/s wr, 1.09k op/s 2024-03-24T12:47:32.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: 5.17 scrub starts 2024-03-24T12:47:32.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: 5.17 scrub ok 2024-03-24T12:47:32.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: osdmap e411: 8 total, 7 up, 6 in 2024-03-24T12:47:32.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:47:32.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:47:32.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:32 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:47:32.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:32.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: 5.6 scrub starts 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: 5.6 scrub ok 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: pgmap v952: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 767 B/s rd, 46 MiB/s wr, 1.09k op/s 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: 5.17 scrub starts 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: 5.17 scrub ok 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: osdmap e411: 8 total, 7 up, 6 in 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:47:32.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: from='client.15510 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: 5.6 scrub starts 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: 5.6 scrub ok 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: pgmap v952: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 767 B/s rd, 46 MiB/s wr, 1.09k op/s 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: 5.17 scrub starts 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: 5.17 scrub ok 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: osdmap e411: 8 total, 7 up, 6 in 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:47:32.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:47:32.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:32 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:47:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:32.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:32.781 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:32.805 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:32.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:32.907 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:32.934 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:32.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:32.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:33.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:33.079 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:33.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:33.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:33.181 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:33.206 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:33.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:33.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:33.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:33.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:33.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:33.460 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:33.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:33.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:33.562 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:33.601 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:33.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:33.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:33 smithi190 ceph-mon[33748]: 1.a scrub starts 2024-03-24T12:47:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:33 smithi190 ceph-mon[33748]: 1.a scrub ok 2024-03-24T12:47:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:33 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:47:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:33 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:47:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:33 smithi190 ceph-mon[33748]: 4.1d scrub starts 2024-03-24T12:47:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:33 smithi190 ceph-mon[33748]: 4.1d scrub ok 2024-03-24T12:47:33.666 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:47:33.479+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:47:33.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[29919]: 1.a scrub starts 2024-03-24T12:47:33.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[29919]: 1.a scrub ok 2024-03-24T12:47:33.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:47:33.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:47:33.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[29919]: 4.1d scrub starts 2024-03-24T12:47:33.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[29919]: 4.1d scrub ok 2024-03-24T12:47:33.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[41500]: 1.a scrub starts 2024-03-24T12:47:33.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[41500]: 1.a scrub ok 2024-03-24T12:47:33.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:47:33.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:47:33.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[41500]: 4.1d scrub starts 2024-03-24T12:47:33.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:33 smithi029 ceph-mon[41500]: 4.1d scrub ok 2024-03-24T12:47:33.703 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:33.730 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:33.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:33.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:33.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:33.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:33.868 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:33.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:33.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:33.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:33.994 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:33.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:34.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:34.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:34.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:34.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:34.281 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:34.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:34.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:34.408 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:34.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:34.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:34.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:34.550 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:34.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: 4.19 scrub starts 2024-03-24T12:47:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: 4.19 scrub ok 2024-03-24T12:47:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: 5.1c scrub starts 2024-03-24T12:47:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: 5.1c scrub ok 2024-03-24T12:47:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: pgmap v954: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.03k op/s 2024-03-24T12:47:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: 11.a scrub starts 2024-03-24T12:47:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:47:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:47:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:47:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: 11.7 scrub ok 2024-03-24T12:47:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: 6.1c scrub starts 2024-03-24T12:47:34.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:34 smithi190 ceph-mon[33748]: 6.1c scrub ok 2024-03-24T12:47:34.652 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: 4.19 scrub starts 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: 4.19 scrub ok 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: 5.1c scrub starts 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: 5.1c scrub ok 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: pgmap v954: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.03k op/s 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: 11.a scrub starts 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: 11.7 scrub ok 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: 6.1c scrub starts 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[29919]: 6.1c scrub ok 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: 4.19 scrub starts 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: 4.19 scrub ok 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: 5.1c scrub starts 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: 5.1c scrub ok 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: pgmap v954: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.4 KiB/s rd, 27 MiB/s wr, 1.03k op/s 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: 11.a scrub starts 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:47:34.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: 11.7 scrub ok 2024-03-24T12:47:34.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:34.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: 6.1c scrub starts 2024-03-24T12:47:34.669 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:34 smithi029 ceph-mon[41500]: 6.1c scrub ok 2024-03-24T12:47:34.679 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:34.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:34.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:34.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:34.823 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:34.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:34.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:34.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:34.963 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:34.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:34.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:35.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:35.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:35.110 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:35.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:35.211 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:35.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:35.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:35.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:35.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:35.374 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:35.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:35.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:35.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:35.502 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:35.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:35.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:35.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:35 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:47:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:35 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:47:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:35 smithi190 ceph-mon[33748]: 5.19 scrub starts 2024-03-24T12:47:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:35 smithi190 ceph-mon[33748]: 5.19 scrub ok 2024-03-24T12:47:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:35 smithi190 ceph-mon[33748]: pgmap v955: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.3 KiB/s rd, 19 MiB/s wr, 1.16k op/s 2024-03-24T12:47:35.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:35.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:35.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:35 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:47:35.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:35 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:47:35.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:35 smithi029 ceph-mon[29919]: 5.19 scrub starts 2024-03-24T12:47:35.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:35 smithi029 ceph-mon[29919]: 5.19 scrub ok 2024-03-24T12:47:35.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:35 smithi029 ceph-mon[29919]: pgmap v955: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.3 KiB/s rd, 19 MiB/s wr, 1.16k op/s 2024-03-24T12:47:35.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:35 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:47:35.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:35 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:47:35.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:35 smithi029 ceph-mon[41500]: 5.19 scrub starts 2024-03-24T12:47:35.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:35 smithi029 ceph-mon[41500]: 5.19 scrub ok 2024-03-24T12:47:35.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:35 smithi029 ceph-mon[41500]: pgmap v955: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.3 KiB/s rd, 19 MiB/s wr, 1.16k op/s 2024-03-24T12:47:35.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:35.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:35.773 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:35.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:35.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:35.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:35.902 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:35.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:36.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:36.043 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:36.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:36.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:36.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:36.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:36.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:36.313 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:36.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:36.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:36.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:36.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:36.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:36.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:36.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:36.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:36 smithi190 ceph-mon[33748]: 5.18 scrub starts 2024-03-24T12:47:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:36 smithi190 ceph-mon[33748]: 5.18 scrub ok 2024-03-24T12:47:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:36 smithi190 ceph-mon[33748]: 11.a scrub ok 2024-03-24T12:47:36.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:36 smithi029 ceph-mon[29919]: 5.18 scrub starts 2024-03-24T12:47:36.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:36 smithi029 ceph-mon[29919]: 5.18 scrub ok 2024-03-24T12:47:36.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:36 smithi029 ceph-mon[29919]: 11.a scrub ok 2024-03-24T12:47:36.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:36 smithi029 ceph-mon[41500]: 5.18 scrub starts 2024-03-24T12:47:36.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:36 smithi029 ceph-mon[41500]: 5.18 scrub ok 2024-03-24T12:47:36.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:36 smithi029 ceph-mon[41500]: 11.a scrub ok 2024-03-24T12:47:36.691 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:36.717 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:36.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:36.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:36.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:36.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:36.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:36.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:36.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:36.986 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:36.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:37.087 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:37.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:37.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:37.216 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:37.243 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:37.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:37.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:37.345 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:37.394 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:37.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:37.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:37.451 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":951,"stamp":"2024-03-24T12:47:28.887472+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6758144350,"num_objects":103310,"num_object_clones":0,"num_object_copies":309930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103310,"num_whiteouts":0,"num_read":2839,"num_read_kb":2591,"num_write":107350,"num_write_kb":6702573,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":273,"num_bytes_recovered":795611,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61719,"ondisk_log_size":61719,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":24675032,"kb_used_data":19858752,"kb_used_omap":9,"kb_used_meta":4815606,"kb_avail":537795880,"statfs":{"total":575970213888,"available":550702981120,"internally_reserved":0,"allocated":20335362048,"data_stored":20328876120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9563,"internal_metadata":4931181221},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,5,7,2,2,0,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":392167424,"num_objects":5984,"num_object_clones":0,"num_object_copies":17952,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5984,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":9214,"num_write_kb":486336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002953"},"pg_stats":[{"pgid":"11.3","version":"410'6576","reported_seq":6642,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.805661+0000","last_change":"2024-03-24T12:45:43.234940+0000","last_active":"2024-03-24T12:47:27.805661+0000","last_peered":"2024-03-24T12:47:27.805661+0000","last_clean":"2024-03-24T12:47:27.805661+0000","last_became_active":"2024-03-24T12:45:43.234520+0000","last_became_peered":"2024-03-24T12:45:43.234520+0000","last_unstale":"2024-03-24T12:47:27.805661+0000","last_undegraded":"2024-03-24T12:47:27.805661+0000","last_fullsized":"2024-03-24T12:47:27.805661+0000","mapping_epoch":382,"log_start":"394'2673","ondisk_log_start":"394'2673","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3903,"log_dups_size":0,"ondisk_log_size":3903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415498240,"num_objects":6340,"num_object_clones":0,"num_object_copies":19020,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6340,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6576,"num_write_kb":413312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"390'30","reported_seq":873,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.255090+0000","last_change":"2024-03-24T12:46:09.601895+0000","last_active":"2024-03-24T12:47:26.255090+0000","last_peered":"2024-03-24T12:47:26.255090+0000","last_clean":"2024-03-24T12:47:26.255090+0000","last_became_active":"2024-03-24T12:45:13.265666+0000","last_became_peered":"2024-03-24T12:45:13.265666+0000","last_unstale":"2024-03-24T12:47:26.255090+0000","last_undegraded":"2024-03-24T12:47:26.255090+0000","last_fullsized":"2024-03-24T12:47:26.255090+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:46:09.601792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:09.601792+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:27.980252+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":81,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.261446+0000","last_change":"2024-03-24T12:45:57.127400+0000","last_active":"2024-03-24T12:47:26.261446+0000","last_peered":"2024-03-24T12:47:26.261446+0000","last_clean":"2024-03-24T12:47:26.261446+0000","last_became_active":"2024-03-24T12:45:40.250678+0000","last_became_peered":"2024-03-24T12:45:40.250678+0000","last_unstale":"2024-03-24T12:47:26.261446+0000","last_undegraded":"2024-03-24T12:47:26.261446+0000","last_fullsized":"2024-03-24T12:47:26.261446+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:57.127283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:57.127283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"354'10","reported_seq":1014,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:18.466439+0000","last_change":"2024-03-24T12:45:06.867846+0000","last_active":"2024-03-24T12:47:18.466439+0000","last_peered":"2024-03-24T12:47:18.466439+0000","last_clean":"2024-03-24T12:47:18.466439+0000","last_became_active":"2024-03-24T12:44:48.938189+0000","last_became_peered":"2024-03-24T12:44:48.938189+0000","last_unstale":"2024-03-24T12:47:18.466439+0000","last_undegraded":"2024-03-24T12:47:18.466439+0000","last_fullsized":"2024-03-24T12:47:18.466439+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'10","last_scrub_stamp":"2024-03-24T12:45:06.867746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:06.867746+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:25.684501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":766,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259506+0000","last_change":"2024-03-24T12:46:01.482058+0000","last_active":"2024-03-24T12:47:26.259506+0000","last_peered":"2024-03-24T12:47:26.259506+0000","last_clean":"2024-03-24T12:47:26.259506+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:47:26.259506+0000","last_undegraded":"2024-03-24T12:47:26.259506+0000","last_fullsized":"2024-03-24T12:47:26.259506+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:01.481956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:42:45.013715+0000","last_clean_scrub_stamp":"2024-03-24T12:46:01.481956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.433495+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1079,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256831+0000","last_change":"2024-03-24T12:45:52.951527+0000","last_active":"2024-03-24T12:47:26.256831+0000","last_peered":"2024-03-24T12:47:26.256831+0000","last_clean":"2024-03-24T12:47:26.256831+0000","last_became_active":"2024-03-24T12:41:52.953388+0000","last_became_peered":"2024-03-24T12:41:52.953388+0000","last_unstale":"2024-03-24T12:47:26.256831+0000","last_undegraded":"2024-03-24T12:47:26.256831+0000","last_fullsized":"2024-03-24T12:47:26.256831+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:45:52.951428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:45:52.951428+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.449078+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.2","version":"408'6419","reported_seq":6481,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.840218+0000","last_change":"2024-03-24T12:45:43.229931+0000","last_active":"2024-03-24T12:47:24.840218+0000","last_peered":"2024-03-24T12:47:24.840218+0000","last_clean":"2024-03-24T12:47:24.840218+0000","last_became_active":"2024-03-24T12:45:43.229611+0000","last_became_peered":"2024-03-24T12:45:43.229611+0000","last_unstale":"2024-03-24T12:47:24.840218+0000","last_undegraded":"2024-03-24T12:47:24.840218+0000","last_fullsized":"2024-03-24T12:47:24.840218+0000","mapping_epoch":382,"log_start":"394'2800","ondisk_log_start":"394'2800","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3619,"log_dups_size":0,"ondisk_log_size":3619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:25.693880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420675584,"num_objects":6419,"num_object_clones":0,"num_object_copies":19257,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6419,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6419,"num_write_kb":410816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"390'53","reported_seq":852,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.255010+0000","last_change":"2024-03-24T12:46:23.746624+0000","last_active":"2024-03-24T12:47:26.255010+0000","last_peered":"2024-03-24T12:47:26.255010+0000","last_clean":"2024-03-24T12:47:26.255010+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:47:26.255010+0000","last_undegraded":"2024-03-24T12:47:26.255010+0000","last_fullsized":"2024-03-24T12:47:26.255010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'53","last_scrub_stamp":"2024-03-24T12:46:23.746506+0000","last_deep_scrub":"207'39","last_deep_scrub_stamp":"2024-03-24T12:36:22.669686+0000","last_clean_scrub_stamp":"2024-03-24T12:46:23.746506+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:41.502009+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":144,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254017+0000","last_change":"2024-03-24T12:45:13.258278+0000","last_active":"2024-03-24T12:47:26.254017+0000","last_peered":"2024-03-24T12:47:26.254017+0000","last_clean":"2024-03-24T12:47:26.254017+0000","last_became_active":"2024-03-24T12:45:13.257987+0000","last_became_peered":"2024-03-24T12:45:13.257987+0000","last_unstale":"2024-03-24T12:47:26.254017+0000","last_undegraded":"2024-03-24T12:47:26.254017+0000","last_fullsized":"2024-03-24T12:47:26.254017+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:38.805830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:44:38.805830+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":363,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254964+0000","last_change":"2024-03-24T12:45:12.448648+0000","last_active":"2024-03-24T12:47:26.254964+0000","last_peered":"2024-03-24T12:47:26.254964+0000","last_clean":"2024-03-24T12:47:26.254964+0000","last_became_active":"2024-03-24T12:39:07.753551+0000","last_became_peered":"2024-03-24T12:39:07.753551+0000","last_unstale":"2024-03-24T12:47:26.254964+0000","last_undegraded":"2024-03-24T12:47:26.254964+0000","last_fullsized":"2024-03-24T12:47:26.254964+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:12.448516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:12.448516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":776,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.427426+0000","last_change":"2024-03-24T12:46:20.834095+0000","last_active":"2024-03-24T12:47:05.427426+0000","last_peered":"2024-03-24T12:47:05.427426+0000","last_clean":"2024-03-24T12:47:05.427426+0000","last_became_active":"2024-03-24T12:45:40.248552+0000","last_became_peered":"2024-03-24T12:45:40.248552+0000","last_unstale":"2024-03-24T12:47:05.427426+0000","last_undegraded":"2024-03-24T12:47:05.427426+0000","last_fullsized":"2024-03-24T12:47:05.427426+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:20.833956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:53.627010+0000","last_clean_scrub_stamp":"2024-03-24T12:46:20.833956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:46.599490+0000","scrub_duration":265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":590,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.288293+0000","last_change":"2024-03-24T12:46:12.525677+0000","last_active":"2024-03-24T12:47:05.288293+0000","last_peered":"2024-03-24T12:47:05.288293+0000","last_clean":"2024-03-24T12:47:05.288293+0000","last_became_active":"2024-03-24T12:45:43.243436+0000","last_became_peered":"2024-03-24T12:45:43.243436+0000","last_unstale":"2024-03-24T12:47:05.288293+0000","last_undegraded":"2024-03-24T12:47:05.288293+0000","last_fullsized":"2024-03-24T12:47:05.288293+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:46:12.525399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:46:12.525399+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:23.447639+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.1","version":"410'6733","reported_seq":6797,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.980817+0000","last_change":"2024-03-24T12:45:43.236083+0000","last_active":"2024-03-24T12:47:27.980817+0000","last_peered":"2024-03-24T12:47:27.980817+0000","last_clean":"2024-03-24T12:47:27.980817+0000","last_became_active":"2024-03-24T12:45:43.235777+0000","last_became_peered":"2024-03-24T12:45:43.235777+0000","last_unstale":"2024-03-24T12:47:27.980817+0000","last_undegraded":"2024-03-24T12:47:27.980817+0000","last_fullsized":"2024-03-24T12:47:27.980817+0000","mapping_epoch":382,"log_start":"397'3500","ondisk_log_start":"397'3500","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3233,"log_dups_size":0,"ondisk_log_size":3233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422641664,"num_objects":6449,"num_object_clones":0,"num_object_copies":19347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6449,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":421824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"390'27","reported_seq":881,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259129+0000","last_change":"2024-03-24T12:45:40.289168+0000","last_active":"2024-03-24T12:47:26.259129+0000","last_peered":"2024-03-24T12:47:26.259129+0000","last_clean":"2024-03-24T12:47:26.259129+0000","last_became_active":"2024-03-24T12:45:40.249367+0000","last_became_peered":"2024-03-24T12:45:40.249367+0000","last_unstale":"2024-03-24T12:47:26.259129+0000","last_undegraded":"2024-03-24T12:47:26.259129+0000","last_fullsized":"2024-03-24T12:47:26.259129+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:44:41.541329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:44:41.541329+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912113+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":770,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253852+0000","last_change":"2024-03-24T12:46:18.685653+0000","last_active":"2024-03-24T12:47:26.253852+0000","last_peered":"2024-03-24T12:47:26.253852+0000","last_clean":"2024-03-24T12:47:26.253852+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:47:26.253852+0000","last_undegraded":"2024-03-24T12:47:26.253852+0000","last_fullsized":"2024-03-24T12:47:26.253852+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:18.685541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:18.685541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:46.514317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":31,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.294261+0000","last_change":"2024-03-24T12:46:46.944940+0000","last_active":"2024-03-24T12:47:26.294261+0000","last_peered":"2024-03-24T12:47:26.294261+0000","last_clean":"2024-03-24T12:47:26.294261+0000","last_became_active":"2024-03-24T12:46:46.944569+0000","last_became_peered":"2024-03-24T12:46:46.944569+0000","last_unstale":"2024-03-24T12:47:26.294261+0000","last_undegraded":"2024-03-24T12:47:26.294261+0000","last_fullsized":"2024-03-24T12:47:26.294261+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:36.501921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:36.501921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.137352+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":741,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257097+0000","last_change":"2024-03-24T12:46:07.891080+0000","last_active":"2024-03-24T12:47:26.257097+0000","last_peered":"2024-03-24T12:47:26.257097+0000","last_clean":"2024-03-24T12:47:26.257097+0000","last_became_active":"2024-03-24T12:45:40.244433+0000","last_became_peered":"2024-03-24T12:45:40.244433+0000","last_unstale":"2024-03-24T12:47:26.257097+0000","last_undegraded":"2024-03-24T12:47:26.257097+0000","last_fullsized":"2024-03-24T12:47:26.257097+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:07.890930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:07.890930+0000","last_clean_scrub_stamp":"2024-03-24T12:46:07.890930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.515746+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.0","version":"410'6866","reported_seq":6931,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.848140+0000","last_change":"2024-03-24T12:45:43.242575+0000","last_active":"2024-03-24T12:47:27.848140+0000","last_peered":"2024-03-24T12:47:27.848140+0000","last_clean":"2024-03-24T12:47:27.848140+0000","last_became_active":"2024-03-24T12:45:43.242256+0000","last_became_peered":"2024-03-24T12:45:43.242256+0000","last_unstale":"2024-03-24T12:47:27.848140+0000","last_undegraded":"2024-03-24T12:47:27.848140+0000","last_fullsized":"2024-03-24T12:47:27.848140+0000","mapping_epoch":382,"log_start":"394'2907","ondisk_log_start":"394'2907","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3959,"log_dups_size":0,"ondisk_log_size":3959,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430440448,"num_objects":6568,"num_object_clones":0,"num_object_copies":19704,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6568,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6866,"num_write_kb":429888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"391'46","reported_seq":821,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.427687+0000","last_change":"2024-03-24T12:46:09.424418+0000","last_active":"2024-03-24T12:47:05.427687+0000","last_peered":"2024-03-24T12:47:05.427687+0000","last_clean":"2024-03-24T12:47:05.427687+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:47:05.427687+0000","last_undegraded":"2024-03-24T12:47:05.427687+0000","last_fullsized":"2024-03-24T12:47:05.427687+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-24T12:46:09.424281+0000","last_deep_scrub":"92'22","last_deep_scrub_stamp":"2024-03-24T12:27:42.645969+0000","last_clean_scrub_stamp":"2024-03-24T12:46:09.424281+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:32.753447+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":79,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.992976+0000","last_change":"2024-03-24T12:45:58.406476+0000","last_active":"2024-03-24T12:47:24.992976+0000","last_peered":"2024-03-24T12:47:24.992976+0000","last_clean":"2024-03-24T12:47:24.992976+0000","last_became_active":"2024-03-24T12:45:40.244230+0000","last_became_peered":"2024-03-24T12:45:40.244230+0000","last_unstale":"2024-03-24T12:47:24.992976+0000","last_undegraded":"2024-03-24T12:47:24.992976+0000","last_fullsized":"2024-03-24T12:47:24.992976+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:58.406348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:58.406348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"354'11","reported_seq":1019,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259404+0000","last_change":"2024-03-24T12:44:56.632534+0000","last_active":"2024-03-24T12:47:26.259404+0000","last_peered":"2024-03-24T12:47:26.259404+0000","last_clean":"2024-03-24T12:47:26.259404+0000","last_became_active":"2024-03-24T12:44:48.870472+0000","last_became_peered":"2024-03-24T12:44:48.870472+0000","last_unstale":"2024-03-24T12:47:26.259404+0000","last_undegraded":"2024-03-24T12:47:26.259404+0000","last_fullsized":"2024-03-24T12:47:26.259404+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'11","last_scrub_stamp":"2024-03-24T12:44:56.632417+0000","last_deep_scrub":"140'9","last_deep_scrub_stamp":"2024-03-24T12:31:21.743220+0000","last_clean_scrub_stamp":"2024-03-24T12:44:56.632417+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912356+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":444,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.994623+0000","last_change":"2024-03-24T12:46:28.074580+0000","last_active":"2024-03-24T12:47:24.994623+0000","last_peered":"2024-03-24T12:47:24.994623+0000","last_clean":"2024-03-24T12:47:24.994623+0000","last_became_active":"2024-03-24T12:44:48.876647+0000","last_became_peered":"2024-03-24T12:44:48.876647+0000","last_unstale":"2024-03-24T12:47:24.994623+0000","last_undegraded":"2024-03-24T12:47:24.994623+0000","last_fullsized":"2024-03-24T12:47:24.994623+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:28.074463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:00.198109+0000","last_clean_scrub_stamp":"2024-03-24T12:46:28.074463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:47.436641+0000","scrub_duration":172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":562,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.439729+0000","last_change":"2024-03-24T12:45:16.221400+0000","last_active":"2024-03-24T12:47:05.439729+0000","last_peered":"2024-03-24T12:47:05.439729+0000","last_clean":"2024-03-24T12:47:05.439729+0000","last_became_active":"2024-03-24T12:45:16.220608+0000","last_became_peered":"2024-03-24T12:45:16.220608+0000","last_unstale":"2024-03-24T12:47:05.439729+0000","last_undegraded":"2024-03-24T12:47:05.439729+0000","last_fullsized":"2024-03-24T12:47:05.439729+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:44:34.685258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:44:34.685258+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:05.587763+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.f","version":"410'6444","reported_seq":6581,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.822321+0000","last_change":"2024-03-24T12:47:27.241395+0000","last_active":"2024-03-24T12:47:27.822321+0000","last_peered":"2024-03-24T12:47:27.822321+0000","last_clean":"2024-03-24T12:47:27.822321+0000","last_became_active":"2024-03-24T12:45:43.235407+0000","last_became_peered":"2024-03-24T12:45:43.235407+0000","last_unstale":"2024-03-24T12:47:27.822321+0000","last_undegraded":"2024-03-24T12:47:27.822321+0000","last_fullsized":"2024-03-24T12:47:27.822321+0000","mapping_epoch":382,"log_start":"393'2505","ondisk_log_start":"393'2505","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"410'6404","last_scrub_stamp":"2024-03-24T12:47:27.241171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:47:27.241171+0000","objects_scrubbed":6288,"log_size":3939,"log_dups_size":0,"ondisk_log_size":3939,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:48:54.989187+0000","scrub_duration":1726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":406847488,"num_objects":6208,"num_object_clones":0,"num_object_copies":18624,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6208,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6444,"num_write_kb":404864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":783,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253670+0000","last_change":"2024-03-24T12:46:21.688315+0000","last_active":"2024-03-24T12:47:26.253670+0000","last_peered":"2024-03-24T12:47:26.253670+0000","last_clean":"2024-03-24T12:47:26.253670+0000","last_became_active":"2024-03-24T12:45:40.232411+0000","last_became_peered":"2024-03-24T12:45:40.232411+0000","last_unstale":"2024-03-24T12:47:26.253670+0000","last_undegraded":"2024-03-24T12:47:26.253670+0000","last_fullsized":"2024-03-24T12:47:26.253670+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:21.688184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:21.688184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:48.313281+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"390'42","reported_seq":479,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.260156+0000","last_change":"2024-03-24T12:45:53.036097+0000","last_active":"2024-03-24T12:47:26.260156+0000","last_peered":"2024-03-24T12:47:26.260156+0000","last_clean":"2024-03-24T12:47:26.260156+0000","last_became_active":"2024-03-24T12:45:40.256722+0000","last_became_peered":"2024-03-24T12:45:40.256722+0000","last_unstale":"2024-03-24T12:47:26.260156+0000","last_undegraded":"2024-03-24T12:47:26.260156+0000","last_fullsized":"2024-03-24T12:47:26.260156+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'30","last_scrub_stamp":"2024-03-24T12:45:53.035945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:53.035945+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":754,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.294374+0000","last_change":"2024-03-24T12:45:56.018473+0000","last_active":"2024-03-24T12:47:26.294374+0000","last_peered":"2024-03-24T12:47:26.294374+0000","last_clean":"2024-03-24T12:47:26.294374+0000","last_became_active":"2024-03-24T12:26:03.160738+0000","last_became_peered":"2024-03-24T12:26:03.160738+0000","last_unstale":"2024-03-24T12:47:26.294374+0000","last_undegraded":"2024-03-24T12:47:26.294374+0000","last_fullsized":"2024-03-24T12:47:26.294374+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:56.018335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:56.018335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":765,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253923+0000","last_change":"2024-03-24T12:46:10.560869+0000","last_active":"2024-03-24T12:47:26.253923+0000","last_peered":"2024-03-24T12:47:26.253923+0000","last_clean":"2024-03-24T12:47:26.253923+0000","last_became_active":"2024-03-24T12:45:40.227539+0000","last_became_peered":"2024-03-24T12:45:40.227539+0000","last_unstale":"2024-03-24T12:47:26.253923+0000","last_undegraded":"2024-03-24T12:47:26.253923+0000","last_fullsized":"2024-03-24T12:47:26.253923+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:10.560773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:10.560773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912586+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":566,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253791+0000","last_change":"2024-03-24T12:46:34.947261+0000","last_active":"2024-03-24T12:47:26.253791+0000","last_peered":"2024-03-24T12:47:26.253791+0000","last_clean":"2024-03-24T12:47:26.253791+0000","last_became_active":"2024-03-24T12:45:16.220449+0000","last_became_peered":"2024-03-24T12:45:16.220449+0000","last_unstale":"2024-03-24T12:47:26.253791+0000","last_undegraded":"2024-03-24T12:47:26.253791+0000","last_fullsized":"2024-03-24T12:47:26.253791+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:46:34.947149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:46:34.947149+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:57.435937+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"410'6747","reported_seq":6813,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.681088+0000","last_change":"2024-03-24T12:45:43.227598+0000","last_active":"2024-03-24T12:47:27.681088+0000","last_peered":"2024-03-24T12:47:27.681088+0000","last_clean":"2024-03-24T12:47:27.681088+0000","last_became_active":"2024-03-24T12:45:43.227183+0000","last_became_peered":"2024-03-24T12:45:43.227183+0000","last_unstale":"2024-03-24T12:47:27.681088+0000","last_undegraded":"2024-03-24T12:47:27.681088+0000","last_fullsized":"2024-03-24T12:47:27.681088+0000","mapping_epoch":382,"log_start":"396'3400","ondisk_log_start":"396'3400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3347,"log_dups_size":0,"ondisk_log_size":3347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425787392,"num_objects":6497,"num_object_clones":0,"num_object_copies":19491,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6497,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6747,"num_write_kb":423808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"380'9","reported_seq":129,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.993587+0000","last_change":"2024-03-24T12:46:17.715550+0000","last_active":"2024-03-24T12:47:24.993587+0000","last_peered":"2024-03-24T12:47:24.993587+0000","last_clean":"2024-03-24T12:47:24.993587+0000","last_became_active":"2024-03-24T12:45:40.248941+0000","last_became_peered":"2024-03-24T12:45:40.248941+0000","last_unstale":"2024-03-24T12:47:24.993587+0000","last_undegraded":"2024-03-24T12:47:24.993587+0000","last_fullsized":"2024-03-24T12:47:24.993587+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'9","last_scrub_stamp":"2024-03-24T12:46:17.715400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:17.715400+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:43.926057+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"390'49","reported_seq":894,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254009+0000","last_change":"2024-03-24T12:46:33.905752+0000","last_active":"2024-03-24T12:47:26.254009+0000","last_peered":"2024-03-24T12:47:26.254009+0000","last_clean":"2024-03-24T12:47:26.254009+0000","last_became_active":"2024-03-24T12:45:13.309157+0000","last_became_peered":"2024-03-24T12:45:13.309157+0000","last_unstale":"2024-03-24T12:47:26.254009+0000","last_undegraded":"2024-03-24T12:47:26.254009+0000","last_fullsized":"2024-03-24T12:47:26.254009+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'49","last_scrub_stamp":"2024-03-24T12:46:33.905630+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:46:33.905630+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:45.309099+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":779,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.260019+0000","last_change":"2024-03-24T12:46:32.466755+0000","last_active":"2024-03-24T12:47:26.260019+0000","last_peered":"2024-03-24T12:47:26.260019+0000","last_clean":"2024-03-24T12:47:26.260019+0000","last_became_active":"2024-03-24T12:26:03.161658+0000","last_became_peered":"2024-03-24T12:26:03.161658+0000","last_unstale":"2024-03-24T12:47:26.260019+0000","last_undegraded":"2024-03-24T12:47:26.260019+0000","last_fullsized":"2024-03-24T12:47:26.260019+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:46:32.466631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:32.466631+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:49.392397+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":777,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256563+0000","last_change":"2024-03-24T12:46:03.105883+0000","last_active":"2024-03-24T12:47:26.256563+0000","last_peered":"2024-03-24T12:47:26.256563+0000","last_clean":"2024-03-24T12:47:26.256563+0000","last_became_active":"2024-03-24T12:45:13.313559+0000","last_became_peered":"2024-03-24T12:45:13.313559+0000","last_unstale":"2024-03-24T12:47:26.256563+0000","last_undegraded":"2024-03-24T12:47:26.256563+0000","last_fullsized":"2024-03-24T12:47:26.256563+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:03.105774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:03.105774+0000","last_clean_scrub_stamp":"2024-03-24T12:46:03.105774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.449011+0000","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"83'128","reported_seq":898,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257055+0000","last_change":"2024-03-24T12:45:48.880834+0000","last_active":"2024-03-24T12:47:26.257055+0000","last_peered":"2024-03-24T12:47:26.257055+0000","last_clean":"2024-03-24T12:47:26.257055+0000","last_became_active":"2024-03-24T12:45:43.234521+0000","last_became_peered":"2024-03-24T12:45:43.234521+0000","last_unstale":"2024-03-24T12:47:26.257055+0000","last_undegraded":"2024-03-24T12:47:26.257055+0000","last_fullsized":"2024-03-24T12:47:26.257055+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:45:48.880729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:45:48.880729+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.d","version":"408'6455","reported_seq":6517,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.843131+0000","last_change":"2024-03-24T12:45:43.236122+0000","last_active":"2024-03-24T12:47:24.843131+0000","last_peered":"2024-03-24T12:47:24.843131+0000","last_clean":"2024-03-24T12:47:24.843131+0000","last_became_active":"2024-03-24T12:45:43.235816+0000","last_became_peered":"2024-03-24T12:45:43.235816+0000","last_unstale":"2024-03-24T12:47:24.843131+0000","last_undegraded":"2024-03-24T12:47:24.843131+0000","last_fullsized":"2024-03-24T12:47:24.843131+0000","mapping_epoch":382,"log_start":"394'2900","ondisk_log_start":"394'2900","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3555,"log_dups_size":0,"ondisk_log_size":3555,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:25.686628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423034880,"num_objects":6455,"num_object_clones":0,"num_object_copies":19365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6455,"num_write_kb":413120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,1],"acting":[2,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3","version":"370'16","reported_seq":1058,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.261220+0000","last_change":"2024-03-24T12:45:54.070898+0000","last_active":"2024-03-24T12:47:26.261220+0000","last_peered":"2024-03-24T12:47:26.261220+0000","last_clean":"2024-03-24T12:47:26.261220+0000","last_became_active":"2024-03-24T12:45:13.320893+0000","last_became_peered":"2024-03-24T12:45:13.320893+0000","last_unstale":"2024-03-24T12:47:26.261220+0000","last_undegraded":"2024-03-24T12:47:26.261220+0000","last_fullsized":"2024-03-24T12:47:26.261220+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'16","last_scrub_stamp":"2024-03-24T12:45:54.070804+0000","last_deep_scrub":"318'14","last_deep_scrub_stamp":"2024-03-24T12:42:45.093908+0000","last_clean_scrub_stamp":"2024-03-24T12:45:54.070804+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"390'35","reported_seq":820,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:25.002386+0000","last_change":"2024-03-24T12:46:19.770424+0000","last_active":"2024-03-24T12:47:25.002386+0000","last_peered":"2024-03-24T12:47:25.002386+0000","last_clean":"2024-03-24T12:47:25.002386+0000","last_became_active":"2024-03-24T12:44:48.870632+0000","last_became_peered":"2024-03-24T12:44:48.870632+0000","last_unstale":"2024-03-24T12:47:25.002386+0000","last_undegraded":"2024-03-24T12:47:25.002386+0000","last_fullsized":"2024-03-24T12:47:25.002386+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'35","last_scrub_stamp":"2024-03-24T12:46:19.770287+0000","last_deep_scrub":"138'15","last_deep_scrub_stamp":"2024-03-24T12:31:12.053497+0000","last_clean_scrub_stamp":"2024-03-24T12:46:19.770287+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:37.559235+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":430,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.425564+0000","last_change":"2024-03-24T12:46:21.642486+0000","last_active":"2024-03-24T12:47:05.425564+0000","last_peered":"2024-03-24T12:47:05.425564+0000","last_clean":"2024-03-24T12:47:05.425564+0000","last_became_active":"2024-03-24T12:45:40.245717+0000","last_became_peered":"2024-03-24T12:45:40.245717+0000","last_unstale":"2024-03-24T12:47:05.425564+0000","last_undegraded":"2024-03-24T12:47:05.425564+0000","last_fullsized":"2024-03-24T12:47:05.425564+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:21.642368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:21.642368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:23.741548+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":765,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259431+0000","last_change":"2024-03-24T12:45:55.103722+0000","last_active":"2024-03-24T12:47:26.259431+0000","last_peered":"2024-03-24T12:47:26.259431+0000","last_clean":"2024-03-24T12:47:26.259431+0000","last_became_active":"2024-03-24T12:45:40.257194+0000","last_became_peered":"2024-03-24T12:45:40.257194+0000","last_unstale":"2024-03-24T12:47:26.259431+0000","last_undegraded":"2024-03-24T12:47:26.259431+0000","last_fullsized":"2024-03-24T12:47:26.259431+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:55.103636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:45:55.103636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":894,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257491+0000","last_change":"2024-03-24T12:45:49.875212+0000","last_active":"2024-03-24T12:47:26.257491+0000","last_peered":"2024-03-24T12:47:26.257491+0000","last_clean":"2024-03-24T12:47:26.257491+0000","last_became_active":"2024-03-24T12:45:43.235500+0000","last_became_peered":"2024-03-24T12:45:43.235500+0000","last_unstale":"2024-03-24T12:47:26.257491+0000","last_undegraded":"2024-03-24T12:47:26.257491+0000","last_fullsized":"2024-03-24T12:47:26.257491+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:45:49.875065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:45:49.875065+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448887+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.c","version":"409'6445","reported_seq":6508,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:25.169086+0000","last_change":"2024-03-24T12:45:43.234493+0000","last_active":"2024-03-24T12:47:25.169086+0000","last_peered":"2024-03-24T12:47:25.169086+0000","last_clean":"2024-03-24T12:47:25.169086+0000","last_became_active":"2024-03-24T12:45:43.234204+0000","last_became_peered":"2024-03-24T12:45:43.234204+0000","last_unstale":"2024-03-24T12:47:25.169086+0000","last_undegraded":"2024-03-24T12:47:25.169086+0000","last_fullsized":"2024-03-24T12:47:25.169086+0000","mapping_epoch":382,"log_start":"397'3500","ondisk_log_start":"397'3500","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":2945,"log_dups_size":2576,"ondisk_log_size":2945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422379520,"num_objects":6445,"num_object_clones":0,"num_object_copies":19335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6445,"num_write_kb":412480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":702,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.891735+0000","last_change":"2024-03-24T12:47:27.891774+0000","last_active":"2024-03-24T12:47:27.891735+0000","last_peered":"2024-03-24T12:47:27.891735+0000","last_clean":"2024-03-24T12:47:27.891735+0000","last_became_active":"2024-03-24T12:45:13.323975+0000","last_became_peered":"2024-03-24T12:45:13.323975+0000","last_unstale":"2024-03-24T12:47:27.891735+0000","last_undegraded":"2024-03-24T12:47:27.891735+0000","last_fullsized":"2024-03-24T12:47:27.891735+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:47:27.891611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.763326+0000","last_clean_scrub_stamp":"2024-03-24T12:47:27.891611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:48:41.359401+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"390'47","reported_seq":836,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.261612+0000","last_change":"2024-03-24T12:44:52.923795+0000","last_active":"2024-03-24T12:47:26.261612+0000","last_peered":"2024-03-24T12:47:26.261612+0000","last_clean":"2024-03-24T12:47:26.261612+0000","last_became_active":"2024-03-24T12:44:48.875042+0000","last_became_peered":"2024-03-24T12:44:48.875042+0000","last_unstale":"2024-03-24T12:47:26.261612+0000","last_undegraded":"2024-03-24T12:47:26.261612+0000","last_fullsized":"2024-03-24T12:47:26.261612+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'37","last_scrub_stamp":"2024-03-24T12:44:52.923646+0000","last_deep_scrub":"195'27","last_deep_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_clean_scrub_stamp":"2024-03-24T12:44:52.923646+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":666,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.428387+0000","last_change":"2024-03-24T12:44:57.808299+0000","last_active":"2024-03-24T12:47:05.428387+0000","last_peered":"2024-03-24T12:47:05.428387+0000","last_clean":"2024-03-24T12:47:05.428387+0000","last_became_active":"2024-03-24T12:44:48.887828+0000","last_became_peered":"2024-03-24T12:44:48.887828+0000","last_unstale":"2024-03-24T12:47:05.428387+0000","last_undegraded":"2024-03-24T12:47:05.428387+0000","last_fullsized":"2024-03-24T12:47:05.428387+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:57.808197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:44:57.808197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:05.587904+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":791,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.994443+0000","last_change":"2024-03-24T12:46:21.792474+0000","last_active":"2024-03-24T12:47:24.994443+0000","last_peered":"2024-03-24T12:47:24.994443+0000","last_clean":"2024-03-24T12:47:24.994443+0000","last_became_active":"2024-03-24T12:45:13.300587+0000","last_became_peered":"2024-03-24T12:45:13.300587+0000","last_unstale":"2024-03-24T12:47:24.994443+0000","last_undegraded":"2024-03-24T12:47:24.994443+0000","last_fullsized":"2024-03-24T12:47:24.994443+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:21.792342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:21.792342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:43.285526+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":407,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259500+0000","last_change":"2024-03-24T12:45:49.950311+0000","last_active":"2024-03-24T12:47:26.259500+0000","last_peered":"2024-03-24T12:47:26.259500+0000","last_clean":"2024-03-24T12:47:26.259500+0000","last_became_active":"2024-03-24T12:45:43.231412+0000","last_became_peered":"2024-03-24T12:45:43.231412+0000","last_unstale":"2024-03-24T12:47:26.259500+0000","last_undegraded":"2024-03-24T12:47:26.259500+0000","last_fullsized":"2024-03-24T12:47:26.259500+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:45:49.950231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:45:49.950231+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"410'6639","reported_seq":6705,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.987844+0000","last_change":"2024-03-24T12:45:43.241315+0000","last_active":"2024-03-24T12:47:27.987844+0000","last_peered":"2024-03-24T12:47:27.987844+0000","last_clean":"2024-03-24T12:47:27.987844+0000","last_became_active":"2024-03-24T12:45:43.240963+0000","last_became_peered":"2024-03-24T12:45:43.240963+0000","last_unstale":"2024-03-24T12:47:27.987844+0000","last_undegraded":"2024-03-24T12:47:27.987844+0000","last_fullsized":"2024-03-24T12:47:27.987844+0000","mapping_epoch":382,"log_start":"396'3400","ondisk_log_start":"396'3400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3239,"log_dups_size":0,"ondisk_log_size":3239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416874496,"num_objects":6361,"num_object_clones":0,"num_object_copies":19083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6639,"num_write_kb":416000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":437,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.994248+0000","last_change":"2024-03-24T12:45:11.355222+0000","last_active":"2024-03-24T12:47:24.994248+0000","last_peered":"2024-03-24T12:47:24.994248+0000","last_clean":"2024-03-24T12:47:24.994248+0000","last_became_active":"2024-03-24T12:44:48.869674+0000","last_became_peered":"2024-03-24T12:44:48.869674+0000","last_unstale":"2024-03-24T12:47:24.994248+0000","last_undegraded":"2024-03-24T12:47:24.994248+0000","last_fullsized":"2024-03-24T12:47:24.994248+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:11.355123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:11.355123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"390'42","reported_seq":827,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259782+0000","last_change":"2024-03-24T12:46:05.173807+0000","last_active":"2024-03-24T12:47:26.259782+0000","last_peered":"2024-03-24T12:47:26.259782+0000","last_clean":"2024-03-24T12:47:26.259782+0000","last_became_active":"2024-03-24T12:26:05.314274+0000","last_became_peered":"2024-03-24T12:26:05.314274+0000","last_unstale":"2024-03-24T12:47:26.259782+0000","last_undegraded":"2024-03-24T12:47:26.259782+0000","last_fullsized":"2024-03-24T12:47:26.259782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'30","last_scrub_stamp":"2024-03-24T12:46:05.173517+0000","last_deep_scrub":"260'30","last_deep_scrub_stamp":"2024-03-24T12:39:51.288276+0000","last_clean_scrub_stamp":"2024-03-24T12:46:05.173517+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2065,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256149+0000","last_change":"2024-03-24T12:45:08.898913+0000","last_active":"2024-03-24T12:47:26.256149+0000","last_peered":"2024-03-24T12:47:26.256149+0000","last_clean":"2024-03-24T12:47:26.256149+0000","last_became_active":"2024-03-24T12:26:03.162971+0000","last_became_peered":"2024-03-24T12:26:03.162971+0000","last_unstale":"2024-03-24T12:47:26.256149+0000","last_undegraded":"2024-03-24T12:47:26.256149+0000","last_fullsized":"2024-03-24T12:47:26.256149+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:45:08.898795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:08.898795+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448592+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1276,"num_read_kb":1271,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":785,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.268102+0000","last_change":"2024-03-24T12:46:14.118009+0000","last_active":"2024-03-24T12:47:26.268102+0000","last_peered":"2024-03-24T12:47:26.268102+0000","last_clean":"2024-03-24T12:47:26.268102+0000","last_became_active":"2024-03-24T12:45:40.239552+0000","last_became_peered":"2024-03-24T12:45:40.239552+0000","last_unstale":"2024-03-24T12:47:26.268102+0000","last_undegraded":"2024-03-24T12:47:26.268102+0000","last_fullsized":"2024-03-24T12:47:26.268102+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:14.117633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:13.629968+0000","last_clean_scrub_stamp":"2024-03-24T12:46:14.117633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:34.104347+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":762,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:25.003015+0000","last_change":"2024-03-24T12:45:40.253070+0000","last_active":"2024-03-24T12:47:25.003015+0000","last_peered":"2024-03-24T12:47:25.003015+0000","last_clean":"2024-03-24T12:47:25.003015+0000","last_became_active":"2024-03-24T12:45:40.252202+0000","last_became_peered":"2024-03-24T12:45:40.252202+0000","last_unstale":"2024-03-24T12:47:25.003015+0000","last_undegraded":"2024-03-24T12:47:25.003015+0000","last_fullsized":"2024-03-24T12:47:25.003015+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:03.186807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:03.186807+0000","last_clean_scrub_stamp":"2024-03-24T12:45:03.186807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":806,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254162+0000","last_change":"2024-03-24T12:45:50.557525+0000","last_active":"2024-03-24T12:47:26.254162+0000","last_peered":"2024-03-24T12:47:26.254162+0000","last_clean":"2024-03-24T12:47:26.254162+0000","last_became_active":"2024-03-24T12:45:16.221340+0000","last_became_peered":"2024-03-24T12:45:16.221340+0000","last_unstale":"2024-03-24T12:47:26.254162+0000","last_undegraded":"2024-03-24T12:47:26.254162+0000","last_fullsized":"2024-03-24T12:47:26.254162+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:45:50.557414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:45:50.557414+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912488+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.a","version":"410'6750","reported_seq":6815,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.928092+0000","last_change":"2024-03-24T12:45:43.228195+0000","last_active":"2024-03-24T12:47:27.928092+0000","last_peered":"2024-03-24T12:47:27.928092+0000","last_clean":"2024-03-24T12:47:27.928092+0000","last_became_active":"2024-03-24T12:45:43.227875+0000","last_became_peered":"2024-03-24T12:45:43.227875+0000","last_unstale":"2024-03-24T12:47:27.928092+0000","last_undegraded":"2024-03-24T12:47:27.928092+0000","last_fullsized":"2024-03-24T12:47:27.928092+0000","mapping_epoch":382,"log_start":"0'0","ondisk_log_start":"0'0","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":6750,"log_dups_size":0,"ondisk_log_size":6750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.137391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424673280,"num_objects":6480,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6750,"num_write_kb":423360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":740,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257441+0000","last_change":"2024-03-24T12:44:52.778468+0000","last_active":"2024-03-24T12:47:26.257441+0000","last_peered":"2024-03-24T12:47:26.257441+0000","last_clean":"2024-03-24T12:47:26.257441+0000","last_became_active":"2024-03-24T12:44:48.868097+0000","last_became_peered":"2024-03-24T12:44:48.868097+0000","last_unstale":"2024-03-24T12:47:26.257441+0000","last_undegraded":"2024-03-24T12:47:26.257441+0000","last_fullsized":"2024-03-24T12:47:26.257441+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:52.778341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:44:52.778341+0000","last_clean_scrub_stamp":"2024-03-24T12:44:52.778341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448559+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"390'43","reported_seq":725,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254822+0000","last_change":"2024-03-24T12:45:13.327526+0000","last_active":"2024-03-24T12:47:26.254822+0000","last_peered":"2024-03-24T12:47:26.254822+0000","last_clean":"2024-03-24T12:47:26.254822+0000","last_became_active":"2024-03-24T12:45:13.308927+0000","last_became_peered":"2024-03-24T12:45:13.308927+0000","last_unstale":"2024-03-24T12:47:26.254822+0000","last_undegraded":"2024-03-24T12:47:26.254822+0000","last_fullsized":"2024-03-24T12:47:26.254822+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"258'30","last_scrub_stamp":"2024-03-24T12:44:44.924886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:44:44.924886+0000","objects_scrubbed":8,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912274+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":762,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253440+0000","last_change":"2024-03-24T12:46:22.722964+0000","last_active":"2024-03-24T12:47:26.253440+0000","last_peered":"2024-03-24T12:47:26.253440+0000","last_clean":"2024-03-24T12:47:26.253440+0000","last_became_active":"2024-03-24T12:45:40.249404+0000","last_became_peered":"2024-03-24T12:45:40.249404+0000","last_unstale":"2024-03-24T12:47:26.253440+0000","last_undegraded":"2024-03-24T12:47:26.253440+0000","last_fullsized":"2024-03-24T12:47:26.253440+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:22.722795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:22.722795+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:33.201103+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1081,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256201+0000","last_change":"2024-03-24T12:46:11.041441+0000","last_active":"2024-03-24T12:47:26.256201+0000","last_peered":"2024-03-24T12:47:26.256201+0000","last_clean":"2024-03-24T12:47:26.256201+0000","last_became_active":"2024-03-24T12:45:16.218785+0000","last_became_peered":"2024-03-24T12:45:16.218785+0000","last_unstale":"2024-03-24T12:47:26.256201+0000","last_undegraded":"2024-03-24T12:47:26.256201+0000","last_fullsized":"2024-03-24T12:47:26.256201+0000","mapping_epoch":372,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:46:11.041313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:46:11.041313+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.449044+0000","scrub_duration":209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":446,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.294526+0000","last_change":"2024-03-24T12:46:37.376873+0000","last_active":"2024-03-24T12:47:26.294526+0000","last_peered":"2024-03-24T12:47:26.294526+0000","last_clean":"2024-03-24T12:47:26.294526+0000","last_became_active":"2024-03-24T12:44:48.874876+0000","last_became_peered":"2024-03-24T12:44:48.874876+0000","last_unstale":"2024-03-24T12:47:26.294526+0000","last_undegraded":"2024-03-24T12:47:26.294526+0000","last_fullsized":"2024-03-24T12:47:26.294526+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:37.376732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:37.376732+0000","last_clean_scrub_stamp":"2024-03-24T12:46:37.376732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:40.882367+0000","scrub_duration":288,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"410'6659","reported_seq":6725,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.970297+0000","last_change":"2024-03-24T12:45:43.236500+0000","last_active":"2024-03-24T12:47:27.970297+0000","last_peered":"2024-03-24T12:47:27.970297+0000","last_clean":"2024-03-24T12:47:27.970297+0000","last_became_active":"2024-03-24T12:45:43.231774+0000","last_became_peered":"2024-03-24T12:45:43.231774+0000","last_unstale":"2024-03-24T12:47:27.970297+0000","last_undegraded":"2024-03-24T12:47:27.970297+0000","last_fullsized":"2024-03-24T12:47:27.970297+0000","mapping_epoch":382,"log_start":"396'3400","ondisk_log_start":"396'3400","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3259,"log_dups_size":0,"ondisk_log_size":3259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":420282368,"num_objects":6413,"num_object_clones":0,"num_object_copies":19239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6659,"num_write_kb":418304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":743,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.423893+0000","last_change":"2024-03-24T12:45:00.789166+0000","last_active":"2024-03-24T12:47:05.423893+0000","last_peered":"2024-03-24T12:47:05.423893+0000","last_clean":"2024-03-24T12:47:05.423893+0000","last_became_active":"2024-03-24T12:44:48.874944+0000","last_became_peered":"2024-03-24T12:44:48.874944+0000","last_unstale":"2024-03-24T12:47:05.423893+0000","last_undegraded":"2024-03-24T12:47:05.423893+0000","last_fullsized":"2024-03-24T12:47:05.423893+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:00.788798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:00.788798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:05.587729+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"390'26","reported_seq":813,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.993977+0000","last_change":"2024-03-24T12:46:28.873246+0000","last_active":"2024-03-24T12:47:24.993977+0000","last_peered":"2024-03-24T12:47:24.993977+0000","last_clean":"2024-03-24T12:47:24.993977+0000","last_became_active":"2024-03-24T12:44:48.870820+0000","last_became_peered":"2024-03-24T12:44:48.870820+0000","last_unstale":"2024-03-24T12:47:24.993977+0000","last_undegraded":"2024-03-24T12:47:24.993977+0000","last_fullsized":"2024-03-24T12:47:24.993977+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'26","last_scrub_stamp":"2024-03-24T12:46:28.873107+0000","last_deep_scrub":"207'22","last_deep_scrub_stamp":"2024-03-24T12:44:50.024489+0000","last_clean_scrub_stamp":"2024-03-24T12:46:28.873107+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:33.847919+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":796,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.424448+0000","last_change":"2024-03-24T12:45:48.379164+0000","last_active":"2024-03-24T12:47:05.424448+0000","last_peered":"2024-03-24T12:47:05.424448+0000","last_clean":"2024-03-24T12:47:05.424448+0000","last_became_active":"2024-03-24T12:45:40.246825+0000","last_became_peered":"2024-03-24T12:45:40.246825+0000","last_unstale":"2024-03-24T12:47:05.424448+0000","last_undegraded":"2024-03-24T12:47:05.424448+0000","last_fullsized":"2024-03-24T12:47:05.424448+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:48.379081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:48.379081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:06.547545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":774,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.423377+0000","last_change":"2024-03-24T12:46:24.909961+0000","last_active":"2024-03-24T12:47:05.423377+0000","last_peered":"2024-03-24T12:47:05.423377+0000","last_clean":"2024-03-24T12:47:05.423377+0000","last_became_active":"2024-03-24T12:45:13.313392+0000","last_became_peered":"2024-03-24T12:45:13.313392+0000","last_unstale":"2024-03-24T12:47:05.423377+0000","last_undegraded":"2024-03-24T12:47:05.423377+0000","last_fullsized":"2024-03-24T12:47:05.423377+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:24.909653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:24.909653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:45.786341+0000","scrub_duration":131,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":785,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257699+0000","last_change":"2024-03-24T12:45:13.312221+0000","last_active":"2024-03-24T12:47:26.257699+0000","last_peered":"2024-03-24T12:47:26.257699+0000","last_clean":"2024-03-24T12:47:26.257699+0000","last_became_active":"2024-03-24T12:45:13.311960+0000","last_became_peered":"2024-03-24T12:45:13.311960+0000","last_unstale":"2024-03-24T12:47:26.257699+0000","last_undegraded":"2024-03-24T12:47:26.257699+0000","last_fullsized":"2024-03-24T12:47:26.257699+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:58.882258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:15.087115+0000","last_clean_scrub_stamp":"2024-03-24T12:44:58.882258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448380+0000","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":527,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259350+0000","last_change":"2024-03-24T12:46:16.236685+0000","last_active":"2024-03-24T12:47:26.259350+0000","last_peered":"2024-03-24T12:47:26.259350+0000","last_clean":"2024-03-24T12:47:26.259350+0000","last_became_active":"2024-03-24T12:42:40.200767+0000","last_became_peered":"2024-03-24T12:42:40.200767+0000","last_unstale":"2024-03-24T12:47:26.259350+0000","last_undegraded":"2024-03-24T12:47:26.259350+0000","last_fullsized":"2024-03-24T12:47:26.259350+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:46:16.236536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:46:16.236536+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:30.198700+0000","scrub_duration":200,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"410'6740","reported_seq":6806,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.992049+0000","last_change":"2024-03-24T12:45:43.242202+0000","last_active":"2024-03-24T12:47:27.992049+0000","last_peered":"2024-03-24T12:47:27.992049+0000","last_clean":"2024-03-24T12:47:27.992049+0000","last_became_active":"2024-03-24T12:45:43.241842+0000","last_became_peered":"2024-03-24T12:45:43.241842+0000","last_unstale":"2024-03-24T12:47:27.992049+0000","last_undegraded":"2024-03-24T12:47:27.992049+0000","last_fullsized":"2024-03-24T12:47:27.992049+0000","mapping_epoch":382,"log_start":"396'3500","ondisk_log_start":"396'3500","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3240,"log_dups_size":0,"ondisk_log_size":3240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":421003264,"num_objects":6424,"num_object_clones":0,"num_object_copies":19272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6740,"num_write_kb":421248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":763,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259640+0000","last_change":"2024-03-24T12:46:08.171021+0000","last_active":"2024-03-24T12:47:26.259640+0000","last_peered":"2024-03-24T12:47:26.259640+0000","last_clean":"2024-03-24T12:47:26.259640+0000","last_became_active":"2024-03-24T12:26:07.331239+0000","last_became_peered":"2024-03-24T12:26:07.331239+0000","last_unstale":"2024-03-24T12:47:26.259640+0000","last_undegraded":"2024-03-24T12:47:26.259640+0000","last_fullsized":"2024-03-24T12:47:26.259640+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:08.170846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:08.170846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"390'64","reported_seq":1033,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:18.463479+0000","last_change":"2024-03-24T12:46:03.462041+0000","last_active":"2024-03-24T12:47:18.463479+0000","last_peered":"2024-03-24T12:47:18.463479+0000","last_clean":"2024-03-24T12:47:18.463479+0000","last_became_active":"2024-03-24T12:45:40.246159+0000","last_became_peered":"2024-03-24T12:45:40.246159+0000","last_unstale":"2024-03-24T12:47:18.463479+0000","last_undegraded":"2024-03-24T12:47:18.463479+0000","last_fullsized":"2024-03-24T12:47:18.463479+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'52","last_scrub_stamp":"2024-03-24T12:46:03.461888+0000","last_deep_scrub":"248'51","last_deep_scrub_stamp":"2024-03-24T12:39:47.606538+0000","last_clean_scrub_stamp":"2024-03-24T12:46:03.461888+0000","objects_scrubbed":10,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:30.129732+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":789,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254485+0000","last_change":"2024-03-24T12:45:47.497683+0000","last_active":"2024-03-24T12:47:26.254485+0000","last_peered":"2024-03-24T12:47:26.254485+0000","last_clean":"2024-03-24T12:47:26.254485+0000","last_became_active":"2024-03-24T12:45:13.300234+0000","last_became_peered":"2024-03-24T12:45:13.300234+0000","last_unstale":"2024-03-24T12:47:26.254485+0000","last_undegraded":"2024-03-24T12:47:26.254485+0000","last_fullsized":"2024-03-24T12:47:26.254485+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:47.497596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:47.497596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":520,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:25.002174+0000","last_change":"2024-03-24T12:46:20.764983+0000","last_active":"2024-03-24T12:47:25.002174+0000","last_peered":"2024-03-24T12:47:25.002174+0000","last_clean":"2024-03-24T12:47:25.002174+0000","last_became_active":"2024-03-24T12:45:40.258591+0000","last_became_peered":"2024-03-24T12:45:40.258591+0000","last_unstale":"2024-03-24T12:47:25.002174+0000","last_undegraded":"2024-03-24T12:47:25.002174+0000","last_fullsized":"2024-03-24T12:47:25.002174+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:46:20.764841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:20.764841+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:43.644952+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":80,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256602+0000","last_change":"2024-03-24T12:45:46.510121+0000","last_active":"2024-03-24T12:47:26.256602+0000","last_peered":"2024-03-24T12:47:26.256602+0000","last_clean":"2024-03-24T12:47:26.256602+0000","last_became_active":"2024-03-24T12:45:40.249680+0000","last_became_peered":"2024-03-24T12:45:40.249680+0000","last_unstale":"2024-03-24T12:47:26.256602+0000","last_undegraded":"2024-03-24T12:47:26.256602+0000","last_fullsized":"2024-03-24T12:47:26.256602+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:46.509996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:24.439296+0000","last_clean_scrub_stamp":"2024-03-24T12:45:46.509996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912535+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":572,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.267551+0000","last_change":"2024-03-24T12:46:09.166237+0000","last_active":"2024-03-24T12:47:26.267551+0000","last_peered":"2024-03-24T12:47:26.267551+0000","last_clean":"2024-03-24T12:47:26.267551+0000","last_became_active":"2024-03-24T12:44:51.223311+0000","last_became_peered":"2024-03-24T12:44:51.223311+0000","last_unstale":"2024-03-24T12:47:26.267551+0000","last_undegraded":"2024-03-24T12:47:26.267551+0000","last_fullsized":"2024-03-24T12:47:26.267551+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:46:09.165580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:46:09.165580+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"410'6673","reported_seq":6739,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.836826+0000","last_change":"2024-03-24T12:45:43.235974+0000","last_active":"2024-03-24T12:47:27.836826+0000","last_peered":"2024-03-24T12:47:27.836826+0000","last_clean":"2024-03-24T12:47:27.836826+0000","last_became_active":"2024-03-24T12:45:43.235557+0000","last_became_peered":"2024-03-24T12:45:43.235557+0000","last_unstale":"2024-03-24T12:47:27.836826+0000","last_undegraded":"2024-03-24T12:47:27.836826+0000","last_fullsized":"2024-03-24T12:47:27.836826+0000","mapping_epoch":382,"log_start":"394'2747","ondisk_log_start":"394'2747","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418447360,"num_objects":6385,"num_object_clones":0,"num_object_copies":19155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6673,"num_write_kb":417856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"390'21","reported_seq":816,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.261525+0000","last_change":"2024-03-24T12:45:59.161020+0000","last_active":"2024-03-24T12:47:26.261525+0000","last_peered":"2024-03-24T12:47:26.261525+0000","last_clean":"2024-03-24T12:47:26.261525+0000","last_became_active":"2024-03-24T12:26:16.875517+0000","last_became_peered":"2024-03-24T12:26:16.875517+0000","last_unstale":"2024-03-24T12:47:26.261525+0000","last_undegraded":"2024-03-24T12:47:26.261525+0000","last_fullsized":"2024-03-24T12:47:26.261525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:45:59.160871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:59.160871+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.082106+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":203,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:25.002915+0000","last_change":"2024-03-24T12:46:12.516626+0000","last_active":"2024-03-24T12:47:25.002915+0000","last_peered":"2024-03-24T12:47:25.002915+0000","last_clean":"2024-03-24T12:47:25.002915+0000","last_became_active":"2024-03-24T12:45:40.252667+0000","last_became_peered":"2024-03-24T12:45:40.252667+0000","last_unstale":"2024-03-24T12:47:25.002915+0000","last_undegraded":"2024-03-24T12:47:25.002915+0000","last_fullsized":"2024-03-24T12:47:25.002915+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:46:12.516495+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:46:12.516495+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:32.355920+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":875,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":772,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.440178+0000","last_change":"2024-03-24T12:45:40.240864+0000","last_active":"2024-03-24T12:47:05.440178+0000","last_peered":"2024-03-24T12:47:05.440178+0000","last_clean":"2024-03-24T12:47:05.440178+0000","last_became_active":"2024-03-24T12:45:40.240504+0000","last_became_peered":"2024-03-24T12:45:40.240504+0000","last_unstale":"2024-03-24T12:47:05.440178+0000","last_undegraded":"2024-03-24T12:47:05.440178+0000","last_fullsized":"2024-03-24T12:47:05.440178+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:11.890544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:11.890544+0000","last_clean_scrub_stamp":"2024-03-24T12:45:11.890544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:05.587878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":758,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.268350+0000","last_change":"2024-03-24T12:46:34.585545+0000","last_active":"2024-03-24T12:47:26.268350+0000","last_peered":"2024-03-24T12:47:26.268350+0000","last_clean":"2024-03-24T12:47:26.268350+0000","last_became_active":"2024-03-24T12:26:09.347251+0000","last_became_peered":"2024-03-24T12:26:09.347251+0000","last_unstale":"2024-03-24T12:47:26.268350+0000","last_undegraded":"2024-03-24T12:47:26.268350+0000","last_fullsized":"2024-03-24T12:47:26.268350+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:34.584424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:34.584424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:45.981954+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.5","version":"410'6735","reported_seq":6801,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.836774+0000","last_change":"2024-03-24T12:45:43.242582+0000","last_active":"2024-03-24T12:47:27.836774+0000","last_peered":"2024-03-24T12:47:27.836774+0000","last_clean":"2024-03-24T12:47:27.836774+0000","last_became_active":"2024-03-24T12:45:43.242256+0000","last_became_peered":"2024-03-24T12:45:43.242256+0000","last_unstale":"2024-03-24T12:47:27.836774+0000","last_undegraded":"2024-03-24T12:47:27.836774+0000","last_fullsized":"2024-03-24T12:47:27.836774+0000","mapping_epoch":382,"log_start":"394'2753","ondisk_log_start":"394'2753","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3982,"log_dups_size":0,"ondisk_log_size":3982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425656320,"num_objects":6495,"num_object_clones":0,"num_object_copies":19485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6495,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6735,"num_write_kb":423360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"391'51","reported_seq":578,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:25.002321+0000","last_change":"2024-03-24T12:46:00.353017+0000","last_active":"2024-03-24T12:47:25.002321+0000","last_peered":"2024-03-24T12:47:25.002321+0000","last_clean":"2024-03-24T12:47:25.002321+0000","last_became_active":"2024-03-24T12:45:13.319445+0000","last_became_peered":"2024-03-24T12:45:13.319445+0000","last_unstale":"2024-03-24T12:47:25.002321+0000","last_undegraded":"2024-03-24T12:47:25.002321+0000","last_fullsized":"2024-03-24T12:47:25.002321+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'35","last_scrub_stamp":"2024-03-24T12:46:00.352851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:00.352851+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:25.389136+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":793,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257628+0000","last_change":"2024-03-24T12:46:16.768092+0000","last_active":"2024-03-24T12:47:26.257628+0000","last_peered":"2024-03-24T12:47:26.257628+0000","last_clean":"2024-03-24T12:47:26.257628+0000","last_became_active":"2024-03-24T12:45:40.233640+0000","last_became_peered":"2024-03-24T12:45:40.233640+0000","last_unstale":"2024-03-24T12:47:26.257628+0000","last_undegraded":"2024-03-24T12:47:26.257628+0000","last_fullsized":"2024-03-24T12:47:26.257628+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:16.767979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:46:16.767979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:36.721481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":743,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.288265+0000","last_change":"2024-03-24T12:45:08.897316+0000","last_active":"2024-03-24T12:47:05.288265+0000","last_peered":"2024-03-24T12:47:05.288265+0000","last_clean":"2024-03-24T12:47:05.288265+0000","last_became_active":"2024-03-24T12:44:48.888020+0000","last_became_peered":"2024-03-24T12:44:48.888020+0000","last_unstale":"2024-03-24T12:47:05.288265+0000","last_undegraded":"2024-03-24T12:47:05.288265+0000","last_fullsized":"2024-03-24T12:47:05.288265+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:08.897184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:08.897184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:05.587950+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":766,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256008+0000","last_change":"2024-03-24T12:46:33.693573+0000","last_active":"2024-03-24T12:47:26.256008+0000","last_peered":"2024-03-24T12:47:26.256008+0000","last_clean":"2024-03-24T12:47:26.256008+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:47:26.256008+0000","last_undegraded":"2024-03-24T12:47:26.256008+0000","last_fullsized":"2024-03-24T12:47:26.256008+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:33.693490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:33.693490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:50.899927+0000","scrub_duration":98,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.6","version":"410'6753","reported_seq":6819,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:28.007032+0000","last_change":"2024-03-24T12:45:43.241533+0000","last_active":"2024-03-24T12:47:28.007032+0000","last_peered":"2024-03-24T12:47:28.007032+0000","last_clean":"2024-03-24T12:47:28.007032+0000","last_became_active":"2024-03-24T12:45:43.241257+0000","last_became_peered":"2024-03-24T12:45:43.241257+0000","last_unstale":"2024-03-24T12:47:28.007032+0000","last_undegraded":"2024-03-24T12:47:28.007032+0000","last_fullsized":"2024-03-24T12:47:28.007032+0000","mapping_epoch":382,"log_start":"396'3500","ondisk_log_start":"396'3500","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3253,"log_dups_size":0,"ondisk_log_size":3253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423690240,"num_objects":6465,"num_object_clones":0,"num_object_copies":19395,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6465,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6753,"num_write_kb":422976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":766,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.288266+0000","last_change":"2024-03-24T12:45:12.924498+0000","last_active":"2024-03-24T12:47:05.288266+0000","last_peered":"2024-03-24T12:47:05.288266+0000","last_clean":"2024-03-24T12:47:05.288266+0000","last_became_active":"2024-03-24T12:26:07.335179+0000","last_became_peered":"2024-03-24T12:26:07.335179+0000","last_unstale":"2024-03-24T12:47:05.288266+0000","last_undegraded":"2024-03-24T12:47:05.288266+0000","last_fullsized":"2024-03-24T12:47:05.288266+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:12.924420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:12.924420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:05.587821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"390'29","reported_seq":838,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254234+0000","last_change":"2024-03-24T12:46:11.591668+0000","last_active":"2024-03-24T12:47:26.254234+0000","last_peered":"2024-03-24T12:47:26.254234+0000","last_clean":"2024-03-24T12:47:26.254234+0000","last_became_active":"2024-03-24T12:45:40.235146+0000","last_became_peered":"2024-03-24T12:45:40.235146+0000","last_unstale":"2024-03-24T12:47:26.254234+0000","last_undegraded":"2024-03-24T12:47:26.254234+0000","last_fullsized":"2024-03-24T12:47:26.254234+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'25","last_scrub_stamp":"2024-03-24T12:46:11.591573+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:46:11.591573+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912571+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":770,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259186+0000","last_change":"2024-03-24T12:46:11.814739+0000","last_active":"2024-03-24T12:47:26.259186+0000","last_peered":"2024-03-24T12:47:26.259186+0000","last_clean":"2024-03-24T12:47:26.259186+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:47:26.259186+0000","last_undegraded":"2024-03-24T12:47:26.259186+0000","last_fullsized":"2024-03-24T12:47:26.259186+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:11.814638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:11.814638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.771958+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":754,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.293589+0000","last_change":"2024-03-24T12:46:00.248579+0000","last_active":"2024-03-24T12:47:26.293589+0000","last_peered":"2024-03-24T12:47:26.293589+0000","last_clean":"2024-03-24T12:47:26.293589+0000","last_became_active":"2024-03-24T12:45:13.264047+0000","last_became_peered":"2024-03-24T12:45:13.264047+0000","last_unstale":"2024-03-24T12:47:26.293589+0000","last_undegraded":"2024-03-24T12:47:26.293589+0000","last_fullsized":"2024-03-24T12:47:26.293589+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:00.248216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:00.248216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.137422+0000","scrub_duration":177,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.7","version":"409'6708","reported_seq":6772,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:25.186961+0000","last_change":"2024-03-24T12:45:43.234010+0000","last_active":"2024-03-24T12:47:25.186961+0000","last_peered":"2024-03-24T12:47:25.186961+0000","last_clean":"2024-03-24T12:47:25.186961+0000","last_became_active":"2024-03-24T12:45:43.233265+0000","last_became_peered":"2024-03-24T12:45:43.233265+0000","last_unstale":"2024-03-24T12:47:25.186961+0000","last_undegraded":"2024-03-24T12:47:25.186961+0000","last_fullsized":"2024-03-24T12:47:25.186961+0000","mapping_epoch":382,"log_start":"396'3700","ondisk_log_start":"396'3700","created":382,"last_epoch_clean":383,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:42.215930+0000","last_clean_scrub_stamp":"2024-03-24T12:45:42.215930+0000","objects_scrubbed":0,"log_size":3008,"log_dups_size":2860,"ondisk_log_size":3008,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439615488,"num_objects":6708,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6708,"num_write_kb":429312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"370'11","reported_seq":1068,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257398+0000","last_change":"2024-03-24T12:46:01.921661+0000","last_active":"2024-03-24T12:47:26.257398+0000","last_peered":"2024-03-24T12:47:26.257398+0000","last_clean":"2024-03-24T12:47:26.257398+0000","last_became_active":"2024-03-24T12:45:13.259309+0000","last_became_peered":"2024-03-24T12:45:13.259309+0000","last_unstale":"2024-03-24T12:47:26.257398+0000","last_undegraded":"2024-03-24T12:47:26.257398+0000","last_fullsized":"2024-03-24T12:47:26.257398+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"370'11","last_scrub_stamp":"2024-03-24T12:46:01.921579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:01.921579+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448922+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"390'39","reported_seq":274,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254219+0000","last_change":"2024-03-24T12:45:40.265367+0000","last_active":"2024-03-24T12:47:26.254219+0000","last_peered":"2024-03-24T12:47:26.254219+0000","last_clean":"2024-03-24T12:47:26.254219+0000","last_became_active":"2024-03-24T12:45:40.229387+0000","last_became_peered":"2024-03-24T12:45:40.229387+0000","last_unstale":"2024-03-24T12:47:26.254219+0000","last_undegraded":"2024-03-24T12:47:26.254219+0000","last_fullsized":"2024-03-24T12:47:26.254219+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-24T12:44:39.791402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:44:39.791402+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":782,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.262072+0000","last_change":"2024-03-24T12:46:17.778655+0000","last_active":"2024-03-24T12:47:26.262072+0000","last_peered":"2024-03-24T12:47:26.262072+0000","last_clean":"2024-03-24T12:47:26.262072+0000","last_became_active":"2024-03-24T12:26:03.161380+0000","last_became_peered":"2024-03-24T12:26:03.161380+0000","last_unstale":"2024-03-24T12:47:26.262072+0000","last_undegraded":"2024-03-24T12:47:26.262072+0000","last_fullsized":"2024-03-24T12:47:26.262072+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:46:17.778517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:17.778517+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.449111+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":392,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259778+0000","last_change":"2024-03-24T12:46:07.137940+0000","last_active":"2024-03-24T12:47:26.259778+0000","last_peered":"2024-03-24T12:47:26.259778+0000","last_clean":"2024-03-24T12:47:26.259778+0000","last_became_active":"2024-03-24T12:45:13.325330+0000","last_became_peered":"2024-03-24T12:45:13.325330+0000","last_unstale":"2024-03-24T12:47:26.259778+0000","last_undegraded":"2024-03-24T12:47:26.259778+0000","last_fullsized":"2024-03-24T12:47:26.259778+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:07.137717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:07.137717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:31.609482+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"390'18","reported_seq":655,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.427363+0000","last_change":"2024-03-24T12:45:47.380906+0000","last_active":"2024-03-24T12:47:05.427363+0000","last_peered":"2024-03-24T12:47:05.427363+0000","last_clean":"2024-03-24T12:47:05.427363+0000","last_became_active":"2024-03-24T12:45:40.245027+0000","last_became_peered":"2024-03-24T12:45:40.245027+0000","last_unstale":"2024-03-24T12:47:05.427363+0000","last_undegraded":"2024-03-24T12:47:05.427363+0000","last_fullsized":"2024-03-24T12:47:05.427363+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'12","last_scrub_stamp":"2024-03-24T12:45:47.380748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:47.380748+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:05.588275+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":715,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259902+0000","last_change":"2024-03-24T12:46:12.200721+0000","last_active":"2024-03-24T12:47:26.259902+0000","last_peered":"2024-03-24T12:47:26.259902+0000","last_clean":"2024-03-24T12:47:26.259902+0000","last_became_active":"2024-03-24T12:45:40.255835+0000","last_became_peered":"2024-03-24T12:45:40.255835+0000","last_unstale":"2024-03-24T12:47:26.259902+0000","last_undegraded":"2024-03-24T12:47:26.259902+0000","last_fullsized":"2024-03-24T12:47:26.259902+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:12.200446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:12.200446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:33.879810+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":426,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.258865+0000","last_change":"2024-03-24T12:45:40.243760+0000","last_active":"2024-03-24T12:47:26.258865+0000","last_peered":"2024-03-24T12:47:26.258865+0000","last_clean":"2024-03-24T12:47:26.258865+0000","last_became_active":"2024-03-24T12:45:40.243393+0000","last_became_peered":"2024-03-24T12:45:40.243393+0000","last_unstale":"2024-03-24T12:47:26.258865+0000","last_undegraded":"2024-03-24T12:47:26.258865+0000","last_fullsized":"2024-03-24T12:47:26.258865+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:57.075121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:57.075121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448349+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":773,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257178+0000","last_change":"2024-03-24T12:46:18.888939+0000","last_active":"2024-03-24T12:47:26.257178+0000","last_peered":"2024-03-24T12:47:26.257178+0000","last_clean":"2024-03-24T12:47:26.257178+0000","last_became_active":"2024-03-24T12:45:13.312958+0000","last_became_peered":"2024-03-24T12:45:13.312958+0000","last_unstale":"2024-03-24T12:47:26.257178+0000","last_undegraded":"2024-03-24T12:47:26.257178+0000","last_fullsized":"2024-03-24T12:47:26.257178+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:18.888854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:55.271679+0000","last_clean_scrub_stamp":"2024-03-24T12:46:18.888854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:31.649819+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"390'21","reported_seq":847,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.260537+0000","last_change":"2024-03-24T12:46:47.973024+0000","last_active":"2024-03-24T12:47:26.260537+0000","last_peered":"2024-03-24T12:47:26.260537+0000","last_clean":"2024-03-24T12:47:26.260537+0000","last_became_active":"2024-03-24T12:46:47.958992+0000","last_became_peered":"2024-03-24T12:46:47.958992+0000","last_unstale":"2024-03-24T12:47:26.260537+0000","last_undegraded":"2024-03-24T12:47:26.260537+0000","last_fullsized":"2024-03-24T12:47:26.260537+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:45:10.860208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:10.860208+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":797,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257243+0000","last_change":"2024-03-24T12:46:26.643534+0000","last_active":"2024-03-24T12:47:26.257243+0000","last_peered":"2024-03-24T12:47:26.257243+0000","last_clean":"2024-03-24T12:47:26.257243+0000","last_became_active":"2024-03-24T12:45:13.324149+0000","last_became_peered":"2024-03-24T12:45:13.324149+0000","last_unstale":"2024-03-24T12:47:26.257243+0000","last_undegraded":"2024-03-24T12:47:26.257243+0000","last_fullsized":"2024-03-24T12:47:26.257243+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:26.643413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:26.643413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.064715+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":78,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.993820+0000","last_change":"2024-03-24T12:45:51.305650+0000","last_active":"2024-03-24T12:47:24.993820+0000","last_peered":"2024-03-24T12:47:24.993820+0000","last_clean":"2024-03-24T12:47:24.993820+0000","last_became_active":"2024-03-24T12:45:40.249711+0000","last_became_peered":"2024-03-24T12:45:40.249711+0000","last_unstale":"2024-03-24T12:47:24.993820+0000","last_undegraded":"2024-03-24T12:47:24.993820+0000","last_fullsized":"2024-03-24T12:47:24.993820+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:51.305535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:45:51.305535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":770,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.428813+0000","last_change":"2024-03-24T12:46:11.562073+0000","last_active":"2024-03-24T12:47:05.428813+0000","last_peered":"2024-03-24T12:47:05.428813+0000","last_clean":"2024-03-24T12:47:05.428813+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:47:05.428813+0000","last_undegraded":"2024-03-24T12:47:05.428813+0000","last_fullsized":"2024-03-24T12:47:05.428813+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:11.561956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:46:11.561956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:17.667679+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"390'25","reported_seq":549,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257703+0000","last_change":"2024-03-24T12:45:13.336981+0000","last_active":"2024-03-24T12:47:26.257703+0000","last_peered":"2024-03-24T12:47:26.257703+0000","last_clean":"2024-03-24T12:47:26.257703+0000","last_became_active":"2024-03-24T12:45:13.312997+0000","last_became_peered":"2024-03-24T12:45:13.312997+0000","last_unstale":"2024-03-24T12:47:26.257703+0000","last_undegraded":"2024-03-24T12:47:26.257703+0000","last_fullsized":"2024-03-24T12:47:26.257703+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:44:54.781997+0000","last_deep_scrub":"207'19","last_deep_scrub_stamp":"2024-03-24T12:44:54.781997+0000","last_clean_scrub_stamp":"2024-03-24T12:44:54.781997+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448501+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":776,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259615+0000","last_change":"2024-03-24T12:45:40.251507+0000","last_active":"2024-03-24T12:47:26.259615+0000","last_peered":"2024-03-24T12:47:26.259615+0000","last_clean":"2024-03-24T12:47:26.259615+0000","last_became_active":"2024-03-24T12:45:40.251051+0000","last_became_peered":"2024-03-24T12:45:40.251051+0000","last_unstale":"2024-03-24T12:47:26.259615+0000","last_undegraded":"2024-03-24T12:47:26.259615+0000","last_fullsized":"2024-03-24T12:47:26.259615+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:12.900147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:12.900147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"380'10","reported_seq":194,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257316+0000","last_change":"2024-03-24T12:45:51.954271+0000","last_active":"2024-03-24T12:47:26.257316+0000","last_peered":"2024-03-24T12:47:26.257316+0000","last_clean":"2024-03-24T12:47:26.257316+0000","last_became_active":"2024-03-24T12:45:40.243508+0000","last_became_peered":"2024-03-24T12:45:40.243508+0000","last_unstale":"2024-03-24T12:47:26.257316+0000","last_undegraded":"2024-03-24T12:47:26.257316+0000","last_fullsized":"2024-03-24T12:47:26.257316+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"380'10","last_scrub_stamp":"2024-03-24T12:45:51.954162+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:45:51.954162+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448813+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":743,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254991+0000","last_change":"2024-03-24T12:44:57.736897+0000","last_active":"2024-03-24T12:47:26.254991+0000","last_peered":"2024-03-24T12:47:26.254991+0000","last_clean":"2024-03-24T12:47:26.254991+0000","last_became_active":"2024-03-24T12:44:48.872901+0000","last_became_peered":"2024-03-24T12:44:48.872901+0000","last_unstale":"2024-03-24T12:47:26.254991+0000","last_undegraded":"2024-03-24T12:47:26.254991+0000","last_fullsized":"2024-03-24T12:47:26.254991+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:57.736450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:57.736450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912289+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,0],"acting":[4,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"390'30","reported_seq":811,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:27.090383+0000","last_change":"2024-03-24T12:47:27.090434+0000","last_active":"2024-03-24T12:47:27.090383+0000","last_peered":"2024-03-24T12:47:27.090383+0000","last_clean":"2024-03-24T12:47:27.090383+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:47:27.090383+0000","last_undegraded":"2024-03-24T12:47:27.090383+0000","last_fullsized":"2024-03-24T12:47:27.090383+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'30","last_scrub_stamp":"2024-03-24T12:47:27.090292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:47:27.090292+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:48:37.624531+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":773,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256373+0000","last_change":"2024-03-24T12:45:58.894456+0000","last_active":"2024-03-24T12:47:26.256373+0000","last_peered":"2024-03-24T12:47:26.256373+0000","last_clean":"2024-03-24T12:47:26.256373+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:47:26.256373+0000","last_undegraded":"2024-03-24T12:47:26.256373+0000","last_fullsized":"2024-03-24T12:47:26.256373+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:58.894376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:58.894376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448949+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"354'11","reported_seq":203,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259948+0000","last_change":"2024-03-24T12:46:37.529150+0000","last_active":"2024-03-24T12:47:26.259948+0000","last_peered":"2024-03-24T12:47:26.259948+0000","last_clean":"2024-03-24T12:47:26.259948+0000","last_became_active":"2024-03-24T12:44:48.881388+0000","last_became_peered":"2024-03-24T12:44:48.881388+0000","last_unstale":"2024-03-24T12:47:26.259948+0000","last_undegraded":"2024-03-24T12:47:26.259948+0000","last_fullsized":"2024-03-24T12:47:26.259948+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"354'11","last_scrub_stamp":"2024-03-24T12:46:37.529010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:37.529010+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:52.335155+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":768,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259605+0000","last_change":"2024-03-24T12:46:01.182095+0000","last_active":"2024-03-24T12:47:26.259605+0000","last_peered":"2024-03-24T12:47:26.259605+0000","last_clean":"2024-03-24T12:47:26.259605+0000","last_became_active":"2024-03-24T12:45:40.242801+0000","last_became_peered":"2024-03-24T12:45:40.242801+0000","last_unstale":"2024-03-24T12:47:26.259605+0000","last_undegraded":"2024-03-24T12:47:26.259605+0000","last_fullsized":"2024-03-24T12:47:26.259605+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:46:01.181962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:01.181962+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.875844+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"390'32","reported_seq":858,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254672+0000","last_change":"2024-03-24T12:46:30.819919+0000","last_active":"2024-03-24T12:47:26.254672+0000","last_peered":"2024-03-24T12:47:26.254672+0000","last_clean":"2024-03-24T12:47:26.254672+0000","last_became_active":"2024-03-24T12:45:13.305687+0000","last_became_peered":"2024-03-24T12:45:13.305687+0000","last_unstale":"2024-03-24T12:47:26.254672+0000","last_undegraded":"2024-03-24T12:47:26.254672+0000","last_fullsized":"2024-03-24T12:47:26.254672+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'32","last_scrub_stamp":"2024-03-24T12:46:30.819781+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:46:30.819781+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:55.708488+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":796,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254031+0000","last_change":"2024-03-24T12:46:32.853515+0000","last_active":"2024-03-24T12:47:26.254031+0000","last_peered":"2024-03-24T12:47:26.254031+0000","last_clean":"2024-03-24T12:47:26.254031+0000","last_became_active":"2024-03-24T12:45:13.305719+0000","last_became_peered":"2024-03-24T12:45:13.305719+0000","last_unstale":"2024-03-24T12:47:26.254031+0000","last_undegraded":"2024-03-24T12:47:26.254031+0000","last_fullsized":"2024-03-24T12:47:26.254031+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:32.853388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:32.853388+0000","last_clean_scrub_stamp":"2024-03-24T12:46:32.853388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:51.332361+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":759,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:25.002484+0000","last_change":"2024-03-24T12:46:31.878736+0000","last_active":"2024-03-24T12:47:25.002484+0000","last_peered":"2024-03-24T12:47:25.002484+0000","last_clean":"2024-03-24T12:47:25.002484+0000","last_became_active":"2024-03-24T12:45:40.244668+0000","last_became_peered":"2024-03-24T12:45:40.244668+0000","last_unstale":"2024-03-24T12:47:25.002484+0000","last_undegraded":"2024-03-24T12:47:25.002484+0000","last_fullsized":"2024-03-24T12:47:25.002484+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:31.878576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:46:31.878576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:59.407040+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":684,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.268196+0000","last_change":"2024-03-24T12:46:19.086662+0000","last_active":"2024-03-24T12:47:26.268196+0000","last_peered":"2024-03-24T12:47:26.268196+0000","last_clean":"2024-03-24T12:47:26.268196+0000","last_became_active":"2024-03-24T12:45:40.240370+0000","last_became_peered":"2024-03-24T12:45:40.240370+0000","last_unstale":"2024-03-24T12:47:26.268196+0000","last_undegraded":"2024-03-24T12:47:26.268196+0000","last_fullsized":"2024-03-24T12:47:26.268196+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:19.086515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:26.431382+0000","last_clean_scrub_stamp":"2024-03-24T12:46:19.086515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:45.346160+0000","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"390'21","reported_seq":163,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.994219+0000","last_change":"2024-03-24T12:46:02.430642+0000","last_active":"2024-03-24T12:47:24.994219+0000","last_peered":"2024-03-24T12:47:24.994219+0000","last_clean":"2024-03-24T12:47:24.994219+0000","last_became_active":"2024-03-24T12:45:13.318491+0000","last_became_peered":"2024-03-24T12:45:13.318491+0000","last_unstale":"2024-03-24T12:47:24.994219+0000","last_undegraded":"2024-03-24T12:47:24.994219+0000","last_fullsized":"2024-03-24T12:47:24.994219+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:46:02.430495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:02.430495+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":769,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257278+0000","last_change":"2024-03-24T12:45:07.883531+0000","last_active":"2024-03-24T12:47:26.257278+0000","last_peered":"2024-03-24T12:47:26.257278+0000","last_clean":"2024-03-24T12:47:26.257278+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:47:26.257278+0000","last_undegraded":"2024-03-24T12:47:26.257278+0000","last_fullsized":"2024-03-24T12:47:26.257278+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:07.883424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:07.883424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":754,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.293492+0000","last_change":"2024-03-24T12:46:28.070294+0000","last_active":"2024-03-24T12:47:26.293492+0000","last_peered":"2024-03-24T12:47:26.293492+0000","last_clean":"2024-03-24T12:47:26.293492+0000","last_became_active":"2024-03-24T12:26:07.336491+0000","last_became_peered":"2024-03-24T12:26:07.336491+0000","last_unstale":"2024-03-24T12:47:26.293492+0000","last_undegraded":"2024-03-24T12:47:26.293492+0000","last_fullsized":"2024-03-24T12:47:26.293492+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:28.070148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:28.070148+0000","last_clean_scrub_stamp":"2024-03-24T12:46:28.070148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:30.014917+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":749,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259316+0000","last_change":"2024-03-24T12:46:35.493000+0000","last_active":"2024-03-24T12:47:26.259316+0000","last_peered":"2024-03-24T12:47:26.259316+0000","last_clean":"2024-03-24T12:47:26.259316+0000","last_became_active":"2024-03-24T12:45:13.305903+0000","last_became_peered":"2024-03-24T12:45:13.305903+0000","last_unstale":"2024-03-24T12:47:26.259316+0000","last_undegraded":"2024-03-24T12:47:26.259316+0000","last_fullsized":"2024-03-24T12:47:26.259316+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:35.492877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:46:35.492877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:56.934755+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,3],"acting":[0,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"390'27","reported_seq":833,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.293095+0000","last_change":"2024-03-24T12:46:29.088327+0000","last_active":"2024-03-24T12:47:26.293095+0000","last_peered":"2024-03-24T12:47:26.293095+0000","last_clean":"2024-03-24T12:47:26.293095+0000","last_became_active":"2024-03-24T12:45:13.326567+0000","last_became_peered":"2024-03-24T12:45:13.326567+0000","last_unstale":"2024-03-24T12:47:26.293095+0000","last_undegraded":"2024-03-24T12:47:26.293095+0000","last_fullsized":"2024-03-24T12:47:26.293095+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'27","last_scrub_stamp":"2024-03-24T12:46:29.088176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:29.088176+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:50.975223+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"67'1","reported_seq":475,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.509762+0000","last_change":"2024-03-24T12:47:26.509791+0000","last_active":"2024-03-24T12:47:26.509762+0000","last_peered":"2024-03-24T12:47:26.509762+0000","last_clean":"2024-03-24T12:47:26.509762+0000","last_became_active":"2024-03-24T12:45:40.243344+0000","last_became_peered":"2024-03-24T12:45:40.243344+0000","last_unstale":"2024-03-24T12:47:26.509762+0000","last_undegraded":"2024-03-24T12:47:26.509762+0000","last_fullsized":"2024-03-24T12:47:26.509762+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:47:26.509703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:47:26.509703+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:48:47.591681+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":780,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256459+0000","last_change":"2024-03-24T12:45:13.307703+0000","last_active":"2024-03-24T12:47:26.256459+0000","last_peered":"2024-03-24T12:47:26.256459+0000","last_clean":"2024-03-24T12:47:26.256459+0000","last_became_active":"2024-03-24T12:45:13.307347+0000","last_became_peered":"2024-03-24T12:45:13.307347+0000","last_unstale":"2024-03-24T12:47:26.256459+0000","last_undegraded":"2024-03-24T12:47:26.256459+0000","last_fullsized":"2024-03-24T12:47:26.256459+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:59.763462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:59.763462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448649+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":742,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.260960+0000","last_change":"2024-03-24T12:45:49.068473+0000","last_active":"2024-03-24T12:47:26.260960+0000","last_peered":"2024-03-24T12:47:26.260960+0000","last_clean":"2024-03-24T12:47:26.260960+0000","last_became_active":"2024-03-24T12:45:40.236267+0000","last_became_peered":"2024-03-24T12:45:40.236267+0000","last_unstale":"2024-03-24T12:47:26.260960+0000","last_undegraded":"2024-03-24T12:47:26.260960+0000","last_fullsized":"2024-03-24T12:47:26.260960+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:49.068372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:29.996197+0000","last_clean_scrub_stamp":"2024-03-24T12:45:49.068372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.875807+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"390'12","reported_seq":786,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253379+0000","last_change":"2024-03-24T12:45:40.256168+0000","last_active":"2024-03-24T12:47:26.253379+0000","last_peered":"2024-03-24T12:47:26.253379+0000","last_clean":"2024-03-24T12:47:26.253379+0000","last_became_active":"2024-03-24T12:45:40.228239+0000","last_became_peered":"2024-03-24T12:45:40.228239+0000","last_unstale":"2024-03-24T12:47:26.253379+0000","last_undegraded":"2024-03-24T12:47:26.253379+0000","last_fullsized":"2024-03-24T12:47:26.253379+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'8","last_scrub_stamp":"2024-03-24T12:44:31.116754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:44:31.116754+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912165+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":758,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.261735+0000","last_change":"2024-03-24T12:45:40.250012+0000","last_active":"2024-03-24T12:47:26.261735+0000","last_peered":"2024-03-24T12:47:26.261735+0000","last_clean":"2024-03-24T12:47:26.261735+0000","last_became_active":"2024-03-24T12:45:40.249671+0000","last_became_peered":"2024-03-24T12:45:40.249671+0000","last_unstale":"2024-03-24T12:47:26.261735+0000","last_undegraded":"2024-03-24T12:47:26.261735+0000","last_fullsized":"2024-03-24T12:47:26.261735+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:01.948746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:01.948746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":144,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253561+0000","last_change":"2024-03-24T12:45:13.309572+0000","last_active":"2024-03-24T12:47:26.253561+0000","last_peered":"2024-03-24T12:47:26.253561+0000","last_clean":"2024-03-24T12:47:26.253561+0000","last_became_active":"2024-03-24T12:45:13.308636+0000","last_became_peered":"2024-03-24T12:45:13.308636+0000","last_unstale":"2024-03-24T12:47:26.253561+0000","last_undegraded":"2024-03-24T12:47:26.253561+0000","last_fullsized":"2024-03-24T12:47:26.253561+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:40.785287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:40.785287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":758,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256848+0000","last_change":"2024-03-24T12:44:57.823516+0000","last_active":"2024-03-24T12:47:26.256848+0000","last_peered":"2024-03-24T12:47:26.256848+0000","last_clean":"2024-03-24T12:47:26.256848+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:47:26.256848+0000","last_undegraded":"2024-03-24T12:47:26.256848+0000","last_fullsized":"2024-03-24T12:47:26.256848+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:57.823399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:57.823399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448467+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"390'27","reported_seq":559,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.994082+0000","last_change":"2024-03-24T12:46:01.333390+0000","last_active":"2024-03-24T12:47:24.994082+0000","last_peered":"2024-03-24T12:47:24.994082+0000","last_clean":"2024-03-24T12:47:24.994082+0000","last_became_active":"2024-03-24T12:45:40.252179+0000","last_became_peered":"2024-03-24T12:45:40.252179+0000","last_unstale":"2024-03-24T12:47:24.994082+0000","last_undegraded":"2024-03-24T12:47:24.994082+0000","last_fullsized":"2024-03-24T12:47:24.994082+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'19","last_scrub_stamp":"2024-03-24T12:46:01.333187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:01.333187+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:25.456289+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":81,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259526+0000","last_change":"2024-03-24T12:45:52.018351+0000","last_active":"2024-03-24T12:47:26.259526+0000","last_peered":"2024-03-24T12:47:26.259526+0000","last_clean":"2024-03-24T12:47:26.259526+0000","last_became_active":"2024-03-24T12:45:40.256265+0000","last_became_peered":"2024-03-24T12:45:40.256265+0000","last_unstale":"2024-03-24T12:47:26.259526+0000","last_undegraded":"2024-03-24T12:47:26.259526+0000","last_fullsized":"2024-03-24T12:47:26.259526+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:52.018268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:12.437708+0000","last_clean_scrub_stamp":"2024-03-24T12:45:52.018268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":763,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.260688+0000","last_change":"2024-03-24T12:46:13.115943+0000","last_active":"2024-03-24T12:47:26.260688+0000","last_peered":"2024-03-24T12:47:26.260688+0000","last_clean":"2024-03-24T12:47:26.260688+0000","last_became_active":"2024-03-24T12:26:07.331598+0000","last_became_peered":"2024-03-24T12:26:07.331598+0000","last_unstale":"2024-03-24T12:47:26.260688+0000","last_undegraded":"2024-03-24T12:47:26.260688+0000","last_fullsized":"2024-03-24T12:47:26.260688+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:13.115801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:13.115801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":657,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253866+0000","last_change":"2024-03-24T12:44:54.658333+0000","last_active":"2024-03-24T12:47:26.253866+0000","last_peered":"2024-03-24T12:47:26.253866+0000","last_clean":"2024-03-24T12:47:26.253866+0000","last_became_active":"2024-03-24T12:44:48.870417+0000","last_became_peered":"2024-03-24T12:44:48.870417+0000","last_unstale":"2024-03-24T12:47:26.253866+0000","last_undegraded":"2024-03-24T12:47:26.253866+0000","last_fullsized":"2024-03-24T12:47:26.253866+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:54.658192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:54.658192+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912181+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"390'30","reported_seq":856,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.260125+0000","last_change":"2024-03-24T12:46:11.145946+0000","last_active":"2024-03-24T12:47:26.260125+0000","last_peered":"2024-03-24T12:47:26.260125+0000","last_clean":"2024-03-24T12:47:26.260125+0000","last_became_active":"2024-03-24T12:45:40.255480+0000","last_became_peered":"2024-03-24T12:45:40.255480+0000","last_unstale":"2024-03-24T12:47:26.260125+0000","last_undegraded":"2024-03-24T12:47:26.260125+0000","last_fullsized":"2024-03-24T12:47:26.260125+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'20","last_scrub_stamp":"2024-03-24T12:46:11.145131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:11.145131+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"392'2","reported_seq":785,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.994241+0000","last_change":"2024-03-24T12:46:18.735937+0000","last_active":"2024-03-24T12:47:24.994241+0000","last_peered":"2024-03-24T12:47:24.994241+0000","last_clean":"2024-03-24T12:47:24.994241+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:47:24.994241+0000","last_undegraded":"2024-03-24T12:47:24.994241+0000","last_fullsized":"2024-03-24T12:47:24.994241+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'2","last_scrub_stamp":"2024-03-24T12:46:18.735806+0000","last_deep_scrub":"392'2","last_deep_scrub_stamp":"2024-03-24T12:46:18.735806+0000","last_clean_scrub_stamp":"2024-03-24T12:46:18.735806+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:36.258237+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"370'12","reported_seq":224,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254846+0000","last_change":"2024-03-24T12:45:13.326830+0000","last_active":"2024-03-24T12:47:26.254846+0000","last_peered":"2024-03-24T12:47:26.254846+0000","last_clean":"2024-03-24T12:47:26.254846+0000","last_became_active":"2024-03-24T12:45:13.310827+0000","last_became_peered":"2024-03-24T12:45:13.310827+0000","last_unstale":"2024-03-24T12:47:26.254846+0000","last_undegraded":"2024-03-24T12:47:26.254846+0000","last_fullsized":"2024-03-24T12:47:26.254846+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"315'11","last_scrub_stamp":"2024-03-24T12:44:42.835854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:42.835854+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":749,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257327+0000","last_change":"2024-03-24T12:45:40.232224+0000","last_active":"2024-03-24T12:47:26.257327+0000","last_peered":"2024-03-24T12:47:26.257327+0000","last_clean":"2024-03-24T12:47:26.257327+0000","last_became_active":"2024-03-24T12:45:40.227973+0000","last_became_peered":"2024-03-24T12:45:40.227973+0000","last_unstale":"2024-03-24T12:47:26.257327+0000","last_undegraded":"2024-03-24T12:47:26.257327+0000","last_fullsized":"2024-03-24T12:47:26.257327+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:44:51.808307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:44:51.808307+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448413+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"391'24","reported_seq":444,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:25.000684+0000","last_change":"2024-03-24T12:45:56.341335+0000","last_active":"2024-03-24T12:47:25.000684+0000","last_peered":"2024-03-24T12:47:25.000684+0000","last_clean":"2024-03-24T12:47:25.000684+0000","last_became_active":"2024-03-24T12:45:40.253908+0000","last_became_peered":"2024-03-24T12:45:40.253908+0000","last_unstale":"2024-03-24T12:47:25.000684+0000","last_undegraded":"2024-03-24T12:47:25.000684+0000","last_fullsized":"2024-03-24T12:47:25.000684+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'18","last_scrub_stamp":"2024-03-24T12:45:56.341162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:56.341162+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":769,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254204+0000","last_change":"2024-03-24T12:45:56.529812+0000","last_active":"2024-03-24T12:47:26.254204+0000","last_peered":"2024-03-24T12:47:26.254204+0000","last_clean":"2024-03-24T12:47:26.254204+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:47:26.254204+0000","last_undegraded":"2024-03-24T12:47:26.254204+0000","last_fullsized":"2024-03-24T12:47:26.254204+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:56.529721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:45:56.529721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912503+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":761,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.255134+0000","last_change":"2024-03-24T12:44:39.546725+0000","last_active":"2024-03-24T12:47:26.255134+0000","last_peered":"2024-03-24T12:47:26.255134+0000","last_clean":"2024-03-24T12:47:26.255134+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:47:26.255134+0000","last_undegraded":"2024-03-24T12:47:26.255134+0000","last_fullsized":"2024-03-24T12:47:26.255134+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:39.546649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:44:39.546649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912132+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":768,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.294626+0000","last_change":"2024-03-24T12:46:02.238230+0000","last_active":"2024-03-24T12:47:26.294626+0000","last_peered":"2024-03-24T12:47:26.294626+0000","last_clean":"2024-03-24T12:47:26.294626+0000","last_became_active":"2024-03-24T12:45:13.313299+0000","last_became_peered":"2024-03-24T12:45:13.313299+0000","last_unstale":"2024-03-24T12:47:26.294626+0000","last_undegraded":"2024-03-24T12:47:26.294626+0000","last_fullsized":"2024-03-24T12:47:26.294626+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:02.237746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:02.237746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.137448+0000","scrub_duration":243,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"390'39","reported_seq":852,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.428487+0000","last_change":"2024-03-24T12:45:49.393129+0000","last_active":"2024-03-24T12:47:05.428487+0000","last_peered":"2024-03-24T12:47:05.428487+0000","last_clean":"2024-03-24T12:47:05.428487+0000","last_became_active":"2024-03-24T12:45:40.245335+0000","last_became_peered":"2024-03-24T12:45:40.245335+0000","last_unstale":"2024-03-24T12:47:05.428487+0000","last_undegraded":"2024-03-24T12:47:05.428487+0000","last_fullsized":"2024-03-24T12:47:05.428487+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'27","last_scrub_stamp":"2024-03-24T12:45:49.393029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:49.393029+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:15.980341+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":769,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.261699+0000","last_change":"2024-03-24T12:45:56.143367+0000","last_active":"2024-03-24T12:47:26.261699+0000","last_peered":"2024-03-24T12:47:26.261699+0000","last_clean":"2024-03-24T12:47:26.261699+0000","last_became_active":"2024-03-24T12:26:03.166126+0000","last_became_peered":"2024-03-24T12:26:03.166126+0000","last_unstale":"2024-03-24T12:47:26.261699+0000","last_undegraded":"2024-03-24T12:47:26.261699+0000","last_fullsized":"2024-03-24T12:47:26.261699+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:56.143236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:45:56.143236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":783,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:25.000943+0000","last_change":"2024-03-24T12:46:15.640190+0000","last_active":"2024-03-24T12:47:25.000943+0000","last_peered":"2024-03-24T12:47:25.000943+0000","last_clean":"2024-03-24T12:47:25.000943+0000","last_became_active":"2024-03-24T12:26:07.329517+0000","last_became_peered":"2024-03-24T12:26:07.329517+0000","last_unstale":"2024-03-24T12:47:25.000943+0000","last_undegraded":"2024-03-24T12:47:25.000943+0000","last_fullsized":"2024-03-24T12:47:25.000943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:15.638516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:15.638516+0000","last_clean_scrub_stamp":"2024-03-24T12:46:15.638516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:34.548383+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":790,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259305+0000","last_change":"2024-03-24T12:46:26.731340+0000","last_active":"2024-03-24T12:47:26.259305+0000","last_peered":"2024-03-24T12:47:26.259305+0000","last_clean":"2024-03-24T12:47:26.259305+0000","last_became_active":"2024-03-24T12:45:40.234852+0000","last_became_peered":"2024-03-24T12:45:40.234852+0000","last_unstale":"2024-03-24T12:47:26.259305+0000","last_undegraded":"2024-03-24T12:47:26.259305+0000","last_fullsized":"2024-03-24T12:47:26.259305+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:26.731239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:46:26.731239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:44.116348+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"390'9","reported_seq":793,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254328+0000","last_change":"2024-03-24T12:46:47.949393+0000","last_active":"2024-03-24T12:47:26.254328+0000","last_peered":"2024-03-24T12:47:26.254328+0000","last_clean":"2024-03-24T12:47:26.254328+0000","last_became_active":"2024-03-24T12:46:47.943482+0000","last_became_peered":"2024-03-24T12:46:47.943482+0000","last_unstale":"2024-03-24T12:47:26.254328+0000","last_undegraded":"2024-03-24T12:47:26.254328+0000","last_fullsized":"2024-03-24T12:47:26.254328+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'7","last_scrub_stamp":"2024-03-24T12:45:52.553430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:45:52.553430+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912520+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":443,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253929+0000","last_change":"2024-03-24T12:45:13.260482+0000","last_active":"2024-03-24T12:47:26.253929+0000","last_peered":"2024-03-24T12:47:26.253929+0000","last_clean":"2024-03-24T12:47:26.253929+0000","last_became_active":"2024-03-24T12:45:13.259915+0000","last_became_peered":"2024-03-24T12:45:13.259915+0000","last_unstale":"2024-03-24T12:47:26.253929+0000","last_undegraded":"2024-03-24T12:47:26.253929+0000","last_fullsized":"2024-03-24T12:47:26.253929+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:55.611389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:44:55.611389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":761,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256744+0000","last_change":"2024-03-24T12:45:06.855502+0000","last_active":"2024-03-24T12:47:26.256744+0000","last_peered":"2024-03-24T12:47:26.256744+0000","last_clean":"2024-03-24T12:47:26.256744+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:47:26.256744+0000","last_undegraded":"2024-03-24T12:47:26.256744+0000","last_fullsized":"2024-03-24T12:47:26.256744+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:06.855388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:17.365696+0000","last_clean_scrub_stamp":"2024-03-24T12:45:06.855388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448621+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":449,"reported_epoch":409,"state":"active+clean","last_fresh":"2024-03-24T12:47:24.994536+0000","last_change":"2024-03-24T12:46:33.991040+0000","last_active":"2024-03-24T12:47:24.994536+0000","last_peered":"2024-03-24T12:47:24.994536+0000","last_clean":"2024-03-24T12:47:24.994536+0000","last_became_active":"2024-03-24T12:45:40.252467+0000","last_became_peered":"2024-03-24T12:45:40.252467+0000","last_unstale":"2024-03-24T12:47:24.994536+0000","last_undegraded":"2024-03-24T12:47:24.994536+0000","last_fullsized":"2024-03-24T12:47:24.994536+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:33.990925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:21.259962+0000","last_clean_scrub_stamp":"2024-03-24T12:46:33.990925+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:46.948320+0000","scrub_duration":115,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"390'21","reported_seq":799,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254119+0000","last_change":"2024-03-24T12:46:29.784711+0000","last_active":"2024-03-24T12:47:26.254119+0000","last_peered":"2024-03-24T12:47:26.254119+0000","last_clean":"2024-03-24T12:47:26.254119+0000","last_became_active":"2024-03-24T12:26:05.317890+0000","last_became_peered":"2024-03-24T12:26:05.317890+0000","last_unstale":"2024-03-24T12:47:26.254119+0000","last_undegraded":"2024-03-24T12:47:26.254119+0000","last_fullsized":"2024-03-24T12:47:26.254119+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:46:29.784586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:46:29.784586+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:32.849608+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":483,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254923+0000","last_change":"2024-03-24T12:46:37.895180+0000","last_active":"2024-03-24T12:47:26.254923+0000","last_peered":"2024-03-24T12:47:26.254923+0000","last_clean":"2024-03-24T12:47:26.254923+0000","last_became_active":"2024-03-24T12:45:40.249430+0000","last_became_peered":"2024-03-24T12:45:40.249430+0000","last_unstale":"2024-03-24T12:47:26.254923+0000","last_undegraded":"2024-03-24T12:47:26.254923+0000","last_fullsized":"2024-03-24T12:47:26.254923+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:37.895098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:37.895098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:41.862772+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":470,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.261138+0000","last_change":"2024-03-24T12:46:18.200110+0000","last_active":"2024-03-24T12:47:26.261138+0000","last_peered":"2024-03-24T12:47:26.261138+0000","last_clean":"2024-03-24T12:47:26.261138+0000","last_became_active":"2024-03-24T12:45:40.250171+0000","last_became_peered":"2024-03-24T12:45:40.250171+0000","last_unstale":"2024-03-24T12:47:26.261138+0000","last_undegraded":"2024-03-24T12:47:26.261138+0000","last_fullsized":"2024-03-24T12:47:26.261138+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:18.199684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:46:18.199684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:44.538094+0000","scrub_duration":166,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":765,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.259631+0000","last_change":"2024-03-24T12:46:06.136390+0000","last_active":"2024-03-24T12:47:26.259631+0000","last_peered":"2024-03-24T12:47:26.259631+0000","last_clean":"2024-03-24T12:47:26.259631+0000","last_became_active":"2024-03-24T12:26:09.351085+0000","last_became_peered":"2024-03-24T12:26:09.351085+0000","last_unstale":"2024-03-24T12:47:26.259631+0000","last_undegraded":"2024-03-24T12:47:26.259631+0000","last_fullsized":"2024-03-24T12:47:26.259631+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:06.135893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:46:06.135893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"391'46","reported_seq":465,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.424597+0000","last_change":"2024-03-24T12:45:40.299662+0000","last_active":"2024-03-24T12:47:05.424597+0000","last_peered":"2024-03-24T12:47:05.424597+0000","last_clean":"2024-03-24T12:47:05.424597+0000","last_became_active":"2024-03-24T12:45:40.247792+0000","last_became_peered":"2024-03-24T12:45:40.247792+0000","last_unstale":"2024-03-24T12:47:05.424597+0000","last_undegraded":"2024-03-24T12:47:05.424597+0000","last_fullsized":"2024-03-24T12:47:05.424597+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'34","last_scrub_stamp":"2024-03-24T12:45:07.975149+0000","last_deep_scrub":"207'34","last_deep_scrub_stamp":"2024-03-24T12:45:07.975149+0000","last_clean_scrub_stamp":"2024-03-24T12:45:07.975149+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:05.587795+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":759,"reported_epoch":408,"state":"active+clean","last_fresh":"2024-03-24T12:47:05.423708+0000","last_change":"2024-03-24T12:46:13.616359+0000","last_active":"2024-03-24T12:47:05.423708+0000","last_peered":"2024-03-24T12:47:05.423708+0000","last_clean":"2024-03-24T12:47:05.423708+0000","last_became_active":"2024-03-24T12:45:40.244546+0000","last_became_peered":"2024-03-24T12:45:40.244546+0000","last_unstale":"2024-03-24T12:47:05.423708+0000","last_undegraded":"2024-03-24T12:47:05.423708+0000","last_fullsized":"2024-03-24T12:47:05.423708+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:46:13.616208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:46:13.616208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:26.870863+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":779,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.254109+0000","last_change":"2024-03-24T12:45:13.311076+0000","last_active":"2024-03-24T12:47:26.254109+0000","last_peered":"2024-03-24T12:47:26.254109+0000","last_clean":"2024-03-24T12:47:26.254109+0000","last_became_active":"2024-03-24T12:45:13.310787+0000","last_became_peered":"2024-03-24T12:45:13.310787+0000","last_unstale":"2024-03-24T12:47:26.254109+0000","last_undegraded":"2024-03-24T12:47:26.254109+0000","last_fullsized":"2024-03-24T12:47:26.254109+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:46.508221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:17.828354+0000","last_clean_scrub_stamp":"2024-03-24T12:44:46.508221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912148+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":741,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.256367+0000","last_change":"2024-03-24T12:45:47.024216+0000","last_active":"2024-03-24T12:47:26.256367+0000","last_peered":"2024-03-24T12:47:26.256367+0000","last_clean":"2024-03-24T12:47:26.256367+0000","last_became_active":"2024-03-24T12:45:40.243426+0000","last_became_peered":"2024-03-24T12:45:40.243426+0000","last_unstale":"2024-03-24T12:47:26.256367+0000","last_undegraded":"2024-03-24T12:47:26.256367+0000","last_fullsized":"2024-03-24T12:47:26.256367+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:47.024115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:45:47.024115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448709+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"390'21","reported_seq":863,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253619+0000","last_change":"2024-03-24T12:45:40.261157+0000","last_active":"2024-03-24T12:47:26.253619+0000","last_peered":"2024-03-24T12:47:26.253619+0000","last_clean":"2024-03-24T12:47:26.253619+0000","last_became_active":"2024-03-24T12:45:40.229470+0000","last_became_peered":"2024-03-24T12:45:40.229470+0000","last_unstale":"2024-03-24T12:47:26.253619+0000","last_undegraded":"2024-03-24T12:47:26.253619+0000","last_fullsized":"2024-03-24T12:47:26.253619+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"207'15","last_scrub_stamp":"2024-03-24T12:44:43.532996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:44:43.532996+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912199+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":754,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.260039+0000","last_change":"2024-03-24T12:45:13.322851+0000","last_active":"2024-03-24T12:47:26.260039+0000","last_peered":"2024-03-24T12:47:26.260039+0000","last_clean":"2024-03-24T12:47:26.260039+0000","last_became_active":"2024-03-24T12:45:13.322473+0000","last_became_peered":"2024-03-24T12:45:13.322473+0000","last_unstale":"2024-03-24T12:47:26.260039+0000","last_undegraded":"2024-03-24T12:47:26.260039+0000","last_fullsized":"2024-03-24T12:47:26.260039+0000","mapping_epoch":369,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":370,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:05.825276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:05.825276+0000","last_clean_scrub_stamp":"2024-03-24T12:45:05.825276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":662,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.257062+0000","last_change":"2024-03-24T12:44:53.810778+0000","last_active":"2024-03-24T12:47:26.257062+0000","last_peered":"2024-03-24T12:47:26.257062+0000","last_clean":"2024-03-24T12:47:26.257062+0000","last_became_active":"2024-03-24T12:44:48.869617+0000","last_became_peered":"2024-03-24T12:44:48.869617+0000","last_unstale":"2024-03-24T12:47:26.257062+0000","last_undegraded":"2024-03-24T12:47:26.257062+0000","last_fullsized":"2024-03-24T12:47:26.257062+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:44:53.810528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:44:53.810528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:28.448441+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":742,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-24T12:47:26.253800+0000","last_change":"2024-03-24T12:45:03.590860+0000","last_active":"2024-03-24T12:47:26.253800+0000","last_peered":"2024-03-24T12:47:26.253800+0000","last_clean":"2024-03-24T12:47:26.253800+0000","last_became_active":"2024-03-24T12:44:48.860436+0000","last_became_peered":"2024-03-24T12:44:48.860436+0000","last_unstale":"2024-03-24T12:47:26.253800+0000","last_undegraded":"2024-03-24T12:47:26.253800+0000","last_fullsized":"2024-03-24T12:47:26.253800+0000","mapping_epoch":353,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":354,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:45:03.590750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:45:03.590750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:47:29.912386+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":6757548032,"num_objects":103112,"num_object_clones":0,"num_object_copies":309336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":106342,"num_write_kb":6702528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20327895040,"data_stored":20327890968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":59157,"ondisk_log_size":59157,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1276,"num_read_kb":1271,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1408,"ondisk_log_size":1408,"up":33,"acting":33,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1671,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":6519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1546,"num_read_kb":1303,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":249,"num_bytes_recovered":5066,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1047,"ondisk_log_size":1047,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90,"ondisk_log_size":90,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038021,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4027488,"kb_used_data":3292192,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":89717664,"statfs":{"total":95995035648,"available":91870887936,"internally_reserved":0,"allocated":3371204608,"data_stored":3370167398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":752679370},"hb_peers":[0,1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":219043332363,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575141,"num_pgs":102,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6232024,"kb_used_data":5370640,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":87513128,"statfs":{"total":95995035648,"available":89613443072,"internally_reserved":0,"allocated":5499535360,"data_stored":5498318688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":882047429},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738666,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4655356,"kb_used_data":3725940,"kb_used_omap":1,"kb_used_meta":929406,"kb_avail":89089796,"statfs":{"total":95995035648,"available":91227951104,"internally_reserved":0,"allocated":3815362560,"data_stored":3814306089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":951712197},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,11],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411615,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5008108,"kb_used_data":4120356,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":88737044,"statfs":{"total":95995035648,"available":90866733056,"internally_reserved":0,"allocated":4219244544,"data_stored":4218072352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":909048261},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,2,2,2,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117273,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1156760,"kb_used_data":421528,"kb_used_omap":1,"kb_used_meta":735038,"kb_avail":92588392,"statfs":{"total":95995035648,"available":94810513408,"internally_reserved":0,"allocated":431644672,"data_stored":430678673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":752679367},"hb_peers":[0,1,2,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921045,"num_pgs":92,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3595296,"kb_used_data":2928096,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":90149856,"statfs":{"total":95995035648,"available":92313452544,"internally_reserved":0,"allocated":2998370304,"data_stored":2997332920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,1,2,3,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":2778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3805347840,"data_stored":3805347840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5553520640,"data_stored":5553520640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4224122880,"data_stored":4224122880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":424673280,"data_stored":424673280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2966491136,"data_stored":2966487064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3353739264,"data_stored":3353739264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:47:37.458 INFO:tasks.thrashosds.thrasher:Setting 6.c to [6, 5, 0, 2, 3, 1] 2024-03-24T12:47:37.458 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.c', '6', '5', '0', '2', '3', '1'] 2024-03-24T12:47:37.458 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pg-upmap-items 6.c 6 5 0 2 3 1 2024-03-24T12:47:37.476 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:47:37.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:37.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:37.523 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:37.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:37.544 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-mon[41500]: 6.b scrub starts 2024-03-24T12:47:37.544 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-mon[41500]: 4.12 scrub starts 2024-03-24T12:47:37.544 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-mon[41500]: 4.12 scrub ok 2024-03-24T12:47:37.544 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-mon[41500]: 6.b scrub ok 2024-03-24T12:47:37.544 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-mon[41500]: pgmap v956: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 17 MiB/s wr, 1.05k op/s 2024-03-24T12:47:37.545 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-mon[29919]: 6.b scrub starts 2024-03-24T12:47:37.545 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-mon[29919]: 4.12 scrub starts 2024-03-24T12:47:37.545 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-mon[29919]: 4.12 scrub ok 2024-03-24T12:47:37.545 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-mon[29919]: 6.b scrub ok 2024-03-24T12:47:37.545 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-mon[29919]: pgmap v956: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 17 MiB/s wr, 1.05k op/s 2024-03-24T12:47:37.570 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:47:37.583 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:47:37.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:37 smithi190 ceph-mon[33748]: 6.b scrub starts 2024-03-24T12:47:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:37 smithi190 ceph-mon[33748]: 4.12 scrub starts 2024-03-24T12:47:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:37 smithi190 ceph-mon[33748]: 4.12 scrub ok 2024-03-24T12:47:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:37 smithi190 ceph-mon[33748]: 6.b scrub ok 2024-03-24T12:47:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:37 smithi190 ceph-mon[33748]: pgmap v956: 158 pgs: 158 active+clean; 5.9 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 17 MiB/s wr, 1.05k op/s 2024-03-24T12:47:37.651 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:37.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:37.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:37.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:37.793 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:47:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:47:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:47:37.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:37.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:37.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:37.905 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:37.931 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:37.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:37.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:38.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:38.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:38.061 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:38.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:38.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:38.190 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:38.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:38.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:38.292 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:38.327 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:38.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:38.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:38.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:38.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:38.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:38.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:38.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:38.599 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:38.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:38.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:38.601 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:38 smithi190 ceph-mon[33748]: 4.3 scrub starts 2024-03-24T12:47:38.602 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:38 smithi190 ceph-mon[33748]: 4.3 scrub ok 2024-03-24T12:47:38.602 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:38 smithi190 ceph-mon[33748]: 6.12 scrub starts 2024-03-24T12:47:38.602 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:38 smithi190 ceph-mon[33748]: 6.12 scrub ok 2024-03-24T12:47:38.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:38 smithi029 ceph-mon[41500]: 4.3 scrub starts 2024-03-24T12:47:38.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:38 smithi029 ceph-mon[41500]: 4.3 scrub ok 2024-03-24T12:47:38.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:38 smithi029 ceph-mon[41500]: 6.12 scrub starts 2024-03-24T12:47:38.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:38 smithi029 ceph-mon[41500]: 6.12 scrub ok 2024-03-24T12:47:38.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:38 smithi029 ceph-mon[29919]: 4.3 scrub starts 2024-03-24T12:47:38.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:38 smithi029 ceph-mon[29919]: 4.3 scrub ok 2024-03-24T12:47:38.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:38 smithi029 ceph-mon[29919]: 6.12 scrub starts 2024-03-24T12:47:38.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:38 smithi029 ceph-mon[29919]: 6.12 scrub ok 2024-03-24T12:47:38.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:38.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:38.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:38.831 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:38.858 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:38.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:38.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:47:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:47:38.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:38.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:39.000 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:39.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:39.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:39.125 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:39.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:39.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:39.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:39.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:39.367 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:39.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:39.406 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:39.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:39.507 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:39.531 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:39.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:39.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:39 smithi190 ceph-mon[33748]: 6.1f scrub starts 2024-03-24T12:47:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:39 smithi190 ceph-mon[33748]: 6.1f scrub ok 2024-03-24T12:47:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:39 smithi190 ceph-mon[33748]: 4.15 scrub starts 2024-03-24T12:47:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:39 smithi190 ceph-mon[33748]: 4.15 scrub ok 2024-03-24T12:47:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:39.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:39 smithi190 ceph-mon[33748]: pgmap v957: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 1.3 MiB/s wr, 1.01k op/s 2024-03-24T12:47:39.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:39.659 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:39.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:39.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[41500]: 6.1f scrub starts 2024-03-24T12:47:39.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[41500]: 6.1f scrub ok 2024-03-24T12:47:39.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[41500]: 4.15 scrub starts 2024-03-24T12:47:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[41500]: 4.15 scrub ok 2024-03-24T12:47:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:39.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[41500]: pgmap v957: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 1.3 MiB/s wr, 1.01k op/s 2024-03-24T12:47:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[29919]: 6.1f scrub starts 2024-03-24T12:47:39.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[29919]: 6.1f scrub ok 2024-03-24T12:47:39.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[29919]: 4.15 scrub starts 2024-03-24T12:47:39.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[29919]: 4.15 scrub ok 2024-03-24T12:47:39.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:39.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:39 smithi029 ceph-mon[29919]: pgmap v957: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.1 KiB/s rd, 1.3 MiB/s wr, 1.01k op/s 2024-03-24T12:47:39.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:39.800 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:39.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:39.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:39.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:39.928 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:39.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:40.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:40.071 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:40.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:40.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:40.173 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:40.197 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:40.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:40.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:40.326 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:40.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:40.428 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:40.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:40.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:40.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:40.593 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:40.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:40.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:40.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:40.721 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:40.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:40.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:40.849 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:40.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:40.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:40.951 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:40.974 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:40.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:41.076 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:41.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:41.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:41.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:41.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:41.230 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:41.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:41.331 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:41.373 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:41.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:41.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:41.475 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:41.501 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:41.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:41.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:41.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[29919]: 3.12 scrub starts 2024-03-24T12:47:41.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[29919]: 3.12 scrub ok 2024-03-24T12:47:41.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:47:41.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:47:41.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[29919]: osdmap e412: 8 total, 7 up, 6 in 2024-03-24T12:47:41.556 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[29919]: pgmap v959: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:47:41.556 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[41500]: 3.12 scrub starts 2024-03-24T12:47:41.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[41500]: 3.12 scrub ok 2024-03-24T12:47:41.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:47:41.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:47:41.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[41500]: osdmap e412: 8 total, 7 up, 6 in 2024-03-24T12:47:41.557 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:41 smithi029 ceph-mon[41500]: pgmap v959: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:47:41.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:41 smithi190 ceph-mon[33748]: 3.12 scrub starts 2024-03-24T12:47:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:41 smithi190 ceph-mon[33748]: 3.12 scrub ok 2024-03-24T12:47:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:41 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:47:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:41 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:47:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:41 smithi190 ceph-mon[33748]: osdmap e412: 8 total, 7 up, 6 in 2024-03-24T12:47:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:41 smithi190 ceph-mon[33748]: pgmap v959: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-24T12:47:41.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:41.643 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:41.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:41.744 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:41.770 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:41.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:41.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:41.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:41.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:41.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:42.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:42.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:42.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:42.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:42.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:42.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:42.169 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:42.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:42.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:42.296 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:42.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:42.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:42.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:42.436 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:42.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:42.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:42.566 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:42.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:42 smithi190 ceph-mon[33748]: 3.1f scrub starts 2024-03-24T12:47:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:42 smithi190 ceph-mon[33748]: 3.1f scrub ok 2024-03-24T12:47:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:42 smithi190 ceph-mon[33748]: 5.13 scrub starts 2024-03-24T12:47:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:42 smithi190 ceph-mon[33748]: 5.13 scrub ok 2024-03-24T12:47:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:42 smithi190 ceph-mon[33748]: 4.10 scrub starts 2024-03-24T12:47:42.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:42 smithi190 ceph-mon[33748]: 4.10 scrub ok 2024-03-24T12:47:42.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[41500]: 3.1f scrub starts 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[41500]: 3.1f scrub ok 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[41500]: 5.13 scrub starts 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[41500]: 5.13 scrub ok 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[41500]: 4.10 scrub starts 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[41500]: 4.10 scrub ok 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[29919]: 3.1f scrub starts 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[29919]: 3.1f scrub ok 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[29919]: 5.13 scrub starts 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[29919]: 5.13 scrub ok 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[29919]: 4.10 scrub starts 2024-03-24T12:47:42.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:42 smithi029 ceph-mon[29919]: 4.10 scrub ok 2024-03-24T12:47:42.668 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:42.691 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:42.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:42.793 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:42.817 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:42.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:42.918 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:42.945 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:42.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:42.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:43.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:43.083 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:43.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:43.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:43.208 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:43.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:43.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:43.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:43.336 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:43.337 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:43.438 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:43.464 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:43.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:43.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:43.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:43.591 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:43.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:43.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:43 smithi190 ceph-mon[33748]: 5.d scrub starts 2024-03-24T12:47:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:43 smithi190 ceph-mon[33748]: 5.d scrub ok 2024-03-24T12:47:43.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:43 smithi190 ceph-mon[33748]: pgmap v960: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 986 op/s 2024-03-24T12:47:43.680 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:43 smithi029 ceph-mon[29919]: 5.d scrub starts 2024-03-24T12:47:43.681 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:43 smithi029 ceph-mon[29919]: 5.d scrub ok 2024-03-24T12:47:43.681 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:43 smithi029 ceph-mon[29919]: pgmap v960: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 986 op/s 2024-03-24T12:47:43.681 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:43 smithi029 ceph-mon[41500]: 5.d scrub starts 2024-03-24T12:47:43.681 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:43 smithi029 ceph-mon[41500]: 5.d scrub ok 2024-03-24T12:47:43.681 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:43 smithi029 ceph-mon[41500]: pgmap v960: 158 pgs: 158 active+clean; 5.5 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 986 op/s 2024-03-24T12:47:43.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:43.741 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:43.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:43.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:43.843 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:43.868 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:43.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:43.969 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:43.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:43.998 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:43.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:44.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:44.136 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:44.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:44.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:44.276 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:44.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:44.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:44.378 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:44.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:44.402 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:44.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:44.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:44.542 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:44.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:44.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:44.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:44 smithi190 ceph-mon[33748]: 6.7 scrub starts 2024-03-24T12:47:44.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:44 smithi190 ceph-mon[33748]: 6.7 scrub ok 2024-03-24T12:47:44.644 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:44.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:44 smithi029 ceph-mon[29919]: 6.7 scrub starts 2024-03-24T12:47:44.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:44 smithi029 ceph-mon[29919]: 6.7 scrub ok 2024-03-24T12:47:44.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:44 smithi029 ceph-mon[41500]: 6.7 scrub starts 2024-03-24T12:47:44.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:44 smithi029 ceph-mon[41500]: 6.7 scrub ok 2024-03-24T12:47:44.667 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:44.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:44.770 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:44.796 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:44.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:44.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:44.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:44.939 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:44.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:44.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:45.041 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:45.067 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:45.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:45.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:45.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:45.198 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:45.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:45.299 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:45.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:45.342 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:45.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:45.443 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:45.468 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:45.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:45.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:45.594 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:45.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:45.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:45.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:45 smithi190 ceph-mon[33748]: 11.c scrub starts 2024-03-24T12:47:45.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:45 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:47:45.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:45 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:47:45.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:45 smithi190 ceph-mon[33748]: 6.1d scrub starts 2024-03-24T12:47:45.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:45 smithi190 ceph-mon[33748]: 6.1d scrub ok 2024-03-24T12:47:45.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:45 smithi190 ceph-mon[33748]: pgmap v961: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 910 op/s 2024-03-24T12:47:45.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[29919]: 11.c scrub starts 2024-03-24T12:47:45.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:47:45.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:47:45.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[29919]: 6.1d scrub starts 2024-03-24T12:47:45.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[29919]: 6.1d scrub ok 2024-03-24T12:47:45.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[29919]: pgmap v961: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 910 op/s 2024-03-24T12:47:45.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[41500]: 11.c scrub starts 2024-03-24T12:47:45.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:47:45.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:47:45.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[41500]: 6.1d scrub starts 2024-03-24T12:47:45.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[41500]: 6.1d scrub ok 2024-03-24T12:47:45.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:45 smithi029 ceph-mon[41500]: pgmap v961: 158 pgs: 158 active+clean; 5.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 910 op/s 2024-03-24T12:47:45.697 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:45.723 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:45.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:45.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:45.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:45.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:45.850 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:45.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:45.951 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:45.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:45.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:46.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:46.109 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:46.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:46.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:46.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:46.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:46.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:46.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:46.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:46.372 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:46.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:46.473 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:46.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:46.498 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:46.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:46.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:46.622 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:46.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:46.632 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:47:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a[66309]: logger=cleanup t=2024-03-24T12:47:46.289901538Z level=info msg="Completed cleanup jobs" duration=38.580887ms 2024-03-24T12:47:46.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:46 smithi190 ceph-mon[33748]: osdmap e413: 8 total, 7 up, 6 in 2024-03-24T12:47:46.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:46 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:47:46.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:46 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:47:46.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:46 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:47:46.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:46 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:47:46.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:46 smithi029 ceph-mon[29919]: osdmap e413: 8 total, 7 up, 6 in 2024-03-24T12:47:46.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:46 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:47:46.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:46 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:47:46.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:46 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:47:46.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:46 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:47:46.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:46 smithi029 ceph-mon[41500]: osdmap e413: 8 total, 7 up, 6 in 2024-03-24T12:47:46.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:46 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:47:46.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:46 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:47:46.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:46 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:47:46.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:46 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:47:46.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:46.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:46.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:46.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:46.912 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:46.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:47.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:47.039 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:47.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:47.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:47.141 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:47.170 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:47.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:47.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:47.295 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:47.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:47.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:47.396 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:47.421 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:47.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:47.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:47.523 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:47.550 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:47.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:47.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:47 smithi029 ceph-mon[41500]: 5.a scrub starts 2024-03-24T12:47:47.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:47 smithi029 ceph-mon[41500]: 5.a scrub ok 2024-03-24T12:47:47.605 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:47 smithi029 ceph-mon[41500]: pgmap v963: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 751 op/s 2024-03-24T12:47:47.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:47 smithi029 ceph-mon[29919]: 5.a scrub starts 2024-03-24T12:47:47.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:47 smithi029 ceph-mon[29919]: 5.a scrub ok 2024-03-24T12:47:47.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:47 smithi029 ceph-mon[29919]: pgmap v963: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 751 op/s 2024-03-24T12:47:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:47 smithi190 ceph-mon[33748]: 5.a scrub starts 2024-03-24T12:47:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:47 smithi190 ceph-mon[33748]: 5.a scrub ok 2024-03-24T12:47:47.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:47 smithi190 ceph-mon[33748]: pgmap v963: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.3 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 751 op/s 2024-03-24T12:47:47.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:47.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:47.779 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:47.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:47.806 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:47.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:47.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:47.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:47:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:47:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:47:47.946 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:47.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:48.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:48.074 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:48.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:48.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:48.202 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:48.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:48.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:48.304 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:48.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:48.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:48.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:48.455 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:48.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:48.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:48.585 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:48.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:48.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:48.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:48 smithi190 ceph-mon[33748]: 3.8 scrub starts 2024-03-24T12:47:48.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:48 smithi190 ceph-mon[33748]: 3.8 scrub ok 2024-03-24T12:47:48.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:48 smithi190 ceph-mon[33748]: 11.c scrub ok 2024-03-24T12:47:48.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:48 smithi190 ceph-mon[33748]: 5.4 scrub starts 2024-03-24T12:47:48.611 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:48 smithi190 ceph-mon[33748]: 5.4 scrub ok 2024-03-24T12:47:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-mon[29919]: 3.8 scrub starts 2024-03-24T12:47:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-mon[29919]: 3.8 scrub ok 2024-03-24T12:47:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-mon[29919]: 11.c scrub ok 2024-03-24T12:47:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-mon[29919]: 5.4 scrub starts 2024-03-24T12:47:48.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-mon[29919]: 5.4 scrub ok 2024-03-24T12:47:48.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:47:48.509+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:47:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-mon[41500]: 3.8 scrub starts 2024-03-24T12:47:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-mon[41500]: 3.8 scrub ok 2024-03-24T12:47:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-mon[41500]: 11.c scrub ok 2024-03-24T12:47:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-mon[41500]: 5.4 scrub starts 2024-03-24T12:47:48.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:48 smithi029 ceph-mon[41500]: 5.4 scrub ok 2024-03-24T12:47:48.686 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:48.716 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:48.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:48.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:48.818 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:48.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:48.856 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:48.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:48.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:47:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:47:48.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:48.989 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:48.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:49.090 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:49.114 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:49.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:49.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:49.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:49.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:49.255 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:49.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:49.356 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:49.398 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:49.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:49.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:49.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:49.526 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:49.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:49.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:49.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:49 smithi190 ceph-mon[33748]: 3.4 scrub starts 2024-03-24T12:47:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:49 smithi190 ceph-mon[33748]: 3.4 scrub ok 2024-03-24T12:47:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:49 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:47:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:49 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:47:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:49 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:47:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:49 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:47:49.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:49 smithi190 ceph-mon[33748]: pgmap v964: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[29919]: 3.4 scrub starts 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[29919]: 3.4 scrub ok 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[29919]: pgmap v964: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[41500]: 3.4 scrub starts 2024-03-24T12:47:49.639 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[41500]: 3.4 scrub ok 2024-03-24T12:47:49.640 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:47:49.640 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:47:49.640 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:47:49.640 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:49.640 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:47:49.640 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:47:49.640 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:49 smithi029 ceph-mon[41500]: pgmap v964: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 910 op/s 2024-03-24T12:47:49.678 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:49.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:49.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:49.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:49.807 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:49.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:49.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:49.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:49.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:49.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:50.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:50.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:50.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:50.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:50.213 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:50.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:50.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:50.315 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:50.360 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:50.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:50.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:47:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:47:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:47:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:47:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:47:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:47:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:47:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:47:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:47:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:47:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:47:50.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:50 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:47:50.462 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:50.491 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:50.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:50.593 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:50.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:50.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:50 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:47:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:50 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:47:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:50 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:47:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:50 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:47:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:50 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:47:50.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:50 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:47:50.722 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:50.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:50.771 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:50.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:50.872 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:50.898 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:50.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:50.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:51.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:51.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:51.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:51.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:51.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:51.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:51.176 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:51.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:51.277 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:51.304 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:51.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:51.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:51.406 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:51.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:51 smithi029 ceph-mon[41500]: 5.b scrub starts 2024-03-24T12:47:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:51 smithi029 ceph-mon[41500]: 5.b scrub ok 2024-03-24T12:47:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:51 smithi029 ceph-mon[41500]: 3.c scrub starts 2024-03-24T12:47:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:51 smithi029 ceph-mon[41500]: 3.c scrub ok 2024-03-24T12:47:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:51 smithi029 ceph-mon[41500]: pgmap v965: 158 pgs: 158 active+clean; 5.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 907 op/s 2024-03-24T12:47:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:51 smithi029 ceph-mon[29919]: 5.b scrub starts 2024-03-24T12:47:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:51 smithi029 ceph-mon[29919]: 5.b scrub ok 2024-03-24T12:47:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:51 smithi029 ceph-mon[29919]: 3.c scrub starts 2024-03-24T12:47:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:51 smithi029 ceph-mon[29919]: 3.c scrub ok 2024-03-24T12:47:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:51 smithi029 ceph-mon[29919]: pgmap v965: 158 pgs: 158 active+clean; 5.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 907 op/s 2024-03-24T12:47:51.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:51.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:51.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:51.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:51.582 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:51.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:51.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:51 smithi190 ceph-mon[33748]: 5.b scrub starts 2024-03-24T12:47:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:51 smithi190 ceph-mon[33748]: 5.b scrub ok 2024-03-24T12:47:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:51 smithi190 ceph-mon[33748]: 3.c scrub starts 2024-03-24T12:47:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:51 smithi190 ceph-mon[33748]: 3.c scrub ok 2024-03-24T12:47:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:51 smithi190 ceph-mon[33748]: pgmap v965: 158 pgs: 158 active+clean; 5.1 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 907 op/s 2024-03-24T12:47:51.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:51.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:51.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:51.821 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:51.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:51.848 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:51.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:51.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:51.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:51.991 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:51.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:52.092 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:52.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:52.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:52.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:52.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:52.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:52.264 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:52.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:47:52.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:52.418 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:52.419 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:52.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:52.568 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:52.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:52.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:52 smithi190 ceph-mon[33748]: 11.d scrub starts 2024-03-24T12:47:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:52 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:47:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:52 smithi190 ceph-mon[33748]: 4.11 scrub ok 2024-03-24T12:47:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:52 smithi190 ceph-mon[33748]: 4.8 scrub starts 2024-03-24T12:47:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:52 smithi190 ceph-mon[33748]: 4.8 scrub ok 2024-03-24T12:47:52.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:52 smithi029 ceph-mon[29919]: 11.d scrub starts 2024-03-24T12:47:52.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:52 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:47:52.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:52 smithi029 ceph-mon[29919]: 4.11 scrub ok 2024-03-24T12:47:52.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:52 smithi029 ceph-mon[29919]: 4.8 scrub starts 2024-03-24T12:47:52.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:52 smithi029 ceph-mon[29919]: 4.8 scrub ok 2024-03-24T12:47:52.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:52 smithi029 ceph-mon[41500]: 11.d scrub starts 2024-03-24T12:47:52.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:52 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:47:52.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:52 smithi029 ceph-mon[41500]: 4.11 scrub ok 2024-03-24T12:47:52.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:52 smithi029 ceph-mon[41500]: 4.8 scrub starts 2024-03-24T12:47:52.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:52 smithi029 ceph-mon[41500]: 4.8 scrub ok 2024-03-24T12:47:52.670 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:52.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:52.699 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:52.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:52.800 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:52.826 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:52.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:52.928 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:52.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:52.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:52.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:53.058 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:53.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:53.192 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:53.218 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:53.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:53.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:53.320 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:53.345 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:53.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:53.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:53.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:53.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:53.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:53.623 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:53.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[29919]: 4.17 scrub starts 2024-03-24T12:47:53.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[29919]: 4.17 scrub ok 2024-03-24T12:47:53.628 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[29919]: 11.6 scrub starts 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[29919]: 4.7 scrub starts 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[29919]: 4.7 scrub ok 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[29919]: 5.1a scrub starts 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[29919]: 5.1a scrub ok 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[29919]: pgmap v966: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 931 op/s 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[41500]: 4.17 scrub starts 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[41500]: 4.17 scrub ok 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[41500]: 11.6 scrub starts 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[41500]: 4.7 scrub starts 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[41500]: 4.7 scrub ok 2024-03-24T12:47:53.629 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[41500]: 5.1a scrub starts 2024-03-24T12:47:53.630 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[41500]: 5.1a scrub ok 2024-03-24T12:47:53.630 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:53 smithi029 ceph-mon[41500]: pgmap v966: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 931 op/s 2024-03-24T12:47:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:53 smithi190 ceph-mon[33748]: 4.17 scrub starts 2024-03-24T12:47:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:53 smithi190 ceph-mon[33748]: 4.17 scrub ok 2024-03-24T12:47:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:53 smithi190 ceph-mon[33748]: 11.6 scrub starts 2024-03-24T12:47:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:53 smithi190 ceph-mon[33748]: 4.7 scrub starts 2024-03-24T12:47:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:53 smithi190 ceph-mon[33748]: 4.7 scrub ok 2024-03-24T12:47:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:53 smithi190 ceph-mon[33748]: 5.1a scrub starts 2024-03-24T12:47:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:53 smithi190 ceph-mon[33748]: 5.1a scrub ok 2024-03-24T12:47:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:53 smithi190 ceph-mon[33748]: pgmap v966: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 931 op/s 2024-03-24T12:47:53.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:53.653 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:53.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:53.754 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:53.782 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:53.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:53.883 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:53.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:53.916 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:53.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:54.017 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:54.043 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:54.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:54.144 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:54.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:54.172 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:54.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:54.273 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:54.326 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:54.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:54.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:54.370 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:47:54.371 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:47:54.371 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-03-24T12:47:54.371 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-03-24T12:47:54.371 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:47:54.428 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:54.468 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:54.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:54.570 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:54.595 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:54.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:54.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:54.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:54 smithi029 ceph-mon[41500]: 11.d scrub ok 2024-03-24T12:47:54.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:54 smithi029 ceph-mon[41500]: 11.1 scrub starts 2024-03-24T12:47:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:54 smithi029 ceph-mon[29919]: 11.d scrub ok 2024-03-24T12:47:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:54 smithi029 ceph-mon[29919]: 11.1 scrub starts 2024-03-24T12:47:54.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:54.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:54.737 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:54.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:54.838 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:54.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:54.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:54.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:54.881 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:54 smithi190 ceph-mon[33748]: 11.d scrub ok 2024-03-24T12:47:54.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:54 smithi190 ceph-mon[33748]: 11.1 scrub starts 2024-03-24T12:47:54.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:55.016 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:55.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:55.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:55.118 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:55.161 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:55.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:55.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:55.308 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:55.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:55.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:55.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:55.435 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:55.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:55.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:55.577 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:55.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:55.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:55 smithi029 ceph-mon[41500]: 11.6 scrub ok 2024-03-24T12:47:55.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:55 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:47:55.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:55 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:47:55.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:55 smithi029 ceph-mon[41500]: pgmap v967: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 963 op/s 2024-03-24T12:47:55.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:55 smithi029 ceph-mon[29919]: 11.6 scrub ok 2024-03-24T12:47:55.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:55 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:47:55.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:55 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:47:55.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:55 smithi029 ceph-mon[29919]: pgmap v967: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 963 op/s 2024-03-24T12:47:55.679 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:55.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:55.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:55.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:55.807 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:55.853 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:55.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:55.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:55 smithi190 ceph-mon[33748]: 11.6 scrub ok 2024-03-24T12:47:55.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:55 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:47:55.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:55 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:47:55.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:55 smithi190 ceph-mon[33748]: pgmap v967: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 963 op/s 2024-03-24T12:47:55.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:56.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:56.002 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:56.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:56.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:56.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:56.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:56.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:56.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:56.291 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:56.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:56.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:56.436 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:56.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:56.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:56.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:56.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:56.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:56.665 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:56.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:56 smithi029 ceph-mon[29919]: 11.2 scrub starts 2024-03-24T12:47:56.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:56 smithi029 ceph-mon[29919]: 11.1 scrub ok 2024-03-24T12:47:56.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:56 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:47:56.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:56 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:47:56.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:56 smithi029 ceph-mon[41500]: 11.2 scrub starts 2024-03-24T12:47:56.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:56 smithi029 ceph-mon[41500]: 11.1 scrub ok 2024-03-24T12:47:56.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:56 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:47:56.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:56 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:47:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:56.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:56.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:56.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:56.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:56.838 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:56.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:56.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:56 smithi190 ceph-mon[33748]: 11.2 scrub starts 2024-03-24T12:47:56.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:56 smithi190 ceph-mon[33748]: 11.1 scrub ok 2024-03-24T12:47:56.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:56 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:47:56.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:56 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:47:56.939 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:56.965 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:56.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:56.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:57.067 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:57.092 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:57.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:57.194 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:57.220 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:57.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:57.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:57.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:47:57.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:57.372 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:57.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:47:57.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:57.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:57.516 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:57.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:57.617 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:57.643 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:57.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:57.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:47:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:47:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[41500]: 1.6 scrub starts 2024-03-24T12:47:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[41500]: 1.6 scrub ok 2024-03-24T12:47:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[41500]: 6.17 scrub starts 2024-03-24T12:47:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[41500]: pgmap v968: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 886 B/s rd, 0 B/s wr, 848 op/s 2024-03-24T12:47:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[41500]: 6.17 scrub ok 2024-03-24T12:47:57.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:47:57.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:47:57.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[29919]: 1.6 scrub starts 2024-03-24T12:47:57.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[29919]: 1.6 scrub ok 2024-03-24T12:47:57.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[29919]: 6.17 scrub starts 2024-03-24T12:47:57.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[29919]: pgmap v968: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 886 B/s rd, 0 B/s wr, 848 op/s 2024-03-24T12:47:57.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-mon[29919]: 6.17 scrub ok 2024-03-24T12:47:57.668 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:47:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:47:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:47:57.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:57.787 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:57.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:57.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:57.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:57 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:47:57.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:57 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:47:57.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:57 smithi190 ceph-mon[33748]: 1.6 scrub starts 2024-03-24T12:47:57.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:57 smithi190 ceph-mon[33748]: 1.6 scrub ok 2024-03-24T12:47:57.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:57 smithi190 ceph-mon[33748]: 6.17 scrub starts 2024-03-24T12:47:57.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:57 smithi190 ceph-mon[33748]: pgmap v968: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.7 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 886 B/s rd, 0 B/s wr, 848 op/s 2024-03-24T12:47:57.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:57 smithi190 ceph-mon[33748]: 6.17 scrub ok 2024-03-24T12:47:57.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:57.916 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:57.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:57.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:58.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:58.062 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:58.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:58.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:58.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:58.210 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:58.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:58.285 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:47:58.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:58.328 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:47:58.353 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:58.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:58.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:58.371 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:47:58.411 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:47:58.438 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:47:58.450 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:47:58.455 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:58.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:58.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:58.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:58.485 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:47:58.505 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:47:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:47:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:47:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:47:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:47:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:47:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:47:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:47:58.506 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:47:58.581 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:58.610 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:58.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:58.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:58 smithi029 ceph-mon[41500]: 11.2 scrub ok 2024-03-24T12:47:58.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:58 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1495722592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:47:58.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:58 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3511517487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:47:58.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:58 smithi029 ceph-mon[29919]: 11.2 scrub ok 2024-03-24T12:47:58.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:58 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1495722592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:47:58.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:58 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3511517487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:47:58.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:47:58.737 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:58.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:47:58.783 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:47:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:47:58.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:58 smithi190 ceph-mon[33748]: 11.2 scrub ok 2024-03-24T12:47:58.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:58 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1495722592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:47:58.783 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:58 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3511517487' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:47:58.840 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:58.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:58.867 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:58.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:58.969 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:47:58.995 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:58.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:47:59.097 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:59.143 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:59.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:59.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:59.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T12:47:59.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:47:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:59.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:59.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:47:59.411 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:47:59.437 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:47:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:59.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:47:59.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:47:59.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:59.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:59.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:47:59.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:59.714 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:59.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:59.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:59.852 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:59.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:59.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:47:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:59 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:47:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:59 smithi190 ceph-mon[33748]: Cluster is now healthy 2024-03-24T12:47:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:59 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1495722592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:47:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:59 smithi190 ceph-mon[33748]: osdmap e414: 8 total, 7 up, 6 in 2024-03-24T12:47:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:59 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:47:59.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:59 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:47:59.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:47:59 smithi190 ceph-mon[33748]: pgmap v970: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T12:47:59.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:47:59.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[41500]: Cluster is now healthy 2024-03-24T12:47:59.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1495722592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:47:59.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[41500]: osdmap e414: 8 total, 7 up, 6 in 2024-03-24T12:47:59.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:59.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:47:59.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:47:59.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[41500]: pgmap v970: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T12:47:59.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:47:59.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[29919]: Cluster is now healthy 2024-03-24T12:47:59.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1495722592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:47:59.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[29919]: osdmap e414: 8 total, 7 up, 6 in 2024-03-24T12:47:59.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:47:59.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:47:59.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:47:59.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:47:59 smithi029 ceph-mon[29919]: pgmap v970: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.5 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.05k op/s 2024-03-24T12:47:59.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:47:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:47:59.985 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:47:59.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:00.086 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:00.112 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:00.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:00.214 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:00.241 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:00.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:00.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:00.343 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:00.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:00.367 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:00.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:00.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:00.495 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:00.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:00.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:00.597 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:00.623 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:00.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:00.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:00.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:00.749 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:00.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:00.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:00.874 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:00.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:00.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:00 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:48:00.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:00 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:48:00.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:00 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:48:00.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:00 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:48:00.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:00 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:48:00.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:00 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:48:00.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:00 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:48:00.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:00 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:48:00.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:00 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:48:00.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:00 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:48:00.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:00 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:48:00.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:00 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:48:00.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:01.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:01.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:01.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:01.102 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:01.131 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:01.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:01.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:01.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:01.275 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:01.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:01.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:01.410 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:01.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:01.511 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:01.540 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:01.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:01.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:01.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:01.677 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:01.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:01.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:01.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:01.802 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:01.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:01.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:01 smithi190 ceph-mon[33748]: 5.e scrub starts 2024-03-24T12:48:01.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:01 smithi190 ceph-mon[33748]: 5.e scrub ok 2024-03-24T12:48:01.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:01 smithi190 ceph-mon[33748]: pgmap v971: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T12:48:01.902 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:01.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:01 smithi029 ceph-mon[29919]: 5.e scrub starts 2024-03-24T12:48:01.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:01 smithi029 ceph-mon[29919]: 5.e scrub ok 2024-03-24T12:48:01.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:01 smithi029 ceph-mon[29919]: pgmap v971: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T12:48:01.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:01 smithi029 ceph-mon[41500]: 5.e scrub starts 2024-03-24T12:48:01.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:01 smithi029 ceph-mon[41500]: 5.e scrub ok 2024-03-24T12:48:01.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:01 smithi029 ceph-mon[41500]: pgmap v971: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-24T12:48:01.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:01.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:01.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:02.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:02.073 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:02.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:02.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:02.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:02.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:02.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:02.306 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:02.334 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:02.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:02.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:02.459 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:02.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:02.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:02.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:02.602 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:02.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:02.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:02.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:02.728 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:02.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:02.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:02.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:02.854 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:02.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:02.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:02 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:48:02.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:02 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:48:02.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:02 smithi190 ceph-mon[33748]: 4.2 scrub starts 2024-03-24T12:48:02.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:02 smithi190 ceph-mon[33748]: 4.2 scrub ok 2024-03-24T12:48:02.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:02 smithi190 ceph-mon[33748]: osdmap e415: 8 total, 7 up, 6 in 2024-03-24T12:48:02.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:02 smithi190 ceph-mon[33748]: 11.4 scrub starts 2024-03-24T12:48:02.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:48:02.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:48:02.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[29919]: 4.2 scrub starts 2024-03-24T12:48:02.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[29919]: 4.2 scrub ok 2024-03-24T12:48:02.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[29919]: osdmap e415: 8 total, 7 up, 6 in 2024-03-24T12:48:02.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[29919]: 11.4 scrub starts 2024-03-24T12:48:02.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:48:02.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:48:02.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[41500]: 4.2 scrub starts 2024-03-24T12:48:02.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[41500]: 4.2 scrub ok 2024-03-24T12:48:02.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[41500]: osdmap e415: 8 total, 7 up, 6 in 2024-03-24T12:48:02.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:02 smithi029 ceph-mon[41500]: 11.4 scrub starts 2024-03-24T12:48:02.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:02.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:02.995 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:02.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:03.096 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:03.133 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:03.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:03.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:03.261 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:03.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:03.294 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:48:03.295 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:48:03.362 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:03.387 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:03.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:03.489 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:03.533 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:03.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:03.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:03 smithi029 ceph-mon[41500]: 11.b scrub starts 2024-03-24T12:48:03.548 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:03 smithi029 ceph-mon[41500]: pgmap v973: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T12:48:03.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:03 smithi029 ceph-mon[29919]: 11.b scrub starts 2024-03-24T12:48:03.549 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:03 smithi029 ceph-mon[29919]: pgmap v973: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T12:48:03.635 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:03.661 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:03.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:03.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:03.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:03.790 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:03.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:03.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:03.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:03 smithi190 ceph-mon[33748]: 11.b scrub starts 2024-03-24T12:48:03.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:03 smithi190 ceph-mon[33748]: pgmap v973: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.3 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-24T12:48:03.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:03.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:48:03 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:48:03.552+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:48:03.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:03.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:04.030 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:04.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:04.066 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:04.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:04.167 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:04.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:04.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:04.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:04.295 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:04.321 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:04.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:04.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:04.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:04.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:04.463 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:04.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:04.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:04.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:04.588 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:04.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:04.689 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:04.714 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:04.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:04.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:04.801 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:48:04.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:48:04.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[29919]: 11.4 scrub ok 2024-03-24T12:48:04.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[29919]: 3.18 deep-scrub starts 2024-03-24T12:48:04.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[29919]: 3.18 deep-scrub ok 2024-03-24T12:48:04.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:48:04.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:04.802 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[29919]: 11.b scrub ok 2024-03-24T12:48:04.802 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:48:04.802 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:48:04.802 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[41500]: 11.4 scrub ok 2024-03-24T12:48:04.803 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[41500]: 3.18 deep-scrub starts 2024-03-24T12:48:04.803 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[41500]: 3.18 deep-scrub ok 2024-03-24T12:48:04.803 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:48:04.803 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:04.803 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:04 smithi029 ceph-mon[41500]: 11.b scrub ok 2024-03-24T12:48:04.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:04.855 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:04.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:04.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:04.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:04 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:48:04.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:04 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:48:04.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:04 smithi190 ceph-mon[33748]: 11.4 scrub ok 2024-03-24T12:48:04.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:04 smithi190 ceph-mon[33748]: 3.18 deep-scrub starts 2024-03-24T12:48:04.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:04 smithi190 ceph-mon[33748]: 3.18 deep-scrub ok 2024-03-24T12:48:04.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:48:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:04.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:04 smithi190 ceph-mon[33748]: 11.b scrub ok 2024-03-24T12:48:04.957 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:04.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:04.981 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:04.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:05.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:05.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:05.119 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:05.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:05.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:05.259 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:05.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:05.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:05.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:48:05.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:05.388 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:05.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:05.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:05.490 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:05.516 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:05.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:05.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:05.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:05.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:05 smithi029 ceph-mon[29919]: 5.3 scrub starts 2024-03-24T12:48:05.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:05 smithi029 ceph-mon[29919]: 5.3 scrub ok 2024-03-24T12:48:05.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:05 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:48:05.653 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:05 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:48:05.654 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:05 smithi029 ceph-mon[29919]: pgmap v974: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:48:05.662 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:05.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:05.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:05.763 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:05.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:05.788 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:05.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:05.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:05.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:05 smithi029 ceph-mon[41500]: 5.3 scrub starts 2024-03-24T12:48:05.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:05 smithi029 ceph-mon[41500]: 5.3 scrub ok 2024-03-24T12:48:05.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:05 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:48:05.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:05 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:48:05.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:05 smithi029 ceph-mon[41500]: pgmap v974: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:48:05.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:05.931 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:05.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:06.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:06.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:06.073 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:06.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:06.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:05 smithi190 ceph-mon[33748]: 5.3 scrub starts 2024-03-24T12:48:06.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:05 smithi190 ceph-mon[33748]: 5.3 scrub ok 2024-03-24T12:48:06.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:05 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:48:06.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:05 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:48:06.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:05 smithi190 ceph-mon[33748]: pgmap v974: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.2 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:48:06.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:06.198 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:06.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:06.299 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:06.340 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:06.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:06.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:06.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:06.499 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:06.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:06.600 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:06.650 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:06.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:06.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:06.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:06.780 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:06.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:06.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:06.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:06.910 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:06.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:06.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:06.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:06 smithi029 ceph-mon[29919]: 11.8 scrub starts 2024-03-24T12:48:06.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:06 smithi029 ceph-mon[29919]: 6.1b deep-scrub starts 2024-03-24T12:48:06.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:06 smithi029 ceph-mon[29919]: 6.1b deep-scrub ok 2024-03-24T12:48:06.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:06 smithi029 ceph-mon[41500]: 11.8 scrub starts 2024-03-24T12:48:06.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:06 smithi029 ceph-mon[41500]: 6.1b deep-scrub starts 2024-03-24T12:48:06.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:06 smithi029 ceph-mon[41500]: 6.1b deep-scrub ok 2024-03-24T12:48:07.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:07.050 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:07.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:07.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:07.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:06 smithi190 ceph-mon[33748]: 11.8 scrub starts 2024-03-24T12:48:07.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:06 smithi190 ceph-mon[33748]: 6.1b deep-scrub starts 2024-03-24T12:48:07.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:06 smithi190 ceph-mon[33748]: 6.1b deep-scrub ok 2024-03-24T12:48:07.153 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:07.179 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:07.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:07.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:07.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:07.323 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:07.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:07.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:07.425 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:07.464 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:07.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:07.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:07.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:07.589 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:07.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:07.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:07.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:07.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:07.820 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:07.847 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:07.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:07.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:07.887 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:07 smithi029 ceph-mon[41500]: 11.3 scrub starts 2024-03-24T12:48:07.887 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:07 smithi029 ceph-mon[41500]: pgmap v975: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 605 B/s rd, 0 B/s wr, 708 op/s 2024-03-24T12:48:07.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:07 smithi029 ceph-mon[29919]: 11.3 scrub starts 2024-03-24T12:48:07.888 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:07 smithi029 ceph-mon[29919]: pgmap v975: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 605 B/s rd, 0 B/s wr, 708 op/s 2024-03-24T12:48:07.889 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:48:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:48:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:48:07.948 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:07.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:07.977 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:07.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:08.078 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:08.101 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:08.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:08.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:08.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:07 smithi190 ceph-mon[33748]: 11.3 scrub starts 2024-03-24T12:48:08.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:07 smithi190 ceph-mon[33748]: pgmap v975: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 4.1 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 605 B/s rd, 0 B/s wr, 708 op/s 2024-03-24T12:48:08.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:08.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:08.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:08.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:08.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:08.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:08.477 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:08.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:08.507 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:08.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:08.608 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:08.635 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:08.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:08.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:08.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:08.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:08.778 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:08.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:08.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:08.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:08 smithi190 ceph-mon[33748]: 11.8 scrub ok 2024-03-24T12:48:08.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:08 smithi190 ceph-mon[33748]: 11.3 scrub ok 2024-03-24T12:48:08.883 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:48:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:48:08.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:08.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:08.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:08 smithi029 ceph-mon[41500]: 11.8 scrub ok 2024-03-24T12:48:08.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:08 smithi029 ceph-mon[41500]: 11.3 scrub ok 2024-03-24T12:48:08.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:08 smithi029 ceph-mon[29919]: 11.8 scrub ok 2024-03-24T12:48:08.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:08 smithi029 ceph-mon[29919]: 11.3 scrub ok 2024-03-24T12:48:09.018 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:09.044 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:09.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:09.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:09.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:09.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:09.181 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:09.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:09.282 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:09.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:09.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:09.410 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:09.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:09.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:09.536 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:09.559 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:09.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:09.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:09.702 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:09.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:09.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:09.804 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:09.827 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:09.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:09.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:09.929 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:09.956 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:09.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:10.058 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:10.084 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:10.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:10.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:10.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:09 smithi190 ceph-mon[33748]: pgmap v976: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 958 op/s 2024-03-24T12:48:10.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:10.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:09 smithi029 ceph-mon[29919]: pgmap v976: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 958 op/s 2024-03-24T12:48:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:09 smithi029 ceph-mon[41500]: pgmap v976: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 958 op/s 2024-03-24T12:48:10.186 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:10.211 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:10.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:10.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:10.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:10.355 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:10.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:10.457 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:10.483 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:10.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:10.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:10.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:10.645 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:10.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:10.677 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:48:10.698 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:48:10.746 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:10.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:10.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:10.824 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:10 smithi029 ceph-mon[41500]: 3.b scrub starts 2024-03-24T12:48:10.824 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:10 smithi029 ceph-mon[41500]: 3.b scrub ok 2024-03-24T12:48:10.824 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:10 smithi029 ceph-mon[41500]: 6.1a scrub starts 2024-03-24T12:48:10.824 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:10 smithi029 ceph-mon[41500]: 6.1a scrub ok 2024-03-24T12:48:10.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:10 smithi029 ceph-mon[29919]: 3.b scrub starts 2024-03-24T12:48:10.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:10 smithi029 ceph-mon[29919]: 3.b scrub ok 2024-03-24T12:48:10.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:10 smithi029 ceph-mon[29919]: 6.1a scrub starts 2024-03-24T12:48:10.824 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:10 smithi029 ceph-mon[29919]: 6.1a scrub ok 2024-03-24T12:48:10.875 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:10.903 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:10.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:10.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:11.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:11.060 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:11.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:11.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:10 smithi190 ceph-mon[33748]: 3.b scrub starts 2024-03-24T12:48:11.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:10 smithi190 ceph-mon[33748]: 3.b scrub ok 2024-03-24T12:48:11.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:10 smithi190 ceph-mon[33748]: 6.1a scrub starts 2024-03-24T12:48:11.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:10 smithi190 ceph-mon[33748]: 6.1a scrub ok 2024-03-24T12:48:11.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:11.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:11.202 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:11.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:11.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:11.327 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:11.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:11.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:11.429 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:11.456 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:11.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:11.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:11.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:11.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:11.694 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:11.700 INFO:teuthology.orchestra.run.smithi029.stderr:set 6.c pg_upmap_items mapping to [6->5,0->2,3->1] 2024-03-24T12:48:11.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:11.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:11.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:11.822 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:11.864 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:11.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:48:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:48:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[41500]: 3.1c scrub starts 2024-03-24T12:48:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[41500]: 3.1c scrub ok 2024-03-24T12:48:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[41500]: pgmap v977: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 945 op/s 2024-03-24T12:48:11.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2883967668' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.c", "id": [6, 5, 0, 2, 3, 1]}]: dispatch 2024-03-24T12:48:11.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:48:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:48:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[29919]: 3.1c scrub starts 2024-03-24T12:48:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[29919]: 3.1c scrub ok 2024-03-24T12:48:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[29919]: pgmap v977: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 945 op/s 2024-03-24T12:48:11.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:11 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2883967668' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.c", "id": [6, 5, 0, 2, 3, 1]}]: dispatch 2024-03-24T12:48:11.966 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:11.992 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:11.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:12.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:12.120 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:12.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:11 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:48:12.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:11 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:48:12.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:11 smithi190 ceph-mon[33748]: 3.1c scrub starts 2024-03-24T12:48:12.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:11 smithi190 ceph-mon[33748]: 3.1c scrub ok 2024-03-24T12:48:12.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:11 smithi190 ceph-mon[33748]: pgmap v977: 158 pgs: 158 active+clean; 3.8 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 945 op/s 2024-03-24T12:48:12.120 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:11 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2883967668' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.c", "id": [6, 5, 0, 2, 3, 1]}]: dispatch 2024-03-24T12:48:12.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:12.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:12.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:12.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:48:12.260 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:12.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:12.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:12.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:12.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:12.427 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:12.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:12.528 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:12.554 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:12.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:12.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:12.655 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:12.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:12.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:12.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:12.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:12.842 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:12.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:12.943 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:12.975 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:12.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:12.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:13.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:13.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:48:13.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:48:13.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[29919]: 4.0 scrub starts 2024-03-24T12:48:13.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[29919]: 4.0 scrub ok 2024-03-24T12:48:13.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[29919]: 1.4 scrub starts 2024-03-24T12:48:13.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[29919]: 1.4 scrub ok 2024-03-24T12:48:13.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:48:13.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:48:13.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2883967668' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.c", "id": [6, 5, 0, 2, 3, 1]}]': finished 2024-03-24T12:48:13.093 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[29919]: osdmap e416: 8 total, 7 up, 6 in 2024-03-24T12:48:13.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:48:13.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:48:13.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[41500]: 4.0 scrub starts 2024-03-24T12:48:13.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[41500]: 4.0 scrub ok 2024-03-24T12:48:13.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[41500]: 1.4 scrub starts 2024-03-24T12:48:13.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[41500]: 1.4 scrub ok 2024-03-24T12:48:13.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:48:13.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:48:13.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2883967668' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.c", "id": [6, 5, 0, 2, 3, 1]}]': finished 2024-03-24T12:48:13.094 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:12 smithi029 ceph-mon[41500]: osdmap e416: 8 total, 7 up, 6 in 2024-03-24T12:48:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:13.123 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:13.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:12 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:48:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:12 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:48:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:12 smithi190 ceph-mon[33748]: 4.0 scrub starts 2024-03-24T12:48:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:12 smithi190 ceph-mon[33748]: 4.0 scrub ok 2024-03-24T12:48:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:12 smithi190 ceph-mon[33748]: 1.4 scrub starts 2024-03-24T12:48:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:12 smithi190 ceph-mon[33748]: 1.4 scrub ok 2024-03-24T12:48:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:12 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:48:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:12 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:48:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:12 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2883967668' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.c", "id": [6, 5, 0, 2, 3, 1]}]': finished 2024-03-24T12:48:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:12 smithi190 ceph-mon[33748]: osdmap e416: 8 total, 7 up, 6 in 2024-03-24T12:48:13.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:13.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:13.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:13.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:13.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:13.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:13.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:13.509 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:13.535 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:13.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:13.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:13.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:13.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:13.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:13.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:13.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:13.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:13.816 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:13.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:13.917 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:13.944 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:13.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:13.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:14.046 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:14.073 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:14.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:14.126 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[29919]: 3.1b scrub starts 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[29919]: 3.1b scrub ok 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[29919]: 5.14 scrub starts 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[29919]: 5.14 scrub ok 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[29919]: osdmap e417: 8 total, 7 up, 6 in 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[29919]: pgmap v980: 158 pgs: 158 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 978 op/s 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[41500]: 3.1b scrub starts 2024-03-24T12:48:14.127 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[41500]: 3.1b scrub ok 2024-03-24T12:48:14.128 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[41500]: 5.14 scrub starts 2024-03-24T12:48:14.128 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[41500]: 5.14 scrub ok 2024-03-24T12:48:14.128 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[41500]: osdmap e417: 8 total, 7 up, 6 in 2024-03-24T12:48:14.128 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:13 smithi029 ceph-mon[41500]: pgmap v980: 158 pgs: 158 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 978 op/s 2024-03-24T12:48:14.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:13 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:48:14.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:13 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:48:14.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:13 smithi190 ceph-mon[33748]: 3.1b scrub starts 2024-03-24T12:48:14.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:13 smithi190 ceph-mon[33748]: 3.1b scrub ok 2024-03-24T12:48:14.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:13 smithi190 ceph-mon[33748]: 5.14 scrub starts 2024-03-24T12:48:14.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:13 smithi190 ceph-mon[33748]: 5.14 scrub ok 2024-03-24T12:48:14.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:13 smithi190 ceph-mon[33748]: osdmap e417: 8 total, 7 up, 6 in 2024-03-24T12:48:14.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:13 smithi190 ceph-mon[33748]: pgmap v980: 158 pgs: 158 active+clean; 3.7 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 978 op/s 2024-03-24T12:48:14.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:14.214 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:14.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:14.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:14.316 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:14.343 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:14.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:14.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:14.445 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:14.475 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:14.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:14.575 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:14.648 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:14.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:14.748 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:14 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:48:14.748 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:14 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:48:14.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:14 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:48:14.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:14 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:48:14.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:14.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:14.793 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:14.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:14.895 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:14.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:14.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:15.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:15.071 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:15.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:15.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:14 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:48:15.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:14 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:48:15.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:15.210 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:15.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:15.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:15.311 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:15.337 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:15.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:15.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:15.381 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:48:15.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:15.489 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:15.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:15.591 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:15.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:15.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:15.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:15.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:15.746 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:15.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:15.847 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:15.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:15.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:15.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:15.974 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:16.001 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:16.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:16.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:16.103 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:15 smithi190 ceph-mon[33748]: 5.0 deep-scrub starts 2024-03-24T12:48:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:15 smithi190 ceph-mon[33748]: 5.0 deep-scrub ok 2024-03-24T12:48:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:15 smithi190 ceph-mon[33748]: 11.0 deep-scrub starts 2024-03-24T12:48:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:15 smithi190 ceph-mon[33748]: pgmap v981: 158 pgs: 158 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:48:16.142 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:16.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:16.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:16.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:15 smithi029 ceph-mon[29919]: 5.0 deep-scrub starts 2024-03-24T12:48:16.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:15 smithi029 ceph-mon[29919]: 5.0 deep-scrub ok 2024-03-24T12:48:16.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:15 smithi029 ceph-mon[29919]: 11.0 deep-scrub starts 2024-03-24T12:48:16.155 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:15 smithi029 ceph-mon[29919]: pgmap v981: 158 pgs: 158 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:48:16.155 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:15 smithi029 ceph-mon[41500]: 5.0 deep-scrub starts 2024-03-24T12:48:16.155 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:15 smithi029 ceph-mon[41500]: 5.0 deep-scrub ok 2024-03-24T12:48:16.156 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:15 smithi029 ceph-mon[41500]: 11.0 deep-scrub starts 2024-03-24T12:48:16.156 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:15 smithi029 ceph-mon[41500]: pgmap v981: 158 pgs: 158 active+clean; 3.6 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-24T12:48:16.245 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:16.271 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:16.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:16.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:16.373 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:16.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:16.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:16.499 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:48:16.499 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:48:16.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:16.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:16.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:16.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:16.641 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:16.668 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:16.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:16.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:16.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:16.807 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:16.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:16.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:16.934 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:16.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:17.036 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:17.059 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:17.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:17.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:17.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:16 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:48:17.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:16 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:48:17.161 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:17.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:16 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:48:17.173 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:16 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:48:17.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:16 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:48:17.174 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:16 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:48:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:17.211 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:17.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:17.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:17.348 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:17.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:17.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:17.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:17.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:17.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:17.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:17.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:17.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:17.605 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:17.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:17.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:17.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:17.743 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:17.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:17.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:17.870 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:17.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:17.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:17.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:48:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:48:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[29919]: 1.8 scrub starts 2024-03-24T12:48:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[29919]: 1.8 scrub ok 2024-03-24T12:48:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:48:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:48:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[29919]: 1.7 deep-scrub starts 2024-03-24T12:48:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[29919]: 1.7 deep-scrub ok 2024-03-24T12:48:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[29919]: pgmap v982: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 812 op/s 2024-03-24T12:48:17.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[41500]: 1.8 scrub starts 2024-03-24T12:48:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[41500]: 1.8 scrub ok 2024-03-24T12:48:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:48:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:48:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[41500]: 1.7 deep-scrub starts 2024-03-24T12:48:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[41500]: 1.7 deep-scrub ok 2024-03-24T12:48:17.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:17 smithi029 ceph-mon[41500]: pgmap v982: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 812 op/s 2024-03-24T12:48:17.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:18.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:18.009 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:18.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:18.110 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:17 smithi190 ceph-mon[33748]: 1.8 scrub starts 2024-03-24T12:48:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:17 smithi190 ceph-mon[33748]: 1.8 scrub ok 2024-03-24T12:48:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:17 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:48:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:17 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:48:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:17 smithi190 ceph-mon[33748]: 1.7 deep-scrub starts 2024-03-24T12:48:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:17 smithi190 ceph-mon[33748]: 1.7 deep-scrub ok 2024-03-24T12:48:18.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:17 smithi190 ceph-mon[33748]: pgmap v982: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 812 op/s 2024-03-24T12:48:18.134 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:18.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:18.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:18.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:18.264 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:18.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:18.365 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:18.401 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:18.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:18.503 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:18.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:18.527 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:18.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:18.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0 2024-03-24T12:48:18.670 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:18.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.0' 2024-03-24T12:48:18.749 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:48:18 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:48:18.569+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:48:18.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:18.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:18.807 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:18.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:18.819 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 7] out_osds: [5, 6] dead_osds: [5] live_osds: [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:48:18.819 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-24T12:48:18.819 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 6, 7] 2024-03-24T12:48:18.819 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T12:48:18.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.0 2024-03-24T12:48:18.909 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:18.935 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:18.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:18.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:19.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:19.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:19.065 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:19.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:19.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:48:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:48:19.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:18 smithi190 ceph-mon[33748]: 6.19 deep-scrub starts 2024-03-24T12:48:19.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:18 smithi190 ceph-mon[33748]: 6.19 deep-scrub ok 2024-03-24T12:48:19.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:19.166 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:19.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:18 smithi029 ceph-mon[41500]: 6.19 deep-scrub starts 2024-03-24T12:48:19.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:18 smithi029 ceph-mon[41500]: 6.19 deep-scrub ok 2024-03-24T12:48:19.171 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:19.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:18 smithi029 ceph-mon[29919]: 6.19 deep-scrub starts 2024-03-24T12:48:19.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:18 smithi029 ceph-mon[29919]: 6.19 deep-scrub ok 2024-03-24T12:48:19.171 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:19.172 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:18 smithi029 systemd[1]: Stopping Ceph osd.0 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:48:19.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:19.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:19.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:19.349 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:19.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:19.451 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:19.480 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:19.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:19.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:19.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:19.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:19.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:19.719 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:19.743 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:19.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:19.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:19.845 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:19.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:19.872 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:19.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:19.973 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:19.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:19.998 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:19.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:20.071 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:20.071 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:19 smithi029 ceph-mon[41500]: 1.3 deep-scrub starts 2024-03-24T12:48:20.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:19 smithi029 ceph-mon[41500]: 1.3 deep-scrub ok 2024-03-24T12:48:20.072 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:19 smithi029 ceph-mon[41500]: pgmap v983: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:48:20.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:20.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:19 smithi029 ceph-mon[29919]: 1.3 deep-scrub starts 2024-03-24T12:48:20.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:19 smithi029 ceph-mon[29919]: 1.3 deep-scrub ok 2024-03-24T12:48:20.072 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:19 smithi029 ceph-mon[29919]: pgmap v983: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:48:20.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:20.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:20.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:19 smithi190 ceph-mon[33748]: 1.3 deep-scrub starts 2024-03-24T12:48:20.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:19 smithi190 ceph-mon[33748]: 1.3 deep-scrub ok 2024-03-24T12:48:20.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:19 smithi190 ceph-mon[33748]: pgmap v983: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.3 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-03-24T12:48:20.136 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:20.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:20.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:20.275 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:20.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:20.377 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:20.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:20.401 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:20.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:20.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:20.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:20.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:20.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:20.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:20.666 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:20.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:20.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:20.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:20.809 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:20.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:20.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:20.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:20.950 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:20.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:21.051 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:21.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:21.075 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:21.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:20 smithi190 ceph-mon[33748]: 3.5 scrub starts 2024-03-24T12:48:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:20 smithi190 ceph-mon[33748]: 3.5 scrub ok 2024-03-24T12:48:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:20 smithi190 ceph-mon[33748]: 6.d scrub starts 2024-03-24T12:48:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:20 smithi190 ceph-mon[33748]: 3.1a scrub starts 2024-03-24T12:48:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:20 smithi190 ceph-mon[33748]: 3.1a scrub ok 2024-03-24T12:48:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:20 smithi190 ceph-mon[33748]: 6.d scrub ok 2024-03-24T12:48:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:20 smithi190 ceph-mon[33748]: 5.9 deep-scrub starts 2024-03-24T12:48:21.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:20 smithi190 ceph-mon[33748]: 5.9 deep-scrub ok 2024-03-24T12:48:21.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[41500]: 3.5 scrub starts 2024-03-24T12:48:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[41500]: 3.5 scrub ok 2024-03-24T12:48:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[41500]: 6.d scrub starts 2024-03-24T12:48:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[41500]: 3.1a scrub starts 2024-03-24T12:48:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[41500]: 3.1a scrub ok 2024-03-24T12:48:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[41500]: 6.d scrub ok 2024-03-24T12:48:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[41500]: 5.9 deep-scrub starts 2024-03-24T12:48:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[41500]: 5.9 deep-scrub ok 2024-03-24T12:48:21.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[29919]: 3.5 scrub starts 2024-03-24T12:48:21.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[29919]: 3.5 scrub ok 2024-03-24T12:48:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[29919]: 6.d scrub starts 2024-03-24T12:48:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[29919]: 3.1a scrub starts 2024-03-24T12:48:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[29919]: 3.1a scrub ok 2024-03-24T12:48:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[29919]: 6.d scrub ok 2024-03-24T12:48:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[29919]: 5.9 deep-scrub starts 2024-03-24T12:48:21.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:20 smithi029 ceph-mon[29919]: 5.9 deep-scrub ok 2024-03-24T12:48:21.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:21.204 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:21.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:21.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:21.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:21.330 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:21.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:21.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:21.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:21.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:21.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:21.572 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:21.599 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:21.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:21.701 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:21.729 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:21.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:21.830 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:21.858 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:21.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:21.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:21.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:21.998 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:21.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:21.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:22.100 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:22.127 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:22.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:22.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:21 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:48:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:21 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:48:22.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:21 smithi190 ceph-mon[33748]: pgmap v984: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T12:48:22.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:21 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:48:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:21 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:48:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:21 smithi029 ceph-mon[41500]: pgmap v984: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T12:48:22.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:21 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:48:22.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:21 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:48:22.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:21 smithi029 ceph-mon[29919]: pgmap v984: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.2 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-24T12:48:22.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:22.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:22.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:22.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:22.394 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:22.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:22.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:22.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:22.555 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:22.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:22.656 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:22.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:22.696 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:22.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:22.797 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:22.823 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:22.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:22.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:22.976 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:22.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:23.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T12:48:23.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:23.114 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:23.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:23.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:23.253 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:23.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:23.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:23.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:23.403 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:23.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:23.495 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:48:23.504 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:23.553 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:23.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:23.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:23.655 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:23.682 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:23.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:23.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:23.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:23.825 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:23.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:23.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:23.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:24.065 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:24.093 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:24.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:24.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:24.195 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:24.233 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:24.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:24.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:23 smithi190 ceph-mon[33748]: 5.12 scrub starts 2024-03-24T12:48:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:23 smithi190 ceph-mon[33748]: 5.12 scrub ok 2024-03-24T12:48:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:23 smithi190 ceph-mon[33748]: pgmap v985: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:48:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:23 smithi190 ceph-mon[33748]: 3.1e scrub starts 2024-03-24T12:48:24.265 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:23 smithi190 ceph-mon[33748]: 3.1e scrub ok 2024-03-24T12:48:24.282 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:23 smithi029 ceph-mon[29919]: 5.12 scrub starts 2024-03-24T12:48:24.282 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:23 smithi029 ceph-mon[29919]: 5.12 scrub ok 2024-03-24T12:48:24.282 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:23 smithi029 ceph-mon[29919]: pgmap v985: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:48:24.282 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:23 smithi029 ceph-mon[29919]: 3.1e scrub starts 2024-03-24T12:48:24.282 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:23 smithi029 ceph-mon[29919]: 3.1e scrub ok 2024-03-24T12:48:24.282 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:23 smithi029 ceph-mon[41500]: 5.12 scrub starts 2024-03-24T12:48:24.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:23 smithi029 ceph-mon[41500]: 5.12 scrub ok 2024-03-24T12:48:24.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:23 smithi029 ceph-mon[41500]: pgmap v985: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.16k op/s 2024-03-24T12:48:24.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:23 smithi029 ceph-mon[41500]: 3.1e scrub starts 2024-03-24T12:48:24.283 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:23 smithi029 ceph-mon[41500]: 3.1e scrub ok 2024-03-24T12:48:24.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:24.374 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:24.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:24.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:24.515 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:24.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:24.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:24.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:24.642 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:24.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:24.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:24 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[45963]: 2024-03-24T12:48:24.282+0000 7f5d573ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:48:24.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:24 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[45963]: 2024-03-24T12:48:24.282+0000 7f5d573ce700 -1 osd.0 417 *** Got signal Terminated *** 2024-03-24T12:48:24.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:24 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[45963]: 2024-03-24T12:48:24.282+0000 7f5d573ce700 -1 osd.0 417 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:48:24.743 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:24.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:24.768 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:24.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:24.841 INFO:teuthology.orchestra.run.smithi029.stdout:osd.0: {} 2024-03-24T12:48:24.842 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:48:24.842 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:48:24.857 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:48:24.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:24.902 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:48:24.907 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:24.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:24.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:24.943 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:48:24.986 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:48:25.000 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:48:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-24T12:48:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:48:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:48:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:48:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:48:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:48:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:48:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:48:25.009 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:25.035 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:25.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:25.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:24 smithi190 ceph-mon[33748]: osd.0 marked itself down and dead 2024-03-24T12:48:25.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:24 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1974570076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:48:25.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:24 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1031891496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:48:25.136 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:25.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:25.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:25.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:25.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:25.288 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:25.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:25.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:25.390 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:25.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:25.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:25.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:25.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:24 smithi029 ceph-mon[29919]: osd.0 marked itself down and dead 2024-03-24T12:48:25.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:24 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1974570076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:48:25.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:24 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1031891496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:48:25.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:24 smithi029 ceph-mon[41500]: osd.0 marked itself down and dead 2024-03-24T12:48:25.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:24 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1974570076' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:48:25.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:24 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1031891496' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:48:25.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:25.542 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:25.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:25.643 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:25.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:25.669 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:25.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:25.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:25.795 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:25.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:25.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:25.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:25.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:25.923 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:25.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:26.024 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:26.056 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:26.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:26.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:26.119 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:48:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:48:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-03-24T12:48:26.119 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:48:26.159 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:26.184 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:26.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:26.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:26.285 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:26.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:26.315 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:26.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:25 smithi190 ceph-mon[33748]: pgmap v986: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 965 op/s 2024-03-24T12:48:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:25 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:48:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:25 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:48:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:25 smithi190 ceph-mon[33748]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T12:48:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:25 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1974570076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:48:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:25 smithi190 ceph-mon[33748]: osdmap e418: 8 total, 6 up, 6 in 2024-03-24T12:48:26.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[29919]: pgmap v986: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 965 op/s 2024-03-24T12:48:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:48:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:48:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[29919]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T12:48:26.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1974570076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:48:26.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[29919]: osdmap e418: 8 total, 6 up, 6 in 2024-03-24T12:48:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[41500]: pgmap v986: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 3.0 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 965 op/s 2024-03-24T12:48:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:48:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:48:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[41500]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-24T12:48:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1974570076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:48:26.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:25 smithi029 ceph-mon[41500]: osdmap e418: 8 total, 6 up, 6 in 2024-03-24T12:48:26.454 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:26.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:26.556 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:26.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:26.693 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:26.724 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:26.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:26.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:26.867 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:26.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:26.964 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T12:48:26.968 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:27.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:27.037 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:27.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:27.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:27.168 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:27.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:27.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:27.270 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:27.311 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:27.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:27.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:27.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:26 smithi029 ceph-mon[29919]: osdmap e419: 8 total, 6 up, 6 in 2024-03-24T12:48:27.359 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:26 smithi029 ceph-mon[41500]: osdmap e419: 8 total, 6 up, 6 in 2024-03-24T12:48:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:26 smithi190 ceph-mon[33748]: osdmap e419: 8 total, 6 up, 6 in 2024-03-24T12:48:27.409 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:48:27.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:27.451 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:27.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:27.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:27.588 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:27.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:27.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:27.667 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:48:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:48:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:48:27.667 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:27 smithi029 podman[207562]: 2024-03-24 12:48:27.360019956 +0000 UTC m=+3.832718785 container died 63340869f18ebc14b0a3ab44c15636c0a1694b12cbbac90c05bf11ef231546b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:48:27.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:27.726 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:27.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:27.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:27.828 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:27.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:27.862 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:27.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:27.963 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:27.994 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:27.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:27.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:28.096 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:28.124 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:28.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:28.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:28.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:27 smithi190 ceph-mon[33748]: pgmap v989: 158 pgs: 1 active+clean+scrubbing, 41 peering, 21 stale+active+clean, 1 active+clean+scrubbing+deep, 94 active+clean; 2.9 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 3.5 KiB/s rd, 27 KiB/s wr, 710 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-24T12:48:28.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:27 smithi190 ceph-mon[33748]: Health check failed: Reduced data availability: 9 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-03-24T12:48:28.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:28.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:28.364 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:28.395 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:28.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:28.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:27 smithi029 ceph-mon[29919]: pgmap v989: 158 pgs: 1 active+clean+scrubbing, 41 peering, 21 stale+active+clean, 1 active+clean+scrubbing+deep, 94 active+clean; 2.9 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 3.5 KiB/s rd, 27 KiB/s wr, 710 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-24T12:48:28.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:27 smithi029 ceph-mon[29919]: Health check failed: Reduced data availability: 9 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-03-24T12:48:28.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:27 smithi029 ceph-mon[41500]: pgmap v989: 158 pgs: 1 active+clean+scrubbing, 41 peering, 21 stale+active+clean, 1 active+clean+scrubbing+deep, 94 active+clean; 2.9 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 3.5 KiB/s rd, 27 KiB/s wr, 710 op/s; 20 KiB/s, 0 objects/s recovering 2024-03-24T12:48:28.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:27 smithi029 ceph-mon[41500]: Health check failed: Reduced data availability: 9 pgs inactive, 14 pgs peering (PG_AVAILABILITY) 2024-03-24T12:48:28.496 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:28.520 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:28.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:28.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:28.622 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:28.665 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:28.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:28.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:28.768 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:28.796 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:28.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:28.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:28.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:28.937 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:28.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:29.034 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:48:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:48:29.037 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:29.060 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:29.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:29.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:29.162 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:29.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:29.195 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:29.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:29.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:29.301 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:29 smithi029 ceph-mon[29919]: Health check failed: Degraded data redundancy: 11214/137442 objects degraded (8.159%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:29.301 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:29 smithi029 ceph-mon[41500]: Health check failed: Degraded data redundancy: 11214/137442 objects degraded (8.159%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:29.326 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:29.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:29.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:29.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:29 smithi190 ceph-mon[33748]: Health check failed: Degraded data redundancy: 11214/137442 objects degraded (8.159%), 19 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:29.428 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:29.467 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:29.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:29.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:29.569 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:29.596 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:29.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:29.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:29 smithi029 podman[207562]: 2024-03-24 12:48:29.406759777 +0000 UTC m=+5.879458611 container cleanup 63340869f18ebc14b0a3ab44c15636c0a1694b12cbbac90c05bf11ef231546b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:48:29.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:29 smithi029 bash[207562]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0 2024-03-24T12:48:29.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:29.737 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:29.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:29.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:29.871 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:29.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:29.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:29.973 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:30.007 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:30.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:30.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:30.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:30.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:30.139 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:30.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:30.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:30.278 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:30.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:30.380 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:30 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:30.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:30 smithi190 ceph-mon[33748]: pgmap v990: 158 pgs: 28 active+undersized, 1 active+clean+scrubbing, 41 peering, 3 stale+active+clean, 19 active+undersized+degraded, 66 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 10 KiB/s rd, 82 KiB/s wr, 912 op/s; 11214/137442 objects degraded (8.159%); 72 KiB/s, 0 objects/s recovering 2024-03-24T12:48:30.407 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:30.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:30.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:30.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:30 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:30.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:30 smithi029 ceph-mon[29919]: pgmap v990: 158 pgs: 28 active+undersized, 1 active+clean+scrubbing, 41 peering, 3 stale+active+clean, 19 active+undersized+degraded, 66 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 10 KiB/s rd, 82 KiB/s wr, 912 op/s; 11214/137442 objects degraded (8.159%); 72 KiB/s, 0 objects/s recovering 2024-03-24T12:48:30.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:30 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:30.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:30 smithi029 ceph-mon[41500]: pgmap v990: 158 pgs: 28 active+undersized, 1 active+clean+scrubbing, 41 peering, 3 stale+active+clean, 19 active+undersized+degraded, 66 active+clean; 2.8 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 10 KiB/s rd, 82 KiB/s wr, 912 op/s; 11214/137442 objects degraded (8.159%); 72 KiB/s, 0 objects/s recovering 2024-03-24T12:48:30.510 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:48:30.545 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:48:30.547 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:30.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:30.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:30.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:30.688 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:30.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:30.790 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:30.830 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:30.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:30.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:30.932 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:30.962 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:30.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:31.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:31.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:31.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:31.205 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:31.234 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:31.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:31.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:31.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:31.379 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:31.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:31.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:31 smithi029 ceph-mon[29919]: pgmap v991: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 9.4 KiB/s rd, 82 KiB/s wr, 1.05k op/s; 24062/129336 objects degraded (18.604%); 72 KiB/s, 0 objects/s recovering 2024-03-24T12:48:31.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:31 smithi029 ceph-mon[41500]: pgmap v991: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 9.4 KiB/s rd, 82 KiB/s wr, 1.05k op/s; 24062/129336 objects degraded (18.604%); 72 KiB/s, 0 objects/s recovering 2024-03-24T12:48:31.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:31.516 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:31.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:31.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:31.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:31 smithi190 ceph-mon[33748]: pgmap v991: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 9.4 KiB/s rd, 82 KiB/s wr, 1.05k op/s; 24062/129336 objects degraded (18.604%); 72 KiB/s, 0 objects/s recovering 2024-03-24T12:48:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:31.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:31.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:31.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:31.793 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:31.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:31.895 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:31.920 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:31.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:32.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:32.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:32.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:32.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:32.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:32.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:31 smithi029 podman[207814]: 2024-03-24 12:48:31.921281886 +0000 UTC m=+6.207982267 container remove 63340869f18ebc14b0a3ab44c15636c0a1694b12cbbac90c05bf11ef231546b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T12:48:32.177 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:32.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:32.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:32.279 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:32.308 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:32.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:32.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:32.410 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:32.433 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:32.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:32.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:32.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:48:32.573 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:32.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:32.606 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:32 smithi029 ceph-mon[29919]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 14 pgs peering) 2024-03-24T12:48:32.607 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:32 smithi029 ceph-mon[41500]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 14 pgs peering) 2024-03-24T12:48:32.607 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:32 smithi029 podman[208353]: 2024-03-24 12:48:32.301499047 +0000 UTC m=+0.039002753 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:48:32.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:32 smithi190 ceph-mon[33748]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 9 pgs inactive, 14 pgs peering) 2024-03-24T12:48:32.678 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:32.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:32.705 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:32.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:32.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:32.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:32.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:32.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:32.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:33.074 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:33.100 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:33.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:33.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:33.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-24T12:48:33.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:33.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:33.247 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:33.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:33.348 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:33.375 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:33.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:33.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:33.477 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:33.522 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:33.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:33.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:33.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:33 smithi029 ceph-mon[29919]: pgmap v992: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 9.4 KiB/s rd, 82 KiB/s wr, 950 op/s; 23649/126753 objects degraded (18.658%); 72 KiB/s, 0 objects/s recovering 2024-03-24T12:48:33.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:33 smithi029 ceph-mon[41500]: pgmap v992: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 9.4 KiB/s rd, 82 KiB/s wr, 950 op/s; 23649/126753 objects degraded (18.658%); 72 KiB/s, 0 objects/s recovering 2024-03-24T12:48:33.624 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:33 smithi190 ceph-mon[33748]: pgmap v992: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 9.4 KiB/s rd, 82 KiB/s wr, 950 op/s; 23649/126753 objects degraded (18.658%); 72 KiB/s, 0 objects/s recovering 2024-03-24T12:48:33.665 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:33.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:33.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:33.767 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:33.797 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:33.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:33.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:33.903 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:48:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:48:33.606+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:48:33.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:33.972 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:33.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:34.073 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:34.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:34.097 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:34.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:34.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:33 smithi029 podman[208353]: 2024-03-24 12:48:33.908090447 +0000 UTC m=+1.645594147 container create 0d4d3bfe3d8845cb5127f763769963befdf0ad2a1c95222e1a894a276cc45f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T12:48:34.199 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:34.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:34.227 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:34.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:34.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:34.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:34.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:34.476 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:34.502 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:34.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:34.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:34.567 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:48:34.567 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:48:34.567 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:34.604 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:34.627 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:34.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:34.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:34.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:48:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:48:34.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:34.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:48:34.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:48:34.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:34.730 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:34.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:34.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:34.792 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:48:34.792 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:48:34.792 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-03-24T12:48:34.792 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-03-24T12:48:34.792 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:48:34.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:34.908 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:34.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:34.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:35.010 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:35.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:35.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:35.174 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:35.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:35.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:35.306 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:35.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:35.409 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:35.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:35.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:35.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:35.561 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:35.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:35.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:35 smithi190 ceph-mon[33748]: pgmap v993: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 8.1 KiB/s rd, 66 KiB/s wr, 1.07k op/s; 22345/117579 objects degraded (19.004%); 58 KiB/s, 0 objects/s recovering 2024-03-24T12:48:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:35 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 23649/126753 objects degraded (18.658%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:35.663 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:35.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:35 smithi029 ceph-mon[41500]: pgmap v993: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 8.1 KiB/s rd, 66 KiB/s wr, 1.07k op/s; 22345/117579 objects degraded (19.004%); 58 KiB/s, 0 objects/s recovering 2024-03-24T12:48:35.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:35 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 23649/126753 objects degraded (18.658%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:35.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:35 smithi029 ceph-mon[29919]: pgmap v993: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.4 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 8.1 KiB/s rd, 66 KiB/s wr, 1.07k op/s; 22345/117579 objects degraded (19.004%); 58 KiB/s, 0 objects/s recovering 2024-03-24T12:48:35.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:35 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 23649/126753 objects degraded (18.658%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:35.693 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:35.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:35.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:35.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:35.825 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:35.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:35.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:35.842 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T12:48:35.926 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:35.962 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:35.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:35.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:36.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:36.100 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:36.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:36.165 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:48:36.202 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:36.226 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:36.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:36.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:36.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:36.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:36.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:36.468 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:36.499 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:36.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:36.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:36 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:48:36.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:36 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:48:36.561 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:36 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:48:36.601 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:36.626 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:36.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:36.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:36 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:48:36.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:36 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:48:36.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:36 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:48:36.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:36 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:48:36.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:36 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:48:36.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:36 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:48:36.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:36.765 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:36.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:36.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:36.867 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:36.890 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:36.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:36.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:37.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:37.029 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:37.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:37.130 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:37.159 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:37.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:37.166 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:36 smithi029 podman[208353]: 2024-03-24 12:48:36.834012363 +0000 UTC m=+4.571516057 container init 0d4d3bfe3d8845cb5127f763769963befdf0ad2a1c95222e1a894a276cc45f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T12:48:37.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:36 smithi029 podman[208353]: 2024-03-24 12:48:36.840314356 +0000 UTC m=+4.577818053 container start 0d4d3bfe3d8845cb5127f763769963befdf0ad2a1c95222e1a894a276cc45f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-24T12:48:37.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:37.266 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:48:37.301 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:48:37.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:37.319 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:37.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:37.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:37.463 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:37.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:37.564 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:37.588 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:37.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:37.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:37.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:37 smithi029 ceph-mon[41500]: pgmap v994: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 5.2 KiB/s rd, 40 KiB/s wr, 1.11k op/s; 20329/109059 objects degraded (18.640%); 38 KiB/s, 0 objects/s recovering 2024-03-24T12:48:37.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:37 smithi029 ceph-mon[29919]: pgmap v994: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 5.2 KiB/s rd, 40 KiB/s wr, 1.11k op/s; 20329/109059 objects degraded (18.640%); 38 KiB/s, 0 objects/s recovering 2024-03-24T12:48:37.604 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:37 smithi029 podman[208353]: 2024-03-24 12:48:37.193285058 +0000 UTC m=+4.930788761 container attach 0d4d3bfe3d8845cb5127f763769963befdf0ad2a1c95222e1a894a276cc45f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-db0330b) 2024-03-24T12:48:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:37 smithi190 ceph-mon[33748]: pgmap v994: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 5.2 KiB/s rd, 40 KiB/s wr, 1.11k op/s; 20329/109059 objects degraded (18.640%); 38 KiB/s, 0 objects/s recovering 2024-03-24T12:48:37.690 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:37.722 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:37.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:37.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:37.824 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:37.861 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:37.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:37.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:48:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:48:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:48:37.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:38.003 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:38.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:38.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:38.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:38.146 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:38.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:38.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:38.281 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:38.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:38.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:38.383 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:38.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:38.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:38.515 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:48:38.515 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-03-24T12:48:38.515 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-03-24T12:48:38.515 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:48:38.515 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.515 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:185019 10.12 10:4f9c5757:::benchmark_data_smithi029_160664_object58045:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:48:38.515 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:56.368465+0000", 2024-03-24T12:48:38.515 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 282.14745569500002, 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1435284029999999, 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185019 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368465+0000", 2024-03-24T12:48:38.516 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368465+0000", 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368466+0000", 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8899999999999998e-07 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.517 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368470+0000", 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.318e-06 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368481+0000", 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1601e-05 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368490+0000", 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2439999999999993e-06 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.518 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368546+0000", 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6743000000000002e-05 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368556+0000", 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.431e-06 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.493465+0000", 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1249090230000001 2024-03-24T12:48:38.519 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for missing object", 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.493561+0000", 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6572999999999998e-05 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.511925+0000", 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.018363593000000001 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.520 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.511993+0000", 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7887999999999998e-05 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:185024 10.12 10:4b6f71a6:::benchmark_data_smithi029_160664_object58050:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:56.368507+0000", 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 282.14741315499998, 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1547719889999999, 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:48:38.521 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185024 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368507+0000", 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.522 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368507+0000", 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368508+0000", 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9200000000000001e-07 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368511+0000", 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9299999999999999e-06 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.523 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368518+0000", 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9859999999999997e-06 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368524+0000", 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7490000000000001e-06 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368599+0000", 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4171999999999994e-05 2024-03-24T12:48:38.524 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368606+0000", 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3749999999999997e-06 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.498894+0000", 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1302883260000001 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for missing object", 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.498975+0000", 2024-03-24T12:48:38.525 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0455000000000001e-05 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.523206+0000", 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.024231058 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.523279+0000", 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3345999999999999e-05 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.526 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:185064 10.12 10:496f5ede:::benchmark_data_smithi029_160664_object58090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:56.386289+0000", 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 282.129631448, 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1193564659999999, 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185064 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.527 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386289+0000", 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386289+0000", 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386289+0000", 2024-03-24T12:48:38.528 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4300000000000003e-07 2024-03-24T12:48:38.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.529 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386292+0000", 2024-03-24T12:48:38.529 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0060000000000001e-06 2024-03-24T12:48:38.529 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.529 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.529 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:38.530 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386300+0000", 2024-03-24T12:48:38.530 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3470000000000001e-06 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386308+0000", 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1499999999999999e-06 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.389925+0000", 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036172639999999998 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.531 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.389945+0000", 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9607999999999999e-05 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.499022+0000", 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.109077343 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.499107+0000", 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4967000000000002e-05 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.532 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.499315+0000", 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020838200000000001 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505173+0000", 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005857831 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505280+0000", 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107166 2024-03-24T12:48:38.533 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505451+0000", 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017029499999999999 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505513+0000", 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2137000000000005e-05 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.534 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505645+0000", 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013242700000000001 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:185065 10.12 10:4e7c706a:::benchmark_data_smithi029_160664_object58091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:56.387863+0000", 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 282.12805724100002, 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1179670509999999, 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.535 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185065 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387863+0000", 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387863+0000", 2024-03-24T12:48:38.536 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387864+0000", 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4000000000000001e-07 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387867+0000", 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3299999999999999e-06 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.537 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387875+0000", 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2539999999999992e-06 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387880+0000", 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2240000000000003e-06 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.538 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.390385+0000", 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025046170000000002 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.390393+0000", 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4590000000000001e-06 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.500335+0000", 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.109942121 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.539 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.500422+0000", 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7647999999999998e-05 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.500567+0000", 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014450800000000001 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505179+0000", 2024-03-24T12:48:38.540 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046126179999999998 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505398+0000", 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021868199999999999 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505719+0000", 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032113699999999998 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.541 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505759+0000", 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9745e-05 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505830+0000", 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1068000000000005e-05 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242879 10.a 10:59050bb8:::benchmark_data_smithi029_160664_object115905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-24T12:48:38.542 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.187072+0000", 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.32884867999999, 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.470001313, 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242879 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187072+0000", 2024-03-24T12:48:38.543 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187072+0000", 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187072+0000", 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4700000000000002e-07 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.544 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187074+0000", 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0209999999999998e-06 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187079+0000", 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3249999999999998e-06 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187084+0000", 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8119999999999998e-06 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.545 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.189749+0000", 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026646040000000001 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.189867+0000", 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011839400000000001 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.189987+0000", 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000119677 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.546 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.193807+0000", 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003819977 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.206807+0000", 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012999892000000001 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208165+0000", 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013583060000000001 2024-03-24T12:48:38.547 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208176+0000", 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0784e-05 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657012+0000", 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4488366720000001 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657073+0000", 2024-03-24T12:48:38.548 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0501999999999998e-05 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242894 10.a 10:56dffcb3:::benchmark_data_smithi029_160664_object115920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.198057+0000", 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.31786299300001, 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4592828680000001, 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.549 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242894 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198057+0000", 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198057+0000", 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.550 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198058+0000", 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2109999999999999e-06 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198063+0000", 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4710000000000001e-06 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198074+0000", 2024-03-24T12:48:38.551 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0937999999999999e-05 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198082+0000", 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9200000000000004e-06 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198127+0000", 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5082e-05 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.552 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198213+0000", 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5724999999999997e-05 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198291+0000", 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8238999999999994e-05 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.206673+0000", 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083826409999999997 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.553 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208220+0000", 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015461610000000001 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208230+0000", 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0702e-05 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657299+0000", 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4490690509999999 2024-03-24T12:48:38.554 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657340+0000", 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0726999999999998e-05 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242898 10.a 10:57906632:::benchmark_data_smithi029_160664_object115924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.198875+0000", 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.317044925, 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4585599380000001, 2024-03-24T12:48:38.555 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242898 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198875+0000", 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.556 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198875+0000", 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198876+0000", 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4099999999999999e-07 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198879+0000", 2024-03-24T12:48:38.557 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7219999999999999e-06 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198885+0000", 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0959999999999997e-06 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198890+0000", 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.972e-06 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.558 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.200493+0000", 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016029149999999999 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.200568+0000", 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5512000000000005e-05 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.200671+0000", 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010329500000000001 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.559 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.207043+0000", 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063716459999999999 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208325+0000", 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012816150000000001 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208330+0000", 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6489999999999996e-06 2024-03-24T12:48:38.560 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657383+0000", 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4490531529999999 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657435+0000", 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1822e-05 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.561 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242899 10.a 10:5fc4cbc7:::benchmark_data_smithi029_160664_object115925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.198897+0000", 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.31702305799999, 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4668848320000001, 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242899 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.562 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198897+0000", 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198897+0000", 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198897+0000", 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4600000000000001e-07 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.563 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198899+0000", 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4500000000000001e-06 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198903+0000", 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3449999999999999e-06 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198908+0000", 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5059999999999997e-06 2024-03-24T12:48:38.564 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.202670+0000", 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003762095 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.202763+0000", 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3602000000000003e-05 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.202917+0000", 2024-03-24T12:48:38.565 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015330399999999999 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.207114+0000", 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041972679999999997 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208358+0000", 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00124391 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.566 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208363+0000", 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.366e-06 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657465+0000", 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4491021100000001 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for missing object", 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657581+0000", 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115147 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.567 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665689+0000", 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0081088710000000001 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665782+0000", 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2612e-05 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242909 10.a 10:5a5b0db0:::benchmark_data_smithi029_160664_object115935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.219926+0000", 2024-03-24T12:48:38.568 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.29599420599999, 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.445532576, 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242909 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219926+0000", 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.569 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219926+0000", 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219927+0000", 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9100000000000002e-07 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219930+0000", 2024-03-24T12:48:38.570 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3969999999999998e-06 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219940+0000", 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9219999999999992e-06 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219947+0000", 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3199999999999996e-06 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.571 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.220010+0000", 2024-03-24T12:48:38.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3509000000000005e-05 2024-03-24T12:48:38.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.572 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.220024+0000", 2024-03-24T12:48:38.572 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4392e-05 2024-03-24T12:48:38.572 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.572 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.572 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.573 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657678+0000", 2024-03-24T12:48:38.573 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.437653662 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657789+0000", 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110549 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657898+0000", 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010885100000000001 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.574 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665298+0000", 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074007999999999999 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665309+0000", 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0481e-05 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665358+0000", 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9308999999999998e-05 2024-03-24T12:48:38.575 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665393+0000", 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4406999999999998e-05 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665459+0000", 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6086000000000004e-05 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.576 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242925 10.a 10:5108928b:::benchmark_data_smithi029_160664_object115951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.242462+0000", 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.27345794499999, 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.427779428, 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242925 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.577 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242462+0000", 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242462+0000", 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.578 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242463+0000", 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1500000000000003e-07 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242467+0000", 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8829999999999999e-06 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242491+0000", 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3615999999999999e-05 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.579 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242499+0000", 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3639999999999999e-06 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242562+0000", 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2490999999999995e-05 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242571+0000", 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3330000000000003e-06 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.580 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.660542+0000", 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4179714620000001 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.660650+0000", 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010791800000000001 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.660767+0000", 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000116977 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.581 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665315+0000", 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045479730000000003 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670080+0000", 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0047643410000000001 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670130+0000", 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0512e-05 2024-03-24T12:48:38.582 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670183+0000", 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2911000000000002e-05 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670242+0000", 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8731999999999999e-05 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.583 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242978 10.a 10:5cfa1afd:::benchmark_data_smithi029_160664_object116004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.279976+0000", 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.23594404799999, 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3931952830000001, 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242978 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.584 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.279976+0000", 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.279976+0000", 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.279977+0000", 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.207e-06 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.585 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.279982+0000", 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1070000000000004e-06 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.279995+0000", 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.218e-05 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280002+0000", 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5109999999999997e-06 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.586 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280070+0000", 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7609999999999998e-05 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280078+0000", 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.456e-06 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.663048+0000", 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.382970129 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.587 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.663178+0000", 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000129457 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.663314+0000", 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013637999999999999 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665652+0000", 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023377350000000001 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.588 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670287+0000", 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046346520000000004 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673035+0000", 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027482729999999999 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673086+0000", 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0701999999999997e-05 2024-03-24T12:48:38.589 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673171+0000", 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5884000000000006e-05 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242979 10.a 10:5130e7c7:::benchmark_data_smithi029_160664_object116005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.280020+0000", 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.23590065900001, 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.399110842, 2024-03-24T12:48:38.590 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242979 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280020+0000", 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.591 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280020+0000", 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280020+0000", 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2500000000000001e-07 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280022+0000", 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4040000000000002e-06 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.592 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280037+0000", 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4293e-05 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280042+0000", 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4659999999999998e-06 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280186+0000", 2024-03-24T12:48:38.593 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014369299999999999 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280194+0000", 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6139999999999992e-06 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665801+0000", 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3856060989999999 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.594 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665887+0000", 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6231999999999997e-05 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665965+0000", 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8607000000000001e-05 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670250+0000", 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042849450000000001 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.595 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673222+0000", 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029716479999999999 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.678966+0000", 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0057435560000000004 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.679019+0000", 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3699999999999997e-05 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.596 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.679130+0000", 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011116 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243008 10.a 10:5682a121:::benchmark_data_smithi029_160664_object116034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.298145+0000", 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.21777565100001, 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3832145630000001, 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.597 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243008 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298145+0000", 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298145+0000", 2024-03-24T12:48:38.598 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298145+0000", 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6100000000000001e-07 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298148+0000", 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0939999999999999e-06 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.599 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298155+0000", 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8700000000000003e-06 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298161+0000", 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2389999999999999e-06 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298221+0000", 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0004999999999999e-05 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.600 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298229+0000", 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9680000000000006e-06 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.667328+0000", 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3690987400000001 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.667444+0000", 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115541 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.601 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.667578+0000", 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013406099999999999 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670255+0000", 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026769789999999999 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673280+0000", 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030254119999999999 2024-03-24T12:48:38.602 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681242+0000", 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007962406 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681281+0000", 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8198999999999998e-05 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.603 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681359+0000", 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8487999999999998e-05 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243059 10.a 10:530f856a:::benchmark_data_smithi029_160664_object116085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.343440+0000", 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.17248011000001, 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3380952930000001, 2024-03-24T12:48:38.605 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243059 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343440+0000", 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.606 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343440+0000", 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343441+0000", 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9400000000000005e-07 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343444+0000", 2024-03-24T12:48:38.607 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2770000000000001e-06 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343453+0000", 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9900000000000003e-06 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343458+0000", 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.322e-06 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.608 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343515+0000", 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.643e-05 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343522+0000", 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7440000000000007e-06 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.668816+0000", 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.325293437 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.609 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.668926+0000", 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011024 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.669004+0000", 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7571999999999995e-05 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670366+0000", 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001362458 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.610 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681400+0000", 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011033612 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681444+0000", 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4146000000000001e-05 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681474+0000", 2024-03-24T12:48:38.611 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0300000000000001e-05 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681535+0000", 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1171000000000001e-05 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243073 10.a 10:5fcea2f5:::benchmark_data_smithi029_160664_object116099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.355624+0000", 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.160296335, 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.339988797, 2024-03-24T12:48:38.612 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243073 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355624+0000", 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.613 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355624+0000", 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355624+0000", 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1099999999999996e-07 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355628+0000", 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.591e-06 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.614 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355635+0000", 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2030000000000003e-06 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355642+0000", 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4239999999999998e-06 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.358531+0000", 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0028895750000000001 2024-03-24T12:48:38.615 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.358539+0000", 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9300000000000003e-06 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670391+0000", 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.311852249 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.616 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670477+0000", 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5298999999999998e-05 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670546+0000", 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9014e-05 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673233+0000", 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026870029999999999 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.617 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681583+0000", 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083506770000000008 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.695514+0000", 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013930471 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.618 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.695562+0000", 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8649000000000001e-05 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.695613+0000", 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0201000000000003e-05 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243083 10.a 10:59ed33c5:::benchmark_data_smithi029_160664_object116109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.619 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.365353+0000", 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.15056732400001, 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.333598788, 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243083 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.620 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365353+0000", 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365353+0000", 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365354+0000", 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.09e-07 2024-03-24T12:48:38.621 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365357+0000", 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6600000000000001e-06 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365367+0000", 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6840000000000008e-06 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365373+0000", 2024-03-24T12:48:38.622 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9000000000000003e-06 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365420+0000", 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7219999999999999e-05 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365427+0000", 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3060000000000003e-06 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.623 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.671684+0000", 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3062570259999999 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.671769+0000", 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5117000000000005e-05 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.671915+0000", 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000146324 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.624 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.674484+0000", 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00256939 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681628+0000", 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0071437640000000004 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.698875+0000", 2024-03-24T12:48:38.625 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017246678000000001 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.698916+0000", 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0667999999999998e-05 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.698952+0000", 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6142000000000001e-05 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.626 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243125 10.a 10:5a44a46d:::benchmark_data_smithi029_160664_object116151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.404780+0000", 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.11114019300001, 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.294317451, 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243125 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.627 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404780+0000", 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404780+0000", 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404781+0000", 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2500000000000005e-07 2024-03-24T12:48:38.628 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404785+0000", 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7790000000000002e-06 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404794+0000", 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0289999999999994e-06 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.629 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404799+0000", 2024-03-24T12:48:38.630 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3639999999999998e-06 2024-03-24T12:48:38.630 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.630 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.630 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.630 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404860+0000", 2024-03-24T12:48:38.631 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1155999999999995e-05 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404872+0000", 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1440000000000001e-05 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673313+0000", 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.268441698 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673391+0000", 2024-03-24T12:48:38.632 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7636e-05 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673466+0000", 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4825000000000003e-05 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681202+0000", 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0077358399999999999 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.692661+0000", 2024-03-24T12:48:38.633 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011459517000000001 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.698999+0000", 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063383349999999996 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699056+0000", 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.677e-05 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.634 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699097+0000", 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1337000000000001e-05 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243130 10.a 10:5eea474b:::benchmark_data_smithi029_160664_object116156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.408457+0000", 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.10746364400001, 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.290807756, 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.635 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243130 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408457+0000", 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408457+0000", 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.636 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408457+0000", 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6300000000000005e-07 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408460+0000", 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1099999999999999e-06 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408469+0000", 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2439999999999993e-06 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.637 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408474+0000", 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2970000000000003e-06 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408515+0000", 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0802e-05 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408521+0000", 2024-03-24T12:48:38.638 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3929999999999996e-06 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.674521+0000", 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2659998699999999 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.674637+0000", 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115696 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.639 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.674721+0000", 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4814000000000005e-05 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681211+0000", 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064895100000000004 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.692899+0000", 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011688298999999999 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.640 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699162+0000", 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006262821 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699218+0000", 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5619999999999999e-05 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699264+0000", 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6717e-05 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.641 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243131 10.a 10:50c3ecbe:::benchmark_data_smithi029_160664_object116157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.411006+0000", 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.104914627, 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2883988180000001, 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243131 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.642 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411006+0000", 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411006+0000", 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411006+0000", 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4499999999999997e-07 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.643 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411009+0000", 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0879999999999999e-06 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411016+0000", 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2429999999999999e-06 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411021+0000", 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8969999999999999e-06 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.644 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411059+0000", 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7589000000000002e-05 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411065+0000", 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.993e-06 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.675805+0000", 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.264740341 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.645 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.675885+0000", 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9600999999999999e-05 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.675962+0000", 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6913e-05 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681410+0000", 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054483989999999996 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.646 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.692933+0000", 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011522897000000001 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699321+0000", 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063874530000000004 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699368+0000", 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7788999999999999e-05 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.647 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699404+0000", 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6069999999999999e-05 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243147 10.a 10:500de219:::benchmark_data_smithi029_160664_object116173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.453385+0000", 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 223.06253513300001, 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.246141194, 2024-03-24T12:48:38.648 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243147 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453385+0000", 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:48:38.649 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453385+0000", 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453386+0000", 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9800000000000003e-07 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453388+0000", 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6029999999999999e-06 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.650 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453396+0000", 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4959999999999999e-06 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453401+0000", 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3229999999999997e-06 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453448+0000", 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7191000000000003e-05 2024-03-24T12:48:38.651 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453458+0000", 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4760000000000004e-06 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.677016+0000", 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.223558675 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.677140+0000", 2024-03-24T12:48:38.652 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000123387 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.677257+0000", 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000116666 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681418+0000", 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041616070000000003 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.653 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.692964+0000", 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011546201000000001 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699446+0000", 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064813780000000003 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699492+0000", 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6221999999999999e-05 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:48:38.654 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:48:38.655 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699526+0000", 2024-03-24T12:48:38.655 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4371e-05 2024-03-24T12:48:38.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.655 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.655 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:48:38.655 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:48:38.655 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:48:38.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:38.656 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:38.656 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:38.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:48:38 smithi029 podman[208353]: 2024-03-24 12:48:38.382732347 +0000 UTC m=+6.120236049 container died 0d4d3bfe3d8845cb5127f763769963befdf0ad2a1c95222e1a894a276cc45f5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T12:48:38.757 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:38.786 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:38.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:38.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:38.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:48:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:48:38.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:38.917 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:38.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:39.019 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:39.046 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:39.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:39.148 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:39.185 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:39.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:39.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:39.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T12:48:39.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:39.317 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:39.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:39.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:39.345 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:48:39.346 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T12:48:39.346 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-03-24T12:48:39.346 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi029.front.sepia.ceph.com 2024-03-24T12:48:39.346 INFO:tasks.thrashosds.thrasher:remote for exp osd 5 is ubuntu@smithi190.front.sepia.ceph.com 2024-03-24T12:48:39.347 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2024-03-24T12:48:39.420 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:39.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:39.451 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:39.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:39.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:39.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:39.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:39.704 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:48:39.730 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:39.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:39.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:39.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:48:39.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:39.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:39.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:39.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:39.964 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:39.964 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:39 smithi190 ceph-mon[33748]: pgmap v995: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 5.2 KiB/s rd, 37 KiB/s wr, 1.28k op/s; 18890/99456 objects degraded (18.993%); 35 KiB/s, 0 objects/s recovering 2024-03-24T12:48:39.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:40.015 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:40.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:40.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:40.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:40.153 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:40.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:40.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:40.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:39 smithi029 ceph-mon[29919]: pgmap v995: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 5.2 KiB/s rd, 37 KiB/s wr, 1.28k op/s; 18890/99456 objects degraded (18.993%); 35 KiB/s, 0 objects/s recovering 2024-03-24T12:48:40.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:40.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:39 smithi029 ceph-mon[41500]: pgmap v995: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 2.0 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 5.2 KiB/s rd, 37 KiB/s wr, 1.28k op/s; 18890/99456 objects degraded (18.993%); 35 KiB/s, 0 objects/s recovering 2024-03-24T12:48:40.255 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:40.284 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:40.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:40.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:40.386 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:40.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:40.442 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:40.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:40.543 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:40.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:40.573 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:40.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:40.675 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:40.704 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:40.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:40.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:40.842 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:40.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:40.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:40.995 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:40.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:41.091 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:40 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 18890/99456 objects degraded (18.993%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:41.097 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:41.128 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:41.128 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:41.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:40 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 18890/99456 objects degraded (18.993%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:41.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:40 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 18890/99456 objects degraded (18.993%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:41.228 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:41.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:41.260 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:41.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:41.361 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:41.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:41.393 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:41.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:41.493 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:41.530 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:41.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:41.633 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:41.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:41.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:41.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:41.802 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:48:41.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:41.808 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:41.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:41.847 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:48:41.871 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:48:41.885 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:48:41.908 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:48:41.909 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:41.926 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:48:41.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:41.949 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:41.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:41.968 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:48:42.007 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:48:42.008 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-24T12:48:42.008 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:48:42.008 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:48:42.008 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:48:42.008 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:48:42.008 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:48:42.008 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:48:42.008 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:48:42.050 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:42.082 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:42.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:42.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:41 smithi190 ceph-mon[33748]: pgmap v996: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 16891/91182 objects degraded (18.524%); 0 B/s, 0 objects/s recovering 2024-03-24T12:48:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:41 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1095748678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:48:42.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:41 smithi029 ceph-mon[41500]: pgmap v996: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 16891/91182 objects degraded (18.524%); 0 B/s, 0 objects/s recovering 2024-03-24T12:48:42.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:41 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1095748678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:48:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:41 smithi029 ceph-mon[29919]: pgmap v996: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.8 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.28k op/s; 16891/91182 objects degraded (18.524%); 0 B/s, 0 objects/s recovering 2024-03-24T12:48:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:41 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1095748678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:48:42.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:42.219 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:42.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:42.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:42.321 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:42.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:42.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:42.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:42.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:42.490 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:42.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:42.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:42.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:42.618 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:42.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:42.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:42.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:42.760 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:42.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:42.861 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:42.901 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:42.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:42.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:43.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:43.026 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:43.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:43.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:43.128 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:43.156 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:43.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:43.258 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:43.286 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:43.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:43.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:43.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:43.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:43.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:43.538 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:43.567 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:43.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:43.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:43.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:43.706 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:43.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:43.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:43.808 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:43.837 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:43.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:43.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:43.961 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:43.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:43.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:44.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:44.075 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:43 smithi190 ceph-mon[33748]: pgmap v997: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 16396/88356 objects degraded (18.557%) 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:8.a 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:8.9 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:8.4 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:8.0 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:6.d 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:6.c 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:6.4 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:6.1e 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:8.3 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:6.1c 2024-03-24T12:48:44.076 INFO:teuthology.orchestra.run.smithi190.stdout:6.7 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:6.11 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:4.1 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:6.1a 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:4.12 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:6.a 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:5.2 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:3.d 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:6.1f 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:3.4 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:4.15 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:3.1c 2024-03-24T12:48:44.077 INFO:teuthology.orchestra.run.smithi190.stdout:3.c 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:4.8 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:3.2 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:5.16 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:3.1f 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:5.7 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:3.5 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:8.2 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:3.10 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:3.1d 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:3.17 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:8.d 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:5.1b 2024-03-24T12:48:44.078 INFO:teuthology.orchestra.run.smithi190.stdout:2.2 2024-03-24T12:48:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:3.16 2024-03-24T12:48:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:4.1f 2024-03-24T12:48:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:6.1d 2024-03-24T12:48:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:5.d 2024-03-24T12:48:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:4.3 2024-03-24T12:48:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:4.7 2024-03-24T12:48:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:5.12 2024-03-24T12:48:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:5.15 2024-03-24T12:48:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:5.3 2024-03-24T12:48:44.079 INFO:teuthology.orchestra.run.smithi190.stdout:5.19 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:6.19 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:5.4 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:4.2 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:5.6 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:5.b 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:5.f 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:3.11 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:5.e 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:8.8 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:6.17 2024-03-24T12:48:44.080 INFO:teuthology.orchestra.run.smithi190.stdout:6.10 2024-03-24T12:48:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:44.108 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:44.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:44.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:44.241 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:48:44.242 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:48:44.242 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:44.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:44.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:44.344 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:44.369 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:44.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:44.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:44.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:43 smithi029 ceph-mon[29919]: pgmap v997: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 16396/88356 objects degraded (18.557%) 2024-03-24T12:48:44.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:43 smithi029 ceph-mon[41500]: pgmap v997: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.8 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.14k op/s; 16396/88356 objects degraded (18.557%) 2024-03-24T12:48:44.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:44.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:44.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:44.604 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:44.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:44.645 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:44.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:44.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:44.783 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:44.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:44.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:44.924 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:44.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:45.026 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:45.054 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:45.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:45.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:45.176 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:48:45.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:45.195 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:45.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:45.296 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:45.325 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:45.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:45.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:45.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:45 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:48:45.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:45 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:48:45.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:45 smithi190 ceph-mon[33748]: osdmap e420: 8 total, 6 up, 6 in 2024-03-24T12:48:45.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:45 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:48:45.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:45 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:48:45.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:45 smithi029 ceph-mon[41500]: osdmap e420: 8 total, 6 up, 6 in 2024-03-24T12:48:45.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:45 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:48:45.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:45 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:48:45.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:45 smithi029 ceph-mon[29919]: osdmap e420: 8 total, 6 up, 6 in 2024-03-24T12:48:45.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:45.450 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:45.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:45.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:45.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:45.587 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:45.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:45.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:45.616 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op export --pgid 8.9 --file /var/log/ceph/exp.8.9.5 2024-03-24T12:48:45.680 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:48:45.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:45.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:45.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:45.850 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:45.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:45.939 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:48:45.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:45.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:45.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:45.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:46.092 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:46.108 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:46 smithi190 ceph-mon[33748]: pgmap v999: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 15007/78654 objects degraded (19.080%) 2024-03-24T12:48:46.108 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:46 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 15007/78654 objects degraded (19.080%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:46.125 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:46.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:46.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:46.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:46.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:46.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:46.369 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:46.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:46.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:46.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:46 smithi029 ceph-mon[41500]: pgmap v999: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 15007/78654 objects degraded (19.080%) 2024-03-24T12:48:46.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:46 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 15007/78654 objects degraded (19.080%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:46.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:46 smithi029 ceph-mon[29919]: pgmap v999: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.6 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.30k op/s; 15007/78654 objects degraded (19.080%) 2024-03-24T12:48:46.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:46 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 15007/78654 objects degraded (19.080%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:46.497 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:46.523 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:46.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:46.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:46.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:46.665 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:46.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:46.767 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:46.791 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:46.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:46.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:46.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:46.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:46.931 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:46.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:47.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:47.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:47.073 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:47.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:47.174 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:47.198 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:47.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:47.299 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:47.327 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:47.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:47.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:47.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:47.470 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:47.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:47.571 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:47.597 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:47.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:47.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:47.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:47.741 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:47.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:47.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:47.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:47.871 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:47.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:47.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:48:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:48:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:48:47.972 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:48.000 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:48.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:48.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:48.100 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:48 smithi190 ceph-mon[33748]: pgmap v1000: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 13094/70608 objects degraded (18.545%) 2024-03-24T12:48:48.100 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:48 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4110952849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:48:48.101 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:48.128 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:48.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:48.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:48.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:48.255 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:48.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:48.356 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:48.392 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:48.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:48.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:48.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:48 smithi029 ceph-mon[29919]: pgmap v1000: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 13094/70608 objects degraded (18.545%) 2024-03-24T12:48:48.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:48 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4110952849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:48:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:48 smithi029 ceph-mon[41500]: pgmap v1000: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.4 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 13094/70608 objects degraded (18.545%) 2024-03-24T12:48:48.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:48 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4110952849' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:48:48.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:48.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:48.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:48.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:48.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:48.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:48.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:48.791 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:48.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:48.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:48:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:48:48.892 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:48.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:48:48 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:48:48.630+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:48:48.921 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:48.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:48.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:49.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:49.033 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:48:49.033 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:48:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:49.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:49.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:49.169 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:49.180 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:48:49.181 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:48:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:49.193 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:49.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:49.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:49.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:49.318 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:49.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:49 smithi190 ceph-mon[33748]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:48:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:49 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4110952849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:48:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:49 smithi190 ceph-mon[33748]: osdmap e421: 8 total, 6 up, 6 in 2024-03-24T12:48:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:49.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:49 smithi029 ceph-mon[29919]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:48:49.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:49 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4110952849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:48:49.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:49 smithi029 ceph-mon[29919]: osdmap e421: 8 total, 6 up, 6 in 2024-03-24T12:48:49.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:49.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:49 smithi029 ceph-mon[41500]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:48:49.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:49 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4110952849' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-24T12:48:49.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:49 smithi029 ceph-mon[41500]: osdmap e421: 8 total, 6 up, 6 in 2024-03-24T12:48:49.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:48:49.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:49.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:49.558 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:49.586 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:49.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:49.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:49.797 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:49.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:49.899 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:49.930 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:49.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:49.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:49.955 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:48:50.033 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:50.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:50.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:50.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:50.164 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:50.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:50.188 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:50.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:50.289 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:50.314 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:50.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:50.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:50.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:50 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:50.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:50 smithi190 ceph-mon[33748]: pgmap v1002: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.2 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 11743/61440 objects degraded (19.113%) 2024-03-24T12:48:50.416 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:50.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:50 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:50 smithi029 ceph-mon[41500]: pgmap v1002: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.2 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 11743/61440 objects degraded (19.113%) 2024-03-24T12:48:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:50 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:48:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:50 smithi029 ceph-mon[29919]: pgmap v1002: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.2 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 11743/61440 objects degraded (19.113%) 2024-03-24T12:48:50.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:48:50.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:50.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:50.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:50.500 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is set 2024-03-24T12:48:50.549 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:50.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:50.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:50.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:50.681 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:50.720 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:50.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:50.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:50.822 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:50.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:50.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:50.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:50.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:50.991 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:50.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:51.010 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:48:51.034 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:48:51.093 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:51.123 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:51.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:51.164 INFO:teuthology.orchestra.run.smithi190.stderr:Exporting 8.9 info 8.9( v 203'12862 (185'6400,203'12862] local-lis/les=158/159 n=1318 ec=158/158 lis/c=158/158 les/c/f=159/159/0 sis=158) 2024-03-24T12:48:51.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:51.232 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9000b2fb:::benchmark_data_smithi029_108893_object112318:head# 2024-03-24T12:48:51.232 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:51 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 11743/61440 objects degraded (19.113%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:51.233 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:900358e5:::benchmark_data_smithi029_108893_object97012:head# 2024-03-24T12:48:51.233 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90037405:::benchmark_data_smithi029_108893_object107896:head# 2024-03-24T12:48:51.233 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9009351c:::benchmark_data_smithi029_108893_object93906:head# 2024-03-24T12:48:51.233 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90095f10:::benchmark_data_smithi029_108893_object106364:head# 2024-03-24T12:48:51.234 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:900c6ff7:::benchmark_data_smithi029_108893_object107228:head# 2024-03-24T12:48:51.235 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:900f7a82:::benchmark_data_smithi029_108893_object104509:head# 2024-03-24T12:48:51.235 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:901163b7:::benchmark_data_smithi029_108893_object98774:head# 2024-03-24T12:48:51.236 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9015ad7d:::benchmark_data_smithi029_108893_object100263:head# 2024-03-24T12:48:51.236 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:901614e2:::benchmark_data_smithi029_108893_object108629:head# 2024-03-24T12:48:51.236 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:901b27d9:::benchmark_data_smithi029_108893_object103576:head# 2024-03-24T12:48:51.237 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:901c8ae9:::benchmark_data_smithi029_108893_object107025:head# 2024-03-24T12:48:51.240 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9021b582:::benchmark_data_smithi029_108893_object104669:head# 2024-03-24T12:48:51.240 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9025de60:::benchmark_data_smithi029_108893_object94122:head# 2024-03-24T12:48:51.240 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90265103:::benchmark_data_smithi029_108893_object106805:head# 2024-03-24T12:48:51.241 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:902ee47f:::benchmark_data_smithi029_108893_object103977:head# 2024-03-24T12:48:51.241 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:902ff1ab:::benchmark_data_smithi029_108893_object109948:head# 2024-03-24T12:48:51.241 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9031f8cf:::benchmark_data_smithi029_108893_object100213:head# 2024-03-24T12:48:51.242 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9032cd75:::benchmark_data_smithi029_108893_object99297:head# 2024-03-24T12:48:51.242 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9033f931:::benchmark_data_smithi029_108893_object98116:head# 2024-03-24T12:48:51.242 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9034df38:::benchmark_data_smithi029_108893_object107190:head# 2024-03-24T12:48:51.242 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90356d0a:::benchmark_data_smithi029_108893_object97754:head# 2024-03-24T12:48:51.243 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9039872a:::benchmark_data_smithi029_108893_object113466:head# 2024-03-24T12:48:51.243 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90403a4a:::benchmark_data_smithi029_108893_object97602:head# 2024-03-24T12:48:51.243 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9041d070:::benchmark_data_smithi029_108893_object113156:head# 2024-03-24T12:48:51.244 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90434f03:::benchmark_data_smithi029_108893_object96887:head# 2024-03-24T12:48:51.246 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90470448:::benchmark_data_smithi029_108893_object101773:head# 2024-03-24T12:48:51.246 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90473c5c:::benchmark_data_smithi029_108893_object110920:head# 2024-03-24T12:48:51.246 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90474497:::benchmark_data_smithi029_108893_object95555:head# 2024-03-24T12:48:51.247 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:904c8912:::benchmark_data_smithi029_108893_object93990:head# 2024-03-24T12:48:51.247 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90514e8f:::benchmark_data_smithi029_108893_object112216:head# 2024-03-24T12:48:51.248 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:905528fa:::benchmark_data_smithi029_108893_object93764:head# 2024-03-24T12:48:51.249 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:905828b4:::benchmark_data_smithi029_108893_object99086:head# 2024-03-24T12:48:51.250 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:905886d0:::benchmark_data_smithi029_108893_object94771:head# 2024-03-24T12:48:51.252 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:905aa559:::benchmark_data_smithi029_108893_object98553:head# 2024-03-24T12:48:51.252 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:905b992b:::benchmark_data_smithi029_108893_object94254:head# 2024-03-24T12:48:51.252 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:905e960c:::benchmark_data_smithi029_108893_object107661:head# 2024-03-24T12:48:51.253 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:905ed818:::benchmark_data_smithi029_108893_object99282:head# 2024-03-24T12:48:51.253 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:906135db:::benchmark_data_smithi029_108893_object104385:head# 2024-03-24T12:48:51.256 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90679149:::benchmark_data_smithi029_108893_object99381:head# 2024-03-24T12:48:51.256 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90685ea9:::benchmark_data_smithi029_108893_object95665:head# 2024-03-24T12:48:51.257 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9068f5d8:::benchmark_data_smithi029_108893_object109130:head# 2024-03-24T12:48:51.257 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:906b5c25:::benchmark_data_smithi029_108893_object102356:head# 2024-03-24T12:48:51.258 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:906c787d:::benchmark_data_smithi029_108893_object103812:head# 2024-03-24T12:48:51.258 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:906dd7fa:::benchmark_data_smithi029_108893_object104537:head# 2024-03-24T12:48:51.259 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9071a57f:::benchmark_data_smithi029_108893_object110306:head# 2024-03-24T12:48:51.260 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9073cd19:::benchmark_data_smithi029_108893_object109093:head# 2024-03-24T12:48:51.261 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90744038:::benchmark_data_smithi029_108893_object104216:head# 2024-03-24T12:48:51.262 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:907b6263:::benchmark_data_smithi029_108893_object101739:head# 2024-03-24T12:48:51.262 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:907f5f77:::benchmark_data_smithi029_108893_object109694:head# 2024-03-24T12:48:51.263 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:908b3dff:::benchmark_data_smithi029_108893_object94108:head# 2024-03-24T12:48:51.263 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:908eb9a8:::benchmark_data_smithi029_108893_object102583:head# 2024-03-24T12:48:51.263 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9092bb36:::benchmark_data_smithi029_108893_object99887:head# 2024-03-24T12:48:51.264 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90969b8a:::benchmark_data_smithi029_108893_object103471:head# 2024-03-24T12:48:51.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:51.272 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:51.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:51.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9097f63e:::benchmark_data_smithi029_108893_object97210:head# 2024-03-24T12:48:51.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90989a8b:::benchmark_data_smithi029_108893_object95728:head# 2024-03-24T12:48:51.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9099c8fd:::benchmark_data_smithi029_108893_object109876:head# 2024-03-24T12:48:51.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:909adcd4:::benchmark_data_smithi029_108893_object103036:head# 2024-03-24T12:48:51.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90a06e28:::benchmark_data_smithi029_108893_object111070:head# 2024-03-24T12:48:51.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90a385ba:::benchmark_data_smithi029_108893_object113776:head# 2024-03-24T12:48:51.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90a7d3f1:::benchmark_data_smithi029_108893_object112200:head# 2024-03-24T12:48:51.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90aa1592:::benchmark_data_smithi029_108893_object112336:head# 2024-03-24T12:48:51.274 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90acf32e:::benchmark_data_smithi029_108893_object98755:head# 2024-03-24T12:48:51.275 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90af6799:::benchmark_data_smithi029_108893_object93932:head# 2024-03-24T12:48:51.276 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90b55c96:::benchmark_data_smithi029_108893_object99488:head# 2024-03-24T12:48:51.276 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90b704b9:::benchmark_data_smithi029_108893_object95905:head# 2024-03-24T12:48:51.276 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90ba49d1:::benchmark_data_smithi029_108893_object106926:head# 2024-03-24T12:48:51.276 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90c05b45:::benchmark_data_smithi029_108893_object111040:head# 2024-03-24T12:48:51.277 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90c7567b:::benchmark_data_smithi029_108893_object94251:head# 2024-03-24T12:48:51.278 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90cb993d:::benchmark_data_smithi029_108893_object105611:head# 2024-03-24T12:48:51.279 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90cc3349:::benchmark_data_smithi029_108893_object108663:head# 2024-03-24T12:48:51.280 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90d0e6a3:::benchmark_data_smithi029_108893_object107242:head# 2024-03-24T12:48:51.280 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90d21b6e:::benchmark_data_smithi029_108893_object102443:head# 2024-03-24T12:48:51.281 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90d34d32:::benchmark_data_smithi029_108893_object112889:head# 2024-03-24T12:48:51.282 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90dd1c7b:::benchmark_data_smithi029_108893_object102292:head# 2024-03-24T12:48:51.282 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90eee412:::benchmark_data_smithi029_108893_object108873:head# 2024-03-24T12:48:51.284 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90f4fe98:::benchmark_data_smithi029_108893_object112724:head# 2024-03-24T12:48:51.286 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90f89a1e:::benchmark_data_smithi029_108893_object97199:head# 2024-03-24T12:48:51.286 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90f97d6c:::benchmark_data_smithi029_108893_object99765:head# 2024-03-24T12:48:51.287 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90fd1ae3:::benchmark_data_smithi029_108893_object99999:head# 2024-03-24T12:48:51.288 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90fe5271:::benchmark_data_smithi029_108893_object98905:head# 2024-03-24T12:48:51.288 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:90fe7c0b:::benchmark_data_smithi029_108893_object101145:head# 2024-03-24T12:48:51.289 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91003619:::benchmark_data_smithi029_108893_object110403:head# 2024-03-24T12:48:51.290 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9103516f:::benchmark_data_smithi029_108893_object106240:head# 2024-03-24T12:48:51.290 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91042d5a:::benchmark_data_smithi029_108893_object102659:head# 2024-03-24T12:48:51.291 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:910452a7:::benchmark_data_smithi029_108893_object111525:head# 2024-03-24T12:48:51.292 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:910bd660:::benchmark_data_smithi029_108893_object93006:head# 2024-03-24T12:48:51.292 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:910c976e:::benchmark_data_smithi029_108893_object106740:head# 2024-03-24T12:48:51.293 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:910ee3f6:::benchmark_data_smithi029_108893_object108956:head# 2024-03-24T12:48:51.293 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91176c1f:::benchmark_data_smithi029_108893_object111756:head# 2024-03-24T12:48:51.294 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:911c06cd:::benchmark_data_smithi029_108893_object110372:head# 2024-03-24T12:48:51.295 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:911d4703:::benchmark_data_smithi029_108893_object94493:head# 2024-03-24T12:48:51.297 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9120f037:::benchmark_data_smithi029_108893_object112813:head# 2024-03-24T12:48:51.297 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9128b6a8:::benchmark_data_smithi029_108893_object102475:head# 2024-03-24T12:48:51.297 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:912df998:::benchmark_data_smithi029_108893_object106874:head# 2024-03-24T12:48:51.297 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91340fe9:::benchmark_data_smithi029_108893_object93167:head# 2024-03-24T12:48:51.299 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:913601b5:::benchmark_data_smithi029_108893_object97581:head# 2024-03-24T12:48:51.301 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:913680f8:::benchmark_data_smithi029_108893_object94227:head# 2024-03-24T12:48:51.302 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:913a9c33:::benchmark_data_smithi029_108893_object111351:head# 2024-03-24T12:48:51.302 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:913af128:::benchmark_data_smithi029_108893_object100235:head# 2024-03-24T12:48:51.303 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:913e8024:::benchmark_data_smithi029_108893_object97303:head# 2024-03-24T12:48:51.303 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91403703:::benchmark_data_smithi029_108893_object103581:head# 2024-03-24T12:48:51.303 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91414417:::benchmark_data_smithi029_108893_object113799:head# 2024-03-24T12:48:51.305 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9145efba:::benchmark_data_smithi029_108893_object113378:head# 2024-03-24T12:48:51.305 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:914d8c71:::benchmark_data_smithi029_108893_object97453:head# 2024-03-24T12:48:51.306 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:914da8c1:::benchmark_data_smithi029_108893_object95681:head# 2024-03-24T12:48:51.307 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:914ef725:::benchmark_data_smithi029_108893_object101558:head# 2024-03-24T12:48:51.307 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9155ccbe:::benchmark_data_smithi029_108893_object104424:head# 2024-03-24T12:48:51.308 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91569aec:::benchmark_data_smithi029_108893_object106241:head# 2024-03-24T12:48:51.308 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:915ae288:::benchmark_data_smithi029_108893_object107007:head# 2024-03-24T12:48:51.308 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:915be4da:::benchmark_data_smithi029_108893_object107635:head# 2024-03-24T12:48:51.309 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:915fbf07:::benchmark_data_smithi029_108893_object108627:head# 2024-03-24T12:48:51.309 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9161684f:::benchmark_data_smithi029_108893_object96670:head# 2024-03-24T12:48:51.310 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:916452d5:::benchmark_data_smithi029_108893_object111042:head# 2024-03-24T12:48:51.310 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9165586d:::benchmark_data_smithi029_108893_object106069:head# 2024-03-24T12:48:51.311 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9166b456:::benchmark_data_smithi029_108893_object94573:head# 2024-03-24T12:48:51.313 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9167663a:::benchmark_data_smithi029_108893_object93324:head# 2024-03-24T12:48:51.314 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:917050dc:::benchmark_data_smithi029_108893_object94222:head# 2024-03-24T12:48:51.315 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91793776:::benchmark_data_smithi029_108893_object102865:head# 2024-03-24T12:48:51.315 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:917a2fef:::benchmark_data_smithi029_108893_object95286:head# 2024-03-24T12:48:51.316 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:917a467f:::benchmark_data_smithi029_108893_object96003:head# 2024-03-24T12:48:51.316 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:917af41e:::benchmark_data_smithi029_108893_object102412:head# 2024-03-24T12:48:51.317 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:917c8501:::benchmark_data_smithi029_108893_object108446:head# 2024-03-24T12:48:51.317 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:917cca72:::benchmark_data_smithi029_108893_object104723:head# 2024-03-24T12:48:51.318 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91830655:::benchmark_data_smithi029_108893_object111543:head# 2024-03-24T12:48:51.318 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91845b1f:::benchmark_data_smithi029_108893_object101270:head# 2024-03-24T12:48:51.320 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9186d114:::benchmark_data_smithi029_108893_object106694:head# 2024-03-24T12:48:51.320 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9187fcf1:::benchmark_data_smithi029_108893_object110860:head# 2024-03-24T12:48:51.320 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:918b62de:::benchmark_data_smithi029_108893_object106828:head# 2024-03-24T12:48:51.321 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91993f4c:::benchmark_data_smithi029_108893_object100279:head# 2024-03-24T12:48:51.322 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:919d266b:::benchmark_data_smithi029_108893_object101255:head# 2024-03-24T12:48:51.323 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:919e220d:::benchmark_data_smithi029_108893_object105947:head# 2024-03-24T12:48:51.323 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:919eb35a:::benchmark_data_smithi029_108893_object100126:head# 2024-03-24T12:48:51.324 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91a278ad:::benchmark_data_smithi029_108893_object105571:head# 2024-03-24T12:48:51.324 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91a61d21:::benchmark_data_smithi029_108893_object95672:head# 2024-03-24T12:48:51.325 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91a7baba:::benchmark_data_smithi029_108893_object93210:head# 2024-03-24T12:48:51.325 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91ac0bf2:::benchmark_data_smithi029_108893_object102090:head# 2024-03-24T12:48:51.326 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91b230ee:::benchmark_data_smithi029_108893_object93423:head# 2024-03-24T12:48:51.327 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91b2fbb4:::benchmark_data_smithi029_108893_object110766:head# 2024-03-24T12:48:51.327 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91b4c2ec:::benchmark_data_smithi029_108893_object100784:head# 2024-03-24T12:48:51.327 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91b5227f:::benchmark_data_smithi029_108893_object103611:head# 2024-03-24T12:48:51.328 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91c02a81:::benchmark_data_smithi029_108893_object110874:head# 2024-03-24T12:48:51.329 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91c0e839:::benchmark_data_smithi029_108893_object113583:head# 2024-03-24T12:48:51.329 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91c6afbc:::benchmark_data_smithi029_108893_object112141:head# 2024-03-24T12:48:51.330 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91ce3342:::benchmark_data_smithi029_108893_object96314:head# 2024-03-24T12:48:51.330 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91d9e4dd:::benchmark_data_smithi029_108893_object101302:head# 2024-03-24T12:48:51.331 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91d9fe8c:::benchmark_data_smithi029_108893_object113450:head# 2024-03-24T12:48:51.331 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91dc1c7d:::benchmark_data_smithi029_108893_object102573:head# 2024-03-24T12:48:51.332 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91dd7fe6:::benchmark_data_smithi029_108893_object102689:head# 2024-03-24T12:48:51.333 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91de62cc:::benchmark_data_smithi029_108893_object98057:head# 2024-03-24T12:48:51.333 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91e09404:::benchmark_data_smithi029_108893_object106808:head# 2024-03-24T12:48:51.334 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91e205cf:::benchmark_data_smithi029_108893_object94684:head# 2024-03-24T12:48:51.334 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91e51899:::benchmark_data_smithi029_108893_object108024:head# 2024-03-24T12:48:51.337 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91ef0061:::benchmark_data_smithi029_108893_object111772:head# 2024-03-24T12:48:51.337 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91ef3682:::benchmark_data_smithi029_108893_object96346:head# 2024-03-24T12:48:51.337 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91f21f07:::benchmark_data_smithi029_108893_object113359:head# 2024-03-24T12:48:51.339 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91f32910:::benchmark_data_smithi029_108893_object99409:head# 2024-03-24T12:48:51.342 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91f32f9f:::benchmark_data_smithi029_108893_object107748:head# 2024-03-24T12:48:51.342 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91f45599:::benchmark_data_smithi029_108893_object113251:head# 2024-03-24T12:48:51.343 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91f94f83:::benchmark_data_smithi029_108893_object95219:head# 2024-03-24T12:48:51.346 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91fa0b99:::benchmark_data_smithi029_108893_object95060:head# 2024-03-24T12:48:51.346 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91fbb585:::benchmark_data_smithi029_108893_object96958:head# 2024-03-24T12:48:51.347 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91fd4fb3:::benchmark_data_smithi029_108893_object93867:head# 2024-03-24T12:48:51.347 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91fe3205:::benchmark_data_smithi029_108893_object110502:head# 2024-03-24T12:48:51.348 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:91ffca07:::benchmark_data_smithi029_108893_object105912:head# 2024-03-24T12:48:51.348 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9208e247:::benchmark_data_smithi029_108893_object108463:head# 2024-03-24T12:48:51.350 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:920a1d2a:::benchmark_data_smithi029_108893_object99957:head# 2024-03-24T12:48:51.350 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:920f29a7:::benchmark_data_smithi029_108893_object111542:head# 2024-03-24T12:48:51.351 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:920f472c:::benchmark_data_smithi029_108893_object98314:head# 2024-03-24T12:48:51.351 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:920fd56b:::benchmark_data_smithi029_108893_object102761:head# 2024-03-24T12:48:51.351 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9215db43:::benchmark_data_smithi029_108893_object97686:head# 2024-03-24T12:48:51.351 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92184df6:::benchmark_data_smithi029_108893_object108153:head# 2024-03-24T12:48:51.351 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92186043:::benchmark_data_smithi029_108893_object96717:head# 2024-03-24T12:48:51.352 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:921d6329:::benchmark_data_smithi029_108893_object100178:head# 2024-03-24T12:48:51.352 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:921f04f0:::benchmark_data_smithi029_108893_object100795:head# 2024-03-24T12:48:51.354 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:921f7c9a:::benchmark_data_smithi029_108893_object109672:head# 2024-03-24T12:48:51.354 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92202c0e:::benchmark_data_smithi029_108893_object94415:head# 2024-03-24T12:48:51.355 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92258825:::benchmark_data_smithi029_108893_object102037:head# 2024-03-24T12:48:51.355 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9225bbe3:::benchmark_data_smithi029_108893_object100310:head# 2024-03-24T12:48:51.356 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92279125:::benchmark_data_smithi029_108893_object113764:head# 2024-03-24T12:48:51.356 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9227c539:::benchmark_data_smithi029_108893_object95807:head# 2024-03-24T12:48:51.356 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9236f54f:::benchmark_data_smithi029_108893_object110007:head# 2024-03-24T12:48:51.357 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:923ca5fc:::benchmark_data_smithi029_108893_object94770:head# 2024-03-24T12:48:51.358 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:923e809d:::benchmark_data_smithi029_108893_object108285:head# 2024-03-24T12:48:51.359 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:923ea804:::benchmark_data_smithi029_108893_object97573:head# 2024-03-24T12:48:51.359 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:923f338b:::benchmark_data_smithi029_108893_object103866:head# 2024-03-24T12:48:51.360 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9240cc8c:::benchmark_data_smithi029_108893_object104919:head# 2024-03-24T12:48:51.361 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92426b17:::benchmark_data_smithi029_108893_object113026:head# 2024-03-24T12:48:51.363 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:924e265f:::benchmark_data_smithi029_108893_object106080:head# 2024-03-24T12:48:51.363 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92512b6b:::benchmark_data_smithi029_108893_object110363:head# 2024-03-24T12:48:51.366 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92526021:::benchmark_data_smithi029_108893_object99462:head# 2024-03-24T12:48:51.367 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9253db63:::benchmark_data_smithi029_108893_object105668:head# 2024-03-24T12:48:51.368 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9254ad85:::benchmark_data_smithi029_108893_object98451:head# 2024-03-24T12:48:51.369 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:925ad2d1:::benchmark_data_smithi029_108893_object107765:head# 2024-03-24T12:48:51.369 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92625e25:::benchmark_data_smithi029_108893_object92987:head# 2024-03-24T12:48:51.371 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:926597c9:::benchmark_data_smithi029_108893_object103989:head# 2024-03-24T12:48:51.371 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92666cfb:::benchmark_data_smithi029_108893_object105097:head# 2024-03-24T12:48:51.371 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:926781fb:::benchmark_data_smithi029_108893_object103726:head# 2024-03-24T12:48:51.371 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:926820aa:::benchmark_data_smithi029_108893_object93903:head# 2024-03-24T12:48:51.371 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:926d416d:::benchmark_data_smithi029_108893_object94007:head# 2024-03-24T12:48:51.372 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:927409f6:::benchmark_data_smithi029_108893_object97477:head# 2024-03-24T12:48:51.372 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92747c7d:::benchmark_data_smithi029_108893_object102789:head# 2024-03-24T12:48:51.379 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:51.383 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9277c1a0:::benchmark_data_smithi029_108893_object99489:head# 2024-03-24T12:48:51.383 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92799f00:::benchmark_data_smithi029_108893_object102932:head# 2024-03-24T12:48:51.383 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92834bb1:::benchmark_data_smithi029_108893_object111001:head# 2024-03-24T12:48:51.383 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92846bdc:::benchmark_data_smithi029_108893_object111524:head# 2024-03-24T12:48:51.383 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9285a60c:::benchmark_data_smithi029_108893_object97109:head# 2024-03-24T12:48:51.383 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:928d192c:::benchmark_data_smithi029_108893_object92801:head# 2024-03-24T12:48:51.383 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:928eaaea:::benchmark_data_smithi029_108893_object100694:head# 2024-03-24T12:48:51.383 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:928eb10a:::benchmark_data_smithi029_108893_object93084:head# 2024-03-24T12:48:51.384 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92905b85:::benchmark_data_smithi029_108893_object94319:head# 2024-03-24T12:48:51.384 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:929ebb99:::benchmark_data_smithi029_108893_object102959:head# 2024-03-24T12:48:51.384 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92a023f0:::benchmark_data_smithi029_108893_object111011:head# 2024-03-24T12:48:51.384 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92a14cff:::benchmark_data_smithi029_108893_object96123:head# 2024-03-24T12:48:51.385 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92a33184:::benchmark_data_smithi029_108893_object92703:head# 2024-03-24T12:48:51.385 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92a3fa68:::benchmark_data_smithi029_108893_object102384:head# 2024-03-24T12:48:51.385 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92a483f2:::benchmark_data_smithi029_108893_object105900:head# 2024-03-24T12:48:51.386 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92a68afb:::benchmark_data_smithi029_108893_object94579:head# 2024-03-24T12:48:51.386 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92a8d699:::benchmark_data_smithi029_108893_object99339:head# 2024-03-24T12:48:51.386 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92ad1b9b:::benchmark_data_smithi029_108893_object112841:head# 2024-03-24T12:48:51.386 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92b153b7:::benchmark_data_smithi029_108893_object105750:head# 2024-03-24T12:48:51.387 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92b2197f:::benchmark_data_smithi029_108893_object98715:head# 2024-03-24T12:48:51.388 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92b8c889:::benchmark_data_smithi029_108893_object107179:head# 2024-03-24T12:48:51.388 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92b98678:::benchmark_data_smithi029_108893_object104830:head# 2024-03-24T12:48:51.389 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92c38674:::benchmark_data_smithi029_108893_object105153:head# 2024-03-24T12:48:51.393 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92c44b71:::benchmark_data_smithi029_108893_object104272:head# 2024-03-24T12:48:51.393 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92c99cf3:::benchmark_data_smithi029_108893_object100035:head# 2024-03-24T12:48:51.394 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92d124d4:::benchmark_data_smithi029_108893_object106281:head# 2024-03-24T12:48:51.394 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92d325b2:::benchmark_data_smithi029_108893_object106372:head# 2024-03-24T12:48:51.394 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92d388c1:::benchmark_data_smithi029_108893_object109095:head# 2024-03-24T12:48:51.395 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92daa97f:::benchmark_data_smithi029_108893_object110538:head# 2024-03-24T12:48:51.395 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92dcc3b6:::benchmark_data_smithi029_108893_object99397:head# 2024-03-24T12:48:51.396 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92e58806:::benchmark_data_smithi029_108893_object104887:head# 2024-03-24T12:48:51.396 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92eb90f8:::benchmark_data_smithi029_108893_object95956:head# 2024-03-24T12:48:51.397 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92edcce7:::benchmark_data_smithi029_108893_object95723:head# 2024-03-24T12:48:51.397 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92eed0a5:::benchmark_data_smithi029_108893_object102840:head# 2024-03-24T12:48:51.398 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92f247c4:::benchmark_data_smithi029_108893_object98594:head# 2024-03-24T12:48:51.399 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92f35201:::benchmark_data_smithi029_108893_object113556:head# 2024-03-24T12:48:51.401 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92f43cbd:::benchmark_data_smithi029_108893_object96553:head# 2024-03-24T12:48:51.401 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92f68ccf:::benchmark_data_smithi029_108893_object104238:head# 2024-03-24T12:48:51.401 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92fb5e99:::benchmark_data_smithi029_108893_object108106:head# 2024-03-24T12:48:51.401 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:92fd8766:::benchmark_data_smithi029_108893_object107303:head# 2024-03-24T12:48:51.401 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9301b63c:::benchmark_data_smithi029_108893_object100668:head# 2024-03-24T12:48:51.402 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:930558cd:::benchmark_data_smithi029_108893_object103687:head# 2024-03-24T12:48:51.402 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9305fe27:::benchmark_data_smithi029_108893_object112871:head# 2024-03-24T12:48:51.403 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9308c5fa:::benchmark_data_smithi029_108893_object110572:head# 2024-03-24T12:48:51.403 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:930a201d:::benchmark_data_smithi029_108893_object93590:head# 2024-03-24T12:48:51.404 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:930de23e:::benchmark_data_smithi029_108893_object102232:head# 2024-03-24T12:48:51.404 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93109b1c:::benchmark_data_smithi029_108893_object112356:head# 2024-03-24T12:48:51.405 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93187cb3:::benchmark_data_smithi029_108893_object97832:head# 2024-03-24T12:48:51.406 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9319587f:::benchmark_data_smithi029_108893_object113505:head# 2024-03-24T12:48:51.406 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:932c8701:::benchmark_data_smithi029_108893_object108998:head# 2024-03-24T12:48:51.406 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:932c8a73:::benchmark_data_smithi029_108893_object106564:head# 2024-03-24T12:48:51.407 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9338e144:::benchmark_data_smithi029_108893_object112787:head# 2024-03-24T12:48:51.407 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9341384d:::benchmark_data_smithi029_108893_object94256:head# 2024-03-24T12:48:51.408 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9346df34:::benchmark_data_smithi029_108893_object107681:head# 2024-03-24T12:48:51.410 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9347cde3:::benchmark_data_smithi029_108893_object93454:head# 2024-03-24T12:48:51.411 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93490ad9:::benchmark_data_smithi029_108893_object105735:head# 2024-03-24T12:48:51.411 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:934c2f15:::benchmark_data_smithi029_108893_object105957:head# 2024-03-24T12:48:51.411 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9351c4f2:::benchmark_data_smithi029_108893_object105494:head# 2024-03-24T12:48:51.411 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:935a725f:::benchmark_data_smithi029_108893_object107270:head# 2024-03-24T12:48:51.411 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:935c50bc:::benchmark_data_smithi029_108893_object95265:head# 2024-03-24T12:48:51.413 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93616ad1:::benchmark_data_smithi029_108893_object101289:head# 2024-03-24T12:48:51.413 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9362d4fd:::benchmark_data_smithi029_108893_object93060:head# 2024-03-24T12:48:51.414 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:51.415 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93641df9:::benchmark_data_smithi029_108893_object104801:head# 2024-03-24T12:48:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:51.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:51.418 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9364b2dd:::benchmark_data_smithi029_108893_object105061:head# 2024-03-24T12:48:51.418 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:936e8a7e:::benchmark_data_smithi029_108893_object97351:head# 2024-03-24T12:48:51.418 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:937302d3:::benchmark_data_smithi029_108893_object102549:head# 2024-03-24T12:48:51.418 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:937449bc:::benchmark_data_smithi029_108893_object92803:head# 2024-03-24T12:48:51.418 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93748808:::benchmark_data_smithi029_108893_object100143:head# 2024-03-24T12:48:51.418 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9377ef03:::benchmark_data_smithi029_108893_object93019:head# 2024-03-24T12:48:51.418 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9377f355:::benchmark_data_smithi029_108893_object106885:head# 2024-03-24T12:48:51.418 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93791938:::benchmark_data_smithi029_108893_object94054:head# 2024-03-24T12:48:51.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:51 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 11743/61440 objects degraded (19.113%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:51.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:51 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 11743/61440 objects degraded (19.113%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:51.420 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9379f119:::benchmark_data_smithi029_108893_object95950:head# 2024-03-24T12:48:51.420 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:937c5ff2:::benchmark_data_smithi029_108893_object111665:head# 2024-03-24T12:48:51.420 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:937e2d47:::benchmark_data_smithi029_108893_object100726:head# 2024-03-24T12:48:51.420 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:937f1fd0:::benchmark_data_smithi029_108893_object112627:head# 2024-03-24T12:48:51.421 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:937f7212:::benchmark_data_smithi029_108893_object106824:head# 2024-03-24T12:48:51.421 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:937f9644:::benchmark_data_smithi029_108893_object96908:head# 2024-03-24T12:48:51.421 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9382850b:::benchmark_data_smithi029_108893_object93644:head# 2024-03-24T12:48:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9382eb57:::benchmark_data_smithi029_108893_object112916:head# 2024-03-24T12:48:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9383346b:::benchmark_data_smithi029_108893_object103315:head# 2024-03-24T12:48:51.422 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93833d05:::benchmark_data_smithi029_108893_object101656:head# 2024-03-24T12:48:51.423 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:938352dd:::benchmark_data_smithi029_108893_object107688:head# 2024-03-24T12:48:51.423 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9386f674:::benchmark_data_smithi029_108893_object97162:head# 2024-03-24T12:48:51.424 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93877f6e:::benchmark_data_smithi029_108893_object106643:head# 2024-03-24T12:48:51.425 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93897638:::benchmark_data_smithi029_108893_object110370:head# 2024-03-24T12:48:51.425 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9393ea61:::benchmark_data_smithi029_108893_object96815:head# 2024-03-24T12:48:51.425 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93947679:::benchmark_data_smithi029_108893_object102010:head# 2024-03-24T12:48:51.426 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93949e1d:::benchmark_data_smithi029_108893_object107355:head# 2024-03-24T12:48:51.426 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9394f718:::benchmark_data_smithi029_108893_object104155:head# 2024-03-24T12:48:51.427 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9395748d:::benchmark_data_smithi029_108893_object107237:head# 2024-03-24T12:48:51.427 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93966d66:::benchmark_data_smithi029_108893_object95830:head# 2024-03-24T12:48:51.428 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:939a1eba:::benchmark_data_smithi029_108893_object113585:head# 2024-03-24T12:48:51.428 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:939a6802:::benchmark_data_smithi029_108893_object111841:head# 2024-03-24T12:48:51.428 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:939e16fd:::benchmark_data_smithi029_108893_object105529:head# 2024-03-24T12:48:51.429 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93a1aa51:::benchmark_data_smithi029_108893_object107346:head# 2024-03-24T12:48:51.430 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93a35b88:::benchmark_data_smithi029_108893_object93271:head# 2024-03-24T12:48:51.430 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93a7e558:::benchmark_data_smithi029_108893_object97359:head# 2024-03-24T12:48:51.433 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93a9d21e:::benchmark_data_smithi029_108893_object102301:head# 2024-03-24T12:48:51.434 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93ae2483:::benchmark_data_smithi029_108893_object93072:head# 2024-03-24T12:48:51.434 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93b443ed:::benchmark_data_smithi029_108893_object110899:head# 2024-03-24T12:48:51.435 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93b5192b:::benchmark_data_smithi029_108893_object112090:head# 2024-03-24T12:48:51.435 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93be7e7c:::benchmark_data_smithi029_108893_object101353:head# 2024-03-24T12:48:51.436 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93c05a2e:::benchmark_data_smithi029_108893_object111058:head# 2024-03-24T12:48:51.436 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93c34ff1:::benchmark_data_smithi029_108893_object100499:head# 2024-03-24T12:48:51.436 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93c6c2d3:::benchmark_data_smithi029_108893_object100068:head# 2024-03-24T12:48:51.437 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93c6c334:::benchmark_data_smithi029_108893_object104122:head# 2024-03-24T12:48:51.437 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93cc1725:::benchmark_data_smithi029_108893_object109933:head# 2024-03-24T12:48:51.438 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93cf91a8:::benchmark_data_smithi029_108893_object109417:head# 2024-03-24T12:48:51.438 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93d09e1d:::benchmark_data_smithi029_108893_object108913:head# 2024-03-24T12:48:51.439 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93d73947:::benchmark_data_smithi029_108893_object110754:head# 2024-03-24T12:48:51.439 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93de74e0:::benchmark_data_smithi029_108893_object104153:head# 2024-03-24T12:48:51.440 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93e6272d:::benchmark_data_smithi029_108893_object93485:head# 2024-03-24T12:48:51.441 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93ed539c:::benchmark_data_smithi029_108893_object107628:head# 2024-03-24T12:48:51.441 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93ee71b8:::benchmark_data_smithi029_108893_object98102:head# 2024-03-24T12:48:51.441 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93efbaae:::benchmark_data_smithi029_108893_object113219:head# 2024-03-24T12:48:51.442 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93f356a5:::benchmark_data_smithi029_108893_object111576:head# 2024-03-24T12:48:51.442 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93f73992:::benchmark_data_smithi029_108893_object106484:head# 2024-03-24T12:48:51.443 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:93fe5736:::benchmark_data_smithi029_108893_object110973:head# 2024-03-24T12:48:51.443 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94020f55:::benchmark_data_smithi029_108893_object98545:head# 2024-03-24T12:48:51.444 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94053e1a:::benchmark_data_smithi029_108893_object102006:head# 2024-03-24T12:48:51.445 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9405a9fc:::benchmark_data_smithi029_108893_object92779:head# 2024-03-24T12:48:51.445 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:940e9c89:::benchmark_data_smithi029_108893_object97097:head# 2024-03-24T12:48:51.446 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9411d2e3:::benchmark_data_smithi029_108893_object94663:head# 2024-03-24T12:48:51.447 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9412460d:::benchmark_data_smithi029_108893_object95109:head# 2024-03-24T12:48:51.447 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9415482b:::benchmark_data_smithi029_108893_object112501:head# 2024-03-24T12:48:51.448 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:941d0055:::benchmark_data_smithi029_108893_object110392:head# 2024-03-24T12:48:51.450 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:941d60d4:::benchmark_data_smithi029_108893_object108829:head# 2024-03-24T12:48:51.454 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:941d8983:::benchmark_data_smithi029_108893_object105066:head# 2024-03-24T12:48:51.455 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:941f8649:::benchmark_data_smithi029_108893_object103586:head# 2024-03-24T12:48:51.455 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9424056f:::benchmark_data_smithi029_108893_object98364:head# 2024-03-24T12:48:51.455 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94294406:::benchmark_data_smithi029_108893_object107946:head# 2024-03-24T12:48:51.455 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:942b7c3f:::benchmark_data_smithi029_108893_object107912:head# 2024-03-24T12:48:51.456 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94337971:::benchmark_data_smithi029_108893_object106603:head# 2024-03-24T12:48:51.456 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9437c723:::benchmark_data_smithi029_108893_object100745:head# 2024-03-24T12:48:51.460 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:943b87ec:::benchmark_data_smithi029_108893_object94621:head# 2024-03-24T12:48:51.461 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:943c0453:::benchmark_data_smithi029_108893_object105638:head# 2024-03-24T12:48:51.461 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:943eed1d:::benchmark_data_smithi029_108893_object108971:head# 2024-03-24T12:48:51.461 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:944332db:::benchmark_data_smithi029_108893_object100145:head# 2024-03-24T12:48:51.461 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:944420f4:::benchmark_data_smithi029_108893_object97609:head# 2024-03-24T12:48:51.461 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:944b8c3a:::benchmark_data_smithi029_108893_object107781:head# 2024-03-24T12:48:51.462 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94503fe3:::benchmark_data_smithi029_108893_object107794:head# 2024-03-24T12:48:51.462 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94523d78:::benchmark_data_smithi029_108893_object107044:head# 2024-03-24T12:48:51.463 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9452feca:::benchmark_data_smithi029_108893_object95625:head# 2024-03-24T12:48:51.464 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94542bd3:::benchmark_data_smithi029_108893_object105564:head# 2024-03-24T12:48:51.465 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9454668e:::benchmark_data_smithi029_108893_object94668:head# 2024-03-24T12:48:51.465 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9457ba55:::benchmark_data_smithi029_108893_object111674:head# 2024-03-24T12:48:51.466 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:945b978e:::benchmark_data_smithi029_108893_object103083:head# 2024-03-24T12:48:51.467 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:945be6ed:::benchmark_data_smithi029_108893_object111216:head# 2024-03-24T12:48:51.468 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:945d8aff:::benchmark_data_smithi029_108893_object94515:head# 2024-03-24T12:48:51.468 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:945f236f:::benchmark_data_smithi029_108893_object93215:head# 2024-03-24T12:48:51.469 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9460ffc4:::benchmark_data_smithi029_108893_object105491:head# 2024-03-24T12:48:51.472 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94657117:::benchmark_data_smithi029_108893_object98517:head# 2024-03-24T12:48:51.475 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94676a20:::benchmark_data_smithi029_108893_object100241:head# 2024-03-24T12:48:51.476 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94682738:::benchmark_data_smithi029_108893_object100377:head# 2024-03-24T12:48:51.477 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:946b29b4:::benchmark_data_smithi029_108893_object107715:head# 2024-03-24T12:48:51.477 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9470b9fb:::benchmark_data_smithi029_108893_object103164:head# 2024-03-24T12:48:51.477 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94733038:::benchmark_data_smithi029_108893_object101222:head# 2024-03-24T12:48:51.479 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9474a538:::benchmark_data_smithi029_108893_object104598:head# 2024-03-24T12:48:51.480 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9477ae20:::benchmark_data_smithi029_108893_object104332:head# 2024-03-24T12:48:51.480 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9477c12d:::benchmark_data_smithi029_108893_object105051:head# 2024-03-24T12:48:51.480 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9479409b:::benchmark_data_smithi029_108893_object113260:head# 2024-03-24T12:48:51.488 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:947bf8eb:::benchmark_data_smithi029_108893_object107510:head# 2024-03-24T12:48:51.488 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:947c5912:::benchmark_data_smithi029_108893_object92796:head# 2024-03-24T12:48:51.488 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:947e5048:::benchmark_data_smithi029_108893_object107520:head# 2024-03-24T12:48:51.488 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9482e408:::benchmark_data_smithi029_108893_object100326:head# 2024-03-24T12:48:51.488 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9484cf56:::benchmark_data_smithi029_108893_object99689:head# 2024-03-24T12:48:51.488 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:948efa96:::benchmark_data_smithi029_108893_object96363:head# 2024-03-24T12:48:51.488 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:948f6ec8:::benchmark_data_smithi029_108893_object111562:head# 2024-03-24T12:48:51.488 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9496ce0c:::benchmark_data_smithi029_108893_object113811:head# 2024-03-24T12:48:51.489 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94a071f5:::benchmark_data_smithi029_108893_object98136:head# 2024-03-24T12:48:51.507 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94a1b2f8:::benchmark_data_smithi029_108893_object108916:head# 2024-03-24T12:48:51.507 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94a2683d:::benchmark_data_smithi029_108893_object94892:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94a8dfe6:::benchmark_data_smithi029_108893_object112584:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94ac9fcf:::benchmark_data_smithi029_108893_object110908:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94afbf8c:::benchmark_data_smithi029_108893_object108046:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94b4b58b:::benchmark_data_smithi029_108893_object101399:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94b901be:::benchmark_data_smithi029_108893_object102182:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94b9a885:::benchmark_data_smithi029_108893_object94619:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94bdfd58:::benchmark_data_smithi029_108893_object97105:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94c177a7:::benchmark_data_smithi029_108893_object101764:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94c2b3e3:::benchmark_data_smithi029_108893_object97699:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94c5f2b1:::benchmark_data_smithi029_108893_object112833:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94c77361:::benchmark_data_smithi029_108893_object95919:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94c7a01d:::benchmark_data_smithi029_108893_object101573:head# 2024-03-24T12:48:51.508 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94d03ec1:::benchmark_data_smithi029_108893_object109630:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94d40414:::benchmark_data_smithi029_108893_object113617:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94d771e6:::benchmark_data_smithi029_108893_object97733:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94de6ca5:::benchmark_data_smithi029_108893_object103942:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94de7e6b:::benchmark_data_smithi029_108893_object112316:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94e0087c:::benchmark_data_smithi029_108893_object99560:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94e0719f:::benchmark_data_smithi029_108893_object95713:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94e21d9a:::benchmark_data_smithi029_108893_object95657:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94e4a302:::benchmark_data_smithi029_108893_object110354:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94e5d0d1:::benchmark_data_smithi029_108893_object111793:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94e7e033:::benchmark_data_smithi029_108893_object107387:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94efdf0f:::benchmark_data_smithi029_108893_object105596:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94f318c1:::benchmark_data_smithi029_108893_object108936:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94f40035:::benchmark_data_smithi029_108893_object100363:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:94f42d94:::benchmark_data_smithi029_108893_object99443:head# 2024-03-24T12:48:51.509 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95078d34:::benchmark_data_smithi029_108893_object108326:head# 2024-03-24T12:48:51.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9507e1eb:::benchmark_data_smithi029_108893_object103073:head# 2024-03-24T12:48:51.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95086773:::benchmark_data_smithi029_108893_object112483:head# 2024-03-24T12:48:51.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:950a06c7:::benchmark_data_smithi029_108893_object95827:head# 2024-03-24T12:48:51.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:950cf255:::benchmark_data_smithi029_108893_object97851:head# 2024-03-24T12:48:51.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95106fc9:::benchmark_data_smithi029_108893_object101149:head# 2024-03-24T12:48:51.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9511d8ca:::benchmark_data_smithi029_108893_object101523:head# 2024-03-24T12:48:51.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9516a7ef:::benchmark_data_smithi029_108893_object111283:head# 2024-03-24T12:48:51.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:951b38dc:::benchmark_data_smithi029_108893_object110155:head# 2024-03-24T12:48:51.514 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:951cc51b:::benchmark_data_smithi029_108893_object101845:head# 2024-03-24T12:48:51.515 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9522d2d4:::benchmark_data_smithi029_108893_object96657:head# 2024-03-24T12:48:51.515 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9523b0c8:::benchmark_data_smithi029_108893_object100477:head# 2024-03-24T12:48:51.515 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95242349:::benchmark_data_smithi029_108893_object109312:head# 2024-03-24T12:48:51.515 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9524632a:::benchmark_data_smithi029_108893_object101209:head# 2024-03-24T12:48:51.516 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95287e05:::benchmark_data_smithi029_108893_object101103:head# 2024-03-24T12:48:51.516 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:51.517 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95289721:::benchmark_data_smithi029_108893_object105410:head# 2024-03-24T12:48:51.517 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9533d8a7:::benchmark_data_smithi029_108893_object101148:head# 2024-03-24T12:48:51.518 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95345909:::benchmark_data_smithi029_108893_object106066:head# 2024-03-24T12:48:51.518 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9535e6d1:::benchmark_data_smithi029_108893_object108026:head# 2024-03-24T12:48:51.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:953cf31e:::benchmark_data_smithi029_108893_object103999:head# 2024-03-24T12:48:51.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:953f62eb:::benchmark_data_smithi029_108893_object104966:head# 2024-03-24T12:48:51.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:953f88b5:::benchmark_data_smithi029_108893_object97767:head# 2024-03-24T12:48:51.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9545a27d:::benchmark_data_smithi029_108893_object103448:head# 2024-03-24T12:48:51.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:954745a1:::benchmark_data_smithi029_108893_object104681:head# 2024-03-24T12:48:51.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:954cf174:::benchmark_data_smithi029_108893_object99296:head# 2024-03-24T12:48:51.522 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:954d3b8e:::benchmark_data_smithi029_108893_object93735:head# 2024-03-24T12:48:51.523 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9552e604:::benchmark_data_smithi029_108893_object100378:head# 2024-03-24T12:48:51.525 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95543b01:::benchmark_data_smithi029_108893_object100771:head# 2024-03-24T12:48:51.525 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95589ef3:::benchmark_data_smithi029_108893_object104748:head# 2024-03-24T12:48:51.527 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:955bedd1:::benchmark_data_smithi029_108893_object109571:head# 2024-03-24T12:48:51.528 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:955ebf8c:::benchmark_data_smithi029_108893_object105021:head# 2024-03-24T12:48:51.530 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:955fe7ea:::benchmark_data_smithi029_108893_object108113:head# 2024-03-24T12:48:51.530 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95613d7f:::benchmark_data_smithi029_108893_object111484:head# 2024-03-24T12:48:51.531 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:956483af:::benchmark_data_smithi029_108893_object107019:head# 2024-03-24T12:48:51.531 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:956949e3:::benchmark_data_smithi029_108893_object100343:head# 2024-03-24T12:48:51.532 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:956ba146:::benchmark_data_smithi029_108893_object108445:head# 2024-03-24T12:48:51.532 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:956bd998:::benchmark_data_smithi029_108893_object109942:head# 2024-03-24T12:48:51.533 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9571f0e5:::benchmark_data_smithi029_108893_object99528:head# 2024-03-24T12:48:51.534 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9571f897:::benchmark_data_smithi029_108893_object106385:head# 2024-03-24T12:48:51.534 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95728645:::benchmark_data_smithi029_108893_object112523:head# 2024-03-24T12:48:51.535 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9577a550:::benchmark_data_smithi029_108893_object102741:head# 2024-03-24T12:48:51.535 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:957b2fa9:::benchmark_data_smithi029_108893_object110420:head# 2024-03-24T12:48:51.535 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:957c9d81:::benchmark_data_smithi029_108893_object105378:head# 2024-03-24T12:48:51.536 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:957e220e:::benchmark_data_smithi029_108893_object101690:head# 2024-03-24T12:48:51.536 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:957efb41:::benchmark_data_smithi029_108893_object98128:head# 2024-03-24T12:48:51.537 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95827adc:::benchmark_data_smithi029_108893_object112577:head# 2024-03-24T12:48:51.539 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95835fde:::benchmark_data_smithi029_108893_object101172:head# 2024-03-24T12:48:51.540 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:958382bf:::benchmark_data_smithi029_108893_object108050:head# 2024-03-24T12:48:51.540 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95852812:::benchmark_data_smithi029_108893_object95124:head# 2024-03-24T12:48:51.543 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:958905a3:::benchmark_data_smithi029_108893_object109803:head# 2024-03-24T12:48:51.543 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:958e7195:::benchmark_data_smithi029_108893_object102067:head# 2024-03-24T12:48:51.545 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:51.546 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:959753fe:::benchmark_data_smithi029_108893_object107538:head# 2024-03-24T12:48:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:51.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:51.548 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95983f5c:::benchmark_data_smithi029_108893_object96890:head# 2024-03-24T12:48:51.548 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95a49a49:::benchmark_data_smithi029_108893_object111659:head# 2024-03-24T12:48:51.548 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95a53f6a:::benchmark_data_smithi029_108893_object100464:head# 2024-03-24T12:48:51.548 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95a76f49:::benchmark_data_smithi029_108893_object109389:head# 2024-03-24T12:48:51.549 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95aaad63:::benchmark_data_smithi029_108893_object111113:head# 2024-03-24T12:48:51.552 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95b08769:::benchmark_data_smithi029_108893_object99758:head# 2024-03-24T12:48:51.555 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95b3198e:::benchmark_data_smithi029_108893_object97343:head# 2024-03-24T12:48:51.558 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95bdd196:::benchmark_data_smithi029_108893_object93890:head# 2024-03-24T12:48:51.558 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95c05ed1:::benchmark_data_smithi029_108893_object108727:head# 2024-03-24T12:48:51.558 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95c5b9d9:::benchmark_data_smithi029_108893_object108068:head# 2024-03-24T12:48:51.559 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95c6d5f4:::benchmark_data_smithi029_108893_object106350:head# 2024-03-24T12:48:51.559 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95c73b68:::benchmark_data_smithi029_108893_object101565:head# 2024-03-24T12:48:51.560 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95cb6d43:::benchmark_data_smithi029_108893_object94982:head# 2024-03-24T12:48:51.561 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95cbd07a:::benchmark_data_smithi029_108893_object96212:head# 2024-03-24T12:48:51.561 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95cbdba7:::benchmark_data_smithi029_108893_object113815:head# 2024-03-24T12:48:51.561 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95cd9b7b:::benchmark_data_smithi029_108893_object100194:head# 2024-03-24T12:48:51.562 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95d5e116:::benchmark_data_smithi029_108893_object106455:head# 2024-03-24T12:48:51.563 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95da3ed7:::benchmark_data_smithi029_108893_object98674:head# 2024-03-24T12:48:51.563 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95dce4d2:::benchmark_data_smithi029_108893_object95022:head# 2024-03-24T12:48:51.564 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95e5fd2c:::benchmark_data_smithi029_108893_object98192:head# 2024-03-24T12:48:51.566 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95e70fd9:::benchmark_data_smithi029_108893_object111110:head# 2024-03-24T12:48:51.570 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95e78353:::benchmark_data_smithi029_108893_object112351:head# 2024-03-24T12:48:51.570 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95e7d7c9:::benchmark_data_smithi029_108893_object98438:head# 2024-03-24T12:48:51.570 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95e85d2d:::benchmark_data_smithi029_108893_object94326:head# 2024-03-24T12:48:51.570 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95ec293f:::benchmark_data_smithi029_108893_object101281:head# 2024-03-24T12:48:51.571 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95ef1a18:::benchmark_data_smithi029_108893_object110904:head# 2024-03-24T12:48:51.571 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95f143bf:::benchmark_data_smithi029_108893_object110321:head# 2024-03-24T12:48:51.572 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95f26808:::benchmark_data_smithi029_108893_object113753:head# 2024-03-24T12:48:51.572 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95f3029b:::benchmark_data_smithi029_108893_object97455:head# 2024-03-24T12:48:51.573 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95f576d8:::benchmark_data_smithi029_108893_object101613:head# 2024-03-24T12:48:51.575 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95f9b433:::benchmark_data_smithi029_108893_object96754:head# 2024-03-24T12:48:51.576 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95fac512:::benchmark_data_smithi029_108893_object97386:head# 2024-03-24T12:48:51.576 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:95fb199b:::benchmark_data_smithi029_108893_object103677:head# 2024-03-24T12:48:51.576 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96019b19:::benchmark_data_smithi029_108893_object95313:head# 2024-03-24T12:48:51.578 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9604e983:::benchmark_data_smithi029_108893_object93670:head# 2024-03-24T12:48:51.578 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:960bc739:::benchmark_data_smithi029_108893_object110171:head# 2024-03-24T12:48:51.579 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9610684b:::benchmark_data_smithi029_108893_object100180:head# 2024-03-24T12:48:51.579 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9610dc62:::benchmark_data_smithi029_108893_object110518:head# 2024-03-24T12:48:51.580 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9617d77e:::benchmark_data_smithi029_108893_object102678:head# 2024-03-24T12:48:51.580 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96194868:::benchmark_data_smithi029_108893_object100919:head# 2024-03-24T12:48:51.581 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9620e2c8:::benchmark_data_smithi029_108893_object97246:head# 2024-03-24T12:48:51.581 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9621f000:::benchmark_data_smithi029_108893_object112084:head# 2024-03-24T12:48:51.581 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:962d4a66:::benchmark_data_smithi029_108893_object109367:head# 2024-03-24T12:48:51.582 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9630f11b:::benchmark_data_smithi029_108893_object102338:head# 2024-03-24T12:48:51.582 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96330edc:::benchmark_data_smithi029_108893_object95038:head# 2024-03-24T12:48:51.583 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96345a91:::benchmark_data_smithi029_108893_object113722:head# 2024-03-24T12:48:51.583 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96369564:::benchmark_data_smithi029_108893_object106960:head# 2024-03-24T12:48:51.584 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:963805fa:::benchmark_data_smithi029_108893_object113457:head# 2024-03-24T12:48:51.585 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:963f4299:::benchmark_data_smithi029_108893_object109186:head# 2024-03-24T12:48:51.585 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96422dd6:::benchmark_data_smithi029_108893_object108290:head# 2024-03-24T12:48:51.586 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:964e0509:::benchmark_data_smithi029_108893_object98516:head# 2024-03-24T12:48:51.586 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96505947:::benchmark_data_smithi029_108893_object102684:head# 2024-03-24T12:48:51.586 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9650ea20:::benchmark_data_smithi029_108893_object111125:head# 2024-03-24T12:48:51.587 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9657c3de:::benchmark_data_smithi029_108893_object93862:head# 2024-03-24T12:48:51.587 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:965a4bf9:::benchmark_data_smithi029_108893_object98567:head# 2024-03-24T12:48:51.588 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:965ab29f:::benchmark_data_smithi029_108893_object110767:head# 2024-03-24T12:48:51.589 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96642c02:::benchmark_data_smithi029_108893_object106543:head# 2024-03-24T12:48:51.589 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:967264b2:::benchmark_data_smithi029_108893_object97237:head# 2024-03-24T12:48:51.589 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9674b114:::benchmark_data_smithi029_108893_object107801:head# 2024-03-24T12:48:51.590 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9675be59:::benchmark_data_smithi029_108893_object103054:head# 2024-03-24T12:48:51.590 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:967667a3:::benchmark_data_smithi029_108893_object103280:head# 2024-03-24T12:48:51.591 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9676ff21:::benchmark_data_smithi029_108893_object106050:head# 2024-03-24T12:48:51.591 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:967b77a6:::benchmark_data_smithi029_108893_object97919:head# 2024-03-24T12:48:51.592 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:967e1883:::benchmark_data_smithi029_108893_object110356:head# 2024-03-24T12:48:51.592 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96927d7f:::benchmark_data_smithi029_108893_object105226:head# 2024-03-24T12:48:51.593 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96937a5c:::benchmark_data_smithi029_108893_object108888:head# 2024-03-24T12:48:51.593 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9696771e:::benchmark_data_smithi029_108893_object100372:head# 2024-03-24T12:48:51.594 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96968281:::benchmark_data_smithi029_108893_object100733:head# 2024-03-24T12:48:51.596 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96972a14:::benchmark_data_smithi029_108893_object95910:head# 2024-03-24T12:48:51.596 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:969c47ea:::benchmark_data_smithi029_108893_object102249:head# 2024-03-24T12:48:51.596 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:969d50f3:::benchmark_data_smithi029_108893_object94420:head# 2024-03-24T12:48:51.597 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:969d65d0:::benchmark_data_smithi029_108893_object102566:head# 2024-03-24T12:48:51.597 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:969dc20b:::benchmark_data_smithi029_108893_object112016:head# 2024-03-24T12:48:51.597 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:969dda54:::benchmark_data_smithi029_108893_object106927:head# 2024-03-24T12:48:51.598 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:969ddfce:::benchmark_data_smithi029_108893_object105430:head# 2024-03-24T12:48:51.598 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:969f24eb:::benchmark_data_smithi029_108893_object92761:head# 2024-03-24T12:48:51.599 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96a27204:::benchmark_data_smithi029_108893_object100307:head# 2024-03-24T12:48:51.599 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96a51895:::benchmark_data_smithi029_108893_object105864:head# 2024-03-24T12:48:51.600 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96a8ee7d:::benchmark_data_smithi029_108893_object111184:head# 2024-03-24T12:48:51.600 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96ac6fc6:::benchmark_data_smithi029_108893_object102873:head# 2024-03-24T12:48:51.601 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96aea56d:::benchmark_data_smithi029_108893_object98908:head# 2024-03-24T12:48:51.601 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96b0d54f:::benchmark_data_smithi029_108893_object95825:head# 2024-03-24T12:48:51.602 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96b1724a:::benchmark_data_smithi029_108893_object110782:head# 2024-03-24T12:48:51.602 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96b87d99:::benchmark_data_smithi029_108893_object105134:head# 2024-03-24T12:48:51.603 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96bf4960:::benchmark_data_smithi029_108893_object102430:head# 2024-03-24T12:48:51.603 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96c32227:::benchmark_data_smithi029_108893_object96460:head# 2024-03-24T12:48:51.604 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96c339e1:::benchmark_data_smithi029_108893_object102203:head# 2024-03-24T12:48:51.604 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96c499bd:::benchmark_data_smithi029_108893_object109853:head# 2024-03-24T12:48:51.605 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96c825d2:::benchmark_data_smithi029_108893_object106029:head# 2024-03-24T12:48:51.605 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96cca793:::benchmark_data_smithi029_108893_object113635:head# 2024-03-24T12:48:51.606 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96cf5b32:::benchmark_data_smithi029_108893_object103094:head# 2024-03-24T12:48:51.606 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96d0f27d:::benchmark_data_smithi029_108893_object93837:head# 2024-03-24T12:48:51.607 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96d69b5b:::benchmark_data_smithi029_108893_object97895:head# 2024-03-24T12:48:51.607 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96d7670f:::benchmark_data_smithi029_108893_object109147:head# 2024-03-24T12:48:51.609 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96d803de:::benchmark_data_smithi029_108893_object109067:head# 2024-03-24T12:48:51.609 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96d94a3e:::benchmark_data_smithi029_108893_object112352:head# 2024-03-24T12:48:51.609 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96d9e9f0:::benchmark_data_smithi029_108893_object112379:head# 2024-03-24T12:48:51.610 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96de2a9d:::benchmark_data_smithi029_108893_object106437:head# 2024-03-24T12:48:51.610 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96e08a08:::benchmark_data_smithi029_108893_object101094:head# 2024-03-24T12:48:51.611 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96e144e2:::benchmark_data_smithi029_108893_object100436:head# 2024-03-24T12:48:51.611 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96e1d223:::benchmark_data_smithi029_108893_object113018:head# 2024-03-24T12:48:51.612 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96e22047:::benchmark_data_smithi029_108893_object93656:head# 2024-03-24T12:48:51.612 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96e5e463:::benchmark_data_smithi029_108893_object98951:head# 2024-03-24T12:48:51.613 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96ef71b1:::benchmark_data_smithi029_108893_object104444:head# 2024-03-24T12:48:51.613 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96f035f6:::benchmark_data_smithi029_108893_object97216:head# 2024-03-24T12:48:51.614 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96f21ecb:::benchmark_data_smithi029_108893_object109254:head# 2024-03-24T12:48:51.614 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96f46efc:::benchmark_data_smithi029_108893_object105934:head# 2024-03-24T12:48:51.614 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96f69009:::benchmark_data_smithi029_108893_object101540:head# 2024-03-24T12:48:51.615 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96f6f0cb:::benchmark_data_smithi029_108893_object104579:head# 2024-03-24T12:48:51.615 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96f7de43:::benchmark_data_smithi029_108893_object99873:head# 2024-03-24T12:48:51.616 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96fd4e4a:::benchmark_data_smithi029_108893_object109773:head# 2024-03-24T12:48:51.619 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:96fe5c49:::benchmark_data_smithi029_108893_object108293:head# 2024-03-24T12:48:51.619 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97054352:::benchmark_data_smithi029_108893_object109925:head# 2024-03-24T12:48:51.619 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:971216cb:::benchmark_data_smithi029_108893_object109364:head# 2024-03-24T12:48:51.620 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:971224d0:::benchmark_data_smithi029_108893_object95435:head# 2024-03-24T12:48:51.620 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9712427b:::benchmark_data_smithi029_108893_object105183:head# 2024-03-24T12:48:51.621 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:971314d3:::benchmark_data_smithi029_108893_object113525:head# 2024-03-24T12:48:51.622 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9714bcd9:::benchmark_data_smithi029_108893_object99347:head# 2024-03-24T12:48:51.622 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:971d1b86:::benchmark_data_smithi029_108893_object107206:head# 2024-03-24T12:48:51.623 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:972163f8:::benchmark_data_smithi029_108893_object97205:head# 2024-03-24T12:48:51.623 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9729b9e6:::benchmark_data_smithi029_108893_object107441:head# 2024-03-24T12:48:51.624 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:972f6a5c:::benchmark_data_smithi029_108893_object94741:head# 2024-03-24T12:48:51.624 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97336909:::benchmark_data_smithi029_108893_object101084:head# 2024-03-24T12:48:51.627 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:973731f3:::benchmark_data_smithi029_108893_object105970:head# 2024-03-24T12:48:51.627 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9737c572:::benchmark_data_smithi029_108893_object103496:head# 2024-03-24T12:48:51.628 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9738c959:::benchmark_data_smithi029_108893_object113262:head# 2024-03-24T12:48:51.631 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:973ae2e8:::benchmark_data_smithi029_108893_object102785:head# 2024-03-24T12:48:51.631 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9743a8e9:::benchmark_data_smithi029_108893_object97776:head# 2024-03-24T12:48:51.631 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97459eb4:::benchmark_data_smithi029_108893_object113528:head# 2024-03-24T12:48:51.632 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97498ac1:::benchmark_data_smithi029_108893_object112219:head# 2024-03-24T12:48:51.632 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:974a6f1a:::benchmark_data_smithi029_108893_object107430:head# 2024-03-24T12:48:51.633 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:974a789c:::benchmark_data_smithi029_108893_object111863:head# 2024-03-24T12:48:51.637 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:974aeca8:::benchmark_data_smithi029_108893_object98627:head# 2024-03-24T12:48:51.637 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97521895:::benchmark_data_smithi029_108893_object105218:head# 2024-03-24T12:48:51.638 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:975b6d68:::benchmark_data_smithi029_108893_object112786:head# 2024-03-24T12:48:51.638 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:975cb605:::benchmark_data_smithi029_108893_object105228:head# 2024-03-24T12:48:51.638 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:975ed77b:::benchmark_data_smithi029_108893_object111738:head# 2024-03-24T12:48:51.638 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9760d1b5:::benchmark_data_smithi029_108893_object106358:head# 2024-03-24T12:48:51.639 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9764d5b1:::benchmark_data_smithi029_108893_object101309:head# 2024-03-24T12:48:51.639 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9768ac33:::benchmark_data_smithi029_108893_object104549:head# 2024-03-24T12:48:51.639 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:976b3040:::benchmark_data_smithi029_108893_object94352:head# 2024-03-24T12:48:51.640 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:976cb110:::benchmark_data_smithi029_108893_object104303:head# 2024-03-24T12:48:51.640 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:977123f1:::benchmark_data_smithi029_108893_object101400:head# 2024-03-24T12:48:51.640 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97717293:::benchmark_data_smithi029_108893_object99215:head# 2024-03-24T12:48:51.640 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9778eefc:::benchmark_data_smithi029_108893_object95377:head# 2024-03-24T12:48:51.641 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:977edecd:::benchmark_data_smithi029_108893_object106162:head# 2024-03-24T12:48:51.645 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9781d325:::benchmark_data_smithi029_108893_object107675:head# 2024-03-24T12:48:51.645 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97821152:::benchmark_data_smithi029_108893_object108848:head# 2024-03-24T12:48:51.646 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97829e75:::benchmark_data_smithi029_108893_object95662:head# 2024-03-24T12:48:51.646 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97879bc1:::benchmark_data_smithi029_108893_object97253:head# 2024-03-24T12:48:51.647 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:978dfdfd:::benchmark_data_smithi029_108893_object112081:head# 2024-03-24T12:48:51.647 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97908ef7:::benchmark_data_smithi029_108893_object98717:head# 2024-03-24T12:48:51.648 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:51.649 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9791c544:::benchmark_data_smithi029_108893_object110516:head# 2024-03-24T12:48:51.649 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:979844db:::benchmark_data_smithi029_108893_object112767:head# 2024-03-24T12:48:51.649 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97993962:::benchmark_data_smithi029_108893_object101044:head# 2024-03-24T12:48:51.649 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:979c722c:::benchmark_data_smithi029_108893_object95262:head# 2024-03-24T12:48:51.651 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:979e348f:::benchmark_data_smithi029_108893_object94489:head# 2024-03-24T12:48:51.651 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97a02770:::benchmark_data_smithi029_108893_object107730:head# 2024-03-24T12:48:51.651 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97a30d21:::benchmark_data_smithi029_108893_object108855:head# 2024-03-24T12:48:51.652 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97a390b5:::benchmark_data_smithi029_108893_object100336:head# 2024-03-24T12:48:51.652 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97a4abb6:::benchmark_data_smithi029_108893_object101487:head# 2024-03-24T12:48:51.653 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97a57e1a:::benchmark_data_smithi029_108893_object94568:head# 2024-03-24T12:48:51.653 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97a820dc:::benchmark_data_smithi029_108893_object98049:head# 2024-03-24T12:48:51.653 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97b1a8aa:::benchmark_data_smithi029_108893_object94140:head# 2024-03-24T12:48:51.654 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97b3347b:::benchmark_data_smithi029_108893_object113307:head# 2024-03-24T12:48:51.655 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97b3f13d:::benchmark_data_smithi029_108893_object109656:head# 2024-03-24T12:48:51.655 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97b4bcf4:::benchmark_data_smithi029_108893_object105453:head# 2024-03-24T12:48:51.655 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97b53d2b:::benchmark_data_smithi029_108893_object106784:head# 2024-03-24T12:48:51.657 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97bb177b:::benchmark_data_smithi029_108893_object97474:head# 2024-03-24T12:48:51.657 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97c17803:::benchmark_data_smithi029_108893_object94871:head# 2024-03-24T12:48:51.657 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97c31582:::benchmark_data_smithi029_108893_object99118:head# 2024-03-24T12:48:51.658 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97c4a91d:::benchmark_data_smithi029_108893_object96005:head# 2024-03-24T12:48:51.659 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97c7554a:::benchmark_data_smithi029_108893_object110175:head# 2024-03-24T12:48:51.659 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97ccafab:::benchmark_data_smithi029_108893_object110010:head# 2024-03-24T12:48:51.660 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97cda196:::benchmark_data_smithi029_108893_object98749:head# 2024-03-24T12:48:51.660 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97d0ce23:::benchmark_data_smithi029_108893_object105795:head# 2024-03-24T12:48:51.661 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97d2dba7:::benchmark_data_smithi029_108893_object113255:head# 2024-03-24T12:48:51.662 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97ddcb67:::benchmark_data_smithi029_108893_object111187:head# 2024-03-24T12:48:51.662 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97de03de:::benchmark_data_smithi029_108893_object102482:head# 2024-03-24T12:48:51.662 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97debb85:::benchmark_data_smithi029_108893_object93208:head# 2024-03-24T12:48:51.663 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97dfdd48:::benchmark_data_smithi029_108893_object105626:head# 2024-03-24T12:48:51.663 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97e00d0a:::benchmark_data_smithi029_108893_object96818:head# 2024-03-24T12:48:51.664 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97e0c164:::benchmark_data_smithi029_108893_object103601:head# 2024-03-24T12:48:51.664 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97e0db41:::benchmark_data_smithi029_108893_object93850:head# 2024-03-24T12:48:51.665 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97e390e3:::benchmark_data_smithi029_108893_object112253:head# 2024-03-24T12:48:51.665 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97e783f3:::benchmark_data_smithi029_108893_object93770:head# 2024-03-24T12:48:51.666 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97e854b8:::benchmark_data_smithi029_108893_object101691:head# 2024-03-24T12:48:51.667 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97ee2051:::benchmark_data_smithi029_108893_object95540:head# 2024-03-24T12:48:51.667 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97efe3fa:::benchmark_data_smithi029_108893_object94822:head# 2024-03-24T12:48:51.667 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97f05771:::benchmark_data_smithi029_108893_object99737:head# 2024-03-24T12:48:51.668 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97f120d8:::benchmark_data_smithi029_108893_object109221:head# 2024-03-24T12:48:51.668 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97f1ef45:::benchmark_data_smithi029_108893_object107771:head# 2024-03-24T12:48:51.669 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97f49e99:::benchmark_data_smithi029_108893_object103450:head# 2024-03-24T12:48:51.669 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97f49eb1:::benchmark_data_smithi029_108893_object106502:head# 2024-03-24T12:48:51.670 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97f539a7:::benchmark_data_smithi029_108893_object112879:head# 2024-03-24T12:48:51.670 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97f66b42:::benchmark_data_smithi029_108893_object109521:head# 2024-03-24T12:48:51.670 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97f890a2:::benchmark_data_smithi029_108893_object99052:head# 2024-03-24T12:48:51.671 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97f9d78f:::benchmark_data_smithi029_108893_object98668:head# 2024-03-24T12:48:51.671 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97fe4669:::benchmark_data_smithi029_108893_object97809:head# 2024-03-24T12:48:51.672 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:97fe5ce5:::benchmark_data_smithi029_108893_object106006:head# 2024-03-24T12:48:51.672 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:980547de:::benchmark_data_smithi029_108893_object93079:head# 2024-03-24T12:48:51.673 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98075d8a:::benchmark_data_smithi029_108893_object93247:head# 2024-03-24T12:48:51.673 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:980b02cb:::benchmark_data_smithi029_108893_object106005:head# 2024-03-24T12:48:51.674 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:980d6835:::benchmark_data_smithi029_108893_object106519:head# 2024-03-24T12:48:51.675 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:981132bd:::benchmark_data_smithi029_108893_object92902:head# 2024-03-24T12:48:51.675 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98114e26:::benchmark_data_smithi029_108893_object103829:head# 2024-03-24T12:48:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9811f057:::benchmark_data_smithi029_108893_object113806:head# 2024-03-24T12:48:51.677 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9813d0bd:::benchmark_data_smithi029_108893_object110352:head# 2024-03-24T12:48:51.678 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:981560c9:::benchmark_data_smithi029_108893_object102480:head# 2024-03-24T12:48:51.678 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98162db3:::benchmark_data_smithi029_108893_object100422:head# 2024-03-24T12:48:51.678 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:51.679 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9817a57f:::benchmark_data_smithi029_108893_object105687:head# 2024-03-24T12:48:51.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:51.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:51.681 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:981de391:::benchmark_data_smithi029_108893_object108879:head# 2024-03-24T12:48:51.681 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:981ef1e0:::benchmark_data_smithi029_108893_object104474:head# 2024-03-24T12:48:51.681 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:981f51ee:::benchmark_data_smithi029_108893_object102675:head# 2024-03-24T12:48:51.681 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:981f5d99:::benchmark_data_smithi029_108893_object98414:head# 2024-03-24T12:48:51.681 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9821611e:::benchmark_data_smithi029_108893_object102740:head# 2024-03-24T12:48:51.681 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:982eb312:::benchmark_data_smithi029_108893_object92819:head# 2024-03-24T12:48:51.682 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:982f6ba5:::benchmark_data_smithi029_108893_object92771:head# 2024-03-24T12:48:51.682 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98304549:::benchmark_data_smithi029_108893_object101513:head# 2024-03-24T12:48:51.683 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98309183:::benchmark_data_smithi029_108893_object102405:head# 2024-03-24T12:48:51.684 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:983637b7:::benchmark_data_smithi029_108893_object105607:head# 2024-03-24T12:48:51.684 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:983a6e25:::benchmark_data_smithi029_108893_object99563:head# 2024-03-24T12:48:51.684 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9840aa8e:::benchmark_data_smithi029_108893_object112802:head# 2024-03-24T12:48:51.685 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9840ebc7:::benchmark_data_smithi029_108893_object103775:head# 2024-03-24T12:48:51.685 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98417a13:::benchmark_data_smithi029_108893_object101701:head# 2024-03-24T12:48:51.686 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9842eaa5:::benchmark_data_smithi029_108893_object95476:head# 2024-03-24T12:48:51.686 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:984386c8:::benchmark_data_smithi029_108893_object92815:head# 2024-03-24T12:48:51.686 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9847bafc:::benchmark_data_smithi029_108893_object94764:head# 2024-03-24T12:48:51.687 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:984d0d30:::benchmark_data_smithi029_108893_object102068:head# 2024-03-24T12:48:51.687 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:984e76d0:::benchmark_data_smithi029_108893_object98222:head# 2024-03-24T12:48:51.688 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98578cac:::benchmark_data_smithi029_108893_object106284:head# 2024-03-24T12:48:51.688 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:985a48e8:::benchmark_data_smithi029_108893_object113276:head# 2024-03-24T12:48:51.689 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:985bba76:::benchmark_data_smithi029_108893_object93256:head# 2024-03-24T12:48:51.689 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9868d60d:::benchmark_data_smithi029_108893_object100965:head# 2024-03-24T12:48:51.690 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:986a3b97:::benchmark_data_smithi029_108893_object93141:head# 2024-03-24T12:48:51.691 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:986cdb3b:::benchmark_data_smithi029_108893_object99785:head# 2024-03-24T12:48:51.691 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9874e167:::benchmark_data_smithi029_108893_object104615:head# 2024-03-24T12:48:51.692 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9879cea7:::benchmark_data_smithi029_108893_object95994:head# 2024-03-24T12:48:51.692 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:987e1fac:::benchmark_data_smithi029_108893_object94658:head# 2024-03-24T12:48:51.693 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9885cd4b:::benchmark_data_smithi029_108893_object94425:head# 2024-03-24T12:48:51.693 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98871339:::benchmark_data_smithi029_108893_object95208:head# 2024-03-24T12:48:51.694 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98874b98:::benchmark_data_smithi029_108893_object107907:head# 2024-03-24T12:48:51.694 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:988b24a0:::benchmark_data_smithi029_108893_object97664:head# 2024-03-24T12:48:51.694 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:988c38c4:::benchmark_data_smithi029_108893_object112838:head# 2024-03-24T12:48:51.695 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:988c60e4:::benchmark_data_smithi029_108893_object93480:head# 2024-03-24T12:48:51.698 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98947251:::benchmark_data_smithi029_108893_object113709:head# 2024-03-24T12:48:51.698 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98952394:::benchmark_data_smithi029_108893_object107471:head# 2024-03-24T12:48:51.700 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:989636ed:::benchmark_data_smithi029_108893_object95044:head# 2024-03-24T12:48:51.701 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98991764:::benchmark_data_smithi029_108893_object110186:head# 2024-03-24T12:48:51.702 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:989a9b35:::benchmark_data_smithi029_108893_object100913:head# 2024-03-24T12:48:51.702 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:989c00c6:::benchmark_data_smithi029_108893_object94664:head# 2024-03-24T12:48:51.703 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98a1124f:::benchmark_data_smithi029_108893_object103404:head# 2024-03-24T12:48:51.703 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98aa1664:::benchmark_data_smithi029_108893_object111579:head# 2024-03-24T12:48:51.704 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98b557b1:::benchmark_data_smithi029_108893_object110465:head# 2024-03-24T12:48:51.704 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98b7d2b3:::benchmark_data_smithi029_108893_object102845:head# 2024-03-24T12:48:51.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98b852fb:::benchmark_data_smithi029_108893_object97525:head# 2024-03-24T12:48:51.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98b85f98:::benchmark_data_smithi029_108893_object103183:head# 2024-03-24T12:48:51.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98bfb852:::benchmark_data_smithi029_108893_object104137:head# 2024-03-24T12:48:51.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98c23448:::benchmark_data_smithi029_108893_object107577:head# 2024-03-24T12:48:51.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98c3147f:::benchmark_data_smithi029_108893_object92956:head# 2024-03-24T12:48:51.711 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98c517ee:::benchmark_data_smithi029_108893_object109662:head# 2024-03-24T12:48:51.712 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98c9ad58:::benchmark_data_smithi029_108893_object105352:head# 2024-03-24T12:48:51.712 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98d0146e:::benchmark_data_smithi029_108893_object95402:head# 2024-03-24T12:48:51.712 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98d18601:::benchmark_data_smithi029_108893_object109545:head# 2024-03-24T12:48:51.712 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98d19c6e:::benchmark_data_smithi029_108893_object99771:head# 2024-03-24T12:48:51.713 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98d42458:::benchmark_data_smithi029_108893_object106911:head# 2024-03-24T12:48:51.713 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98d65f6c:::benchmark_data_smithi029_108893_object97005:head# 2024-03-24T12:48:51.713 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98dadfe0:::benchmark_data_smithi029_108893_object97551:head# 2024-03-24T12:48:51.713 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98db691c:::benchmark_data_smithi029_108893_object98646:head# 2024-03-24T12:48:51.714 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98df98bb:::benchmark_data_smithi029_108893_object104025:head# 2024-03-24T12:48:51.714 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98f310e0:::benchmark_data_smithi029_108893_object105562:head# 2024-03-24T12:48:51.714 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98f37da3:::benchmark_data_smithi029_108893_object109926:head# 2024-03-24T12:48:51.715 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98f8adac:::benchmark_data_smithi029_108893_object101687:head# 2024-03-24T12:48:51.715 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:98fa2b88:::benchmark_data_smithi029_108893_object101731:head# 2024-03-24T12:48:51.716 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99048e3a:::benchmark_data_smithi029_108893_object112987:head# 2024-03-24T12:48:51.716 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9904e447:::benchmark_data_smithi029_108893_object110192:head# 2024-03-24T12:48:51.719 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:990a2e68:::benchmark_data_smithi029_108893_object112780:head# 2024-03-24T12:48:51.720 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:990ed0d6:::benchmark_data_smithi029_108893_object93058:head# 2024-03-24T12:48:51.720 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:990f6a70:::benchmark_data_smithi029_108893_object103454:head# 2024-03-24T12:48:51.720 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99110421:::benchmark_data_smithi029_108893_object94870:head# 2024-03-24T12:48:51.721 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9916e6e4:::benchmark_data_smithi029_108893_object108643:head# 2024-03-24T12:48:51.721 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9917a51c:::benchmark_data_smithi029_108893_object96086:head# 2024-03-24T12:48:51.723 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:991f8832:::benchmark_data_smithi029_108893_object110916:head# 2024-03-24T12:48:51.723 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9925f21e:::benchmark_data_smithi029_108893_object107343:head# 2024-03-24T12:48:51.724 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99279c8b:::benchmark_data_smithi029_108893_object105937:head# 2024-03-24T12:48:51.724 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:992a5bea:::benchmark_data_smithi029_108893_object98878:head# 2024-03-24T12:48:51.725 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9933ad91:::benchmark_data_smithi029_108893_object99152:head# 2024-03-24T12:48:51.725 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:993a2f8c:::benchmark_data_smithi029_108893_object111838:head# 2024-03-24T12:48:51.725 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:993be375:::benchmark_data_smithi029_108893_object102536:head# 2024-03-24T12:48:51.726 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:993fd5ca:::benchmark_data_smithi029_108893_object112695:head# 2024-03-24T12:48:51.726 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99426faf:::benchmark_data_smithi029_108893_object110257:head# 2024-03-24T12:48:51.727 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9942d72f:::benchmark_data_smithi029_108893_object110869:head# 2024-03-24T12:48:51.727 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:994506c1:::benchmark_data_smithi029_108893_object102002:head# 2024-03-24T12:48:51.730 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9946ed14:::benchmark_data_smithi029_108893_object107977:head# 2024-03-24T12:48:51.731 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:994a2782:::benchmark_data_smithi029_108893_object112725:head# 2024-03-24T12:48:51.731 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:994a34ed:::benchmark_data_smithi029_108893_object102011:head# 2024-03-24T12:48:51.732 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:994f735c:::benchmark_data_smithi029_108893_object101202:head# 2024-03-24T12:48:51.733 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99514b57:::benchmark_data_smithi029_108893_object103020:head# 2024-03-24T12:48:51.733 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99527896:::benchmark_data_smithi029_108893_object110256:head# 2024-03-24T12:48:51.734 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9954c8b4:::benchmark_data_smithi029_108893_object93919:head# 2024-03-24T12:48:51.735 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9955027d:::benchmark_data_smithi029_108893_object111932:head# 2024-03-24T12:48:51.735 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:995508ce:::benchmark_data_smithi029_108893_object104848:head# 2024-03-24T12:48:51.736 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:995c6687:::benchmark_data_smithi029_108893_object96048:head# 2024-03-24T12:48:51.736 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99602e02:::benchmark_data_smithi029_108893_object113213:head# 2024-03-24T12:48:51.737 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99604a29:::benchmark_data_smithi029_108893_object106098:head# 2024-03-24T12:48:51.737 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9961454f:::benchmark_data_smithi029_108893_object104905:head# 2024-03-24T12:48:51.739 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9966c2c9:::benchmark_data_smithi029_108893_object106122:head# 2024-03-24T12:48:51.741 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9967b97a:::benchmark_data_smithi029_108893_object112759:head# 2024-03-24T12:48:51.741 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9969132a:::benchmark_data_smithi029_108893_object107779:head# 2024-03-24T12:48:51.742 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:996b0d8b:::benchmark_data_smithi029_108893_object93798:head# 2024-03-24T12:48:51.742 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9971f1de:::benchmark_data_smithi029_108893_object99201:head# 2024-03-24T12:48:51.743 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:997374f0:::benchmark_data_smithi029_108893_object99561:head# 2024-03-24T12:48:51.743 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:997383fd:::benchmark_data_smithi029_108893_object95829:head# 2024-03-24T12:48:51.744 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99745ea7:::benchmark_data_smithi029_108893_object103703:head# 2024-03-24T12:48:51.745 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99774bbf:::benchmark_data_smithi029_108893_object102334:head# 2024-03-24T12:48:51.747 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:997a1bdb:::benchmark_data_smithi029_108893_object101857:head# 2024-03-24T12:48:51.747 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:997e3dba:::benchmark_data_smithi029_108893_object105131:head# 2024-03-24T12:48:51.750 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9983167f:::benchmark_data_smithi029_108893_object103029:head# 2024-03-24T12:48:51.751 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99847ae2:::benchmark_data_smithi029_108893_object97420:head# 2024-03-24T12:48:51.753 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9985e1b9:::benchmark_data_smithi029_108893_object98932:head# 2024-03-24T12:48:51.753 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9985e48c:::benchmark_data_smithi029_108893_object105383:head# 2024-03-24T12:48:51.754 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99877736:::benchmark_data_smithi029_108893_object94725:head# 2024-03-24T12:48:51.754 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9987b33f:::benchmark_data_smithi029_108893_object95771:head# 2024-03-24T12:48:51.756 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:998bb3c8:::benchmark_data_smithi029_108893_object104260:head# 2024-03-24T12:48:51.756 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99915994:::benchmark_data_smithi029_108893_object112283:head# 2024-03-24T12:48:51.757 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9992f85c:::benchmark_data_smithi029_108893_object92713:head# 2024-03-24T12:48:51.758 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9995385d:::benchmark_data_smithi029_108893_object98021:head# 2024-03-24T12:48:51.759 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9998d5df:::benchmark_data_smithi029_108893_object104714:head# 2024-03-24T12:48:51.759 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99996429:::benchmark_data_smithi029_108893_object107451:head# 2024-03-24T12:48:51.760 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99a5f429:::benchmark_data_smithi029_108893_object98601:head# 2024-03-24T12:48:51.760 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99ab14d6:::benchmark_data_smithi029_108893_object111937:head# 2024-03-24T12:48:51.761 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99b22302:::benchmark_data_smithi029_108893_object97053:head# 2024-03-24T12:48:51.761 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99b8665a:::benchmark_data_smithi029_108893_object95268:head# 2024-03-24T12:48:51.762 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99b894cf:::benchmark_data_smithi029_108893_object95093:head# 2024-03-24T12:48:51.762 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99bec825:::benchmark_data_smithi029_108893_object111275:head# 2024-03-24T12:48:51.763 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99bf4393:::benchmark_data_smithi029_108893_object97550:head# 2024-03-24T12:48:51.767 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99c15abb:::benchmark_data_smithi029_108893_object93062:head# 2024-03-24T12:48:51.767 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99d34dcf:::benchmark_data_smithi029_108893_object94784:head# 2024-03-24T12:48:51.767 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99d62001:::benchmark_data_smithi029_108893_object105484:head# 2024-03-24T12:48:51.767 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99d767b4:::benchmark_data_smithi029_108893_object101943:head# 2024-03-24T12:48:51.767 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99da438c:::benchmark_data_smithi029_108893_object96780:head# 2024-03-24T12:48:51.768 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99dd5111:::benchmark_data_smithi029_108893_object108813:head# 2024-03-24T12:48:51.768 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99e10726:::benchmark_data_smithi029_108893_object111784:head# 2024-03-24T12:48:51.768 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99eaee4e:::benchmark_data_smithi029_108893_object110467:head# 2024-03-24T12:48:51.769 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99ef7aa4:::benchmark_data_smithi029_108893_object105044:head# 2024-03-24T12:48:51.769 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99f06970:::benchmark_data_smithi029_108893_object101337:head# 2024-03-24T12:48:51.770 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99f28857:::benchmark_data_smithi029_108893_object98081:head# 2024-03-24T12:48:51.770 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99f4f7b1:::benchmark_data_smithi029_108893_object102296:head# 2024-03-24T12:48:51.771 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99fccc7d:::benchmark_data_smithi029_108893_object103088:head# 2024-03-24T12:48:51.771 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:99fed363:::benchmark_data_smithi029_108893_object96104:head# 2024-03-24T12:48:51.772 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a01fafd:::benchmark_data_smithi029_108893_object108616:head# 2024-03-24T12:48:51.772 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a043809:::benchmark_data_smithi029_108893_object96901:head# 2024-03-24T12:48:51.773 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a0669e7:::benchmark_data_smithi029_108893_object99310:head# 2024-03-24T12:48:51.773 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a085fae:::benchmark_data_smithi029_108893_object102612:head# 2024-03-24T12:48:51.773 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a0ac9fc:::benchmark_data_smithi029_108893_object94670:head# 2024-03-24T12:48:51.774 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a0bec75:::benchmark_data_smithi029_108893_object107967:head# 2024-03-24T12:48:51.774 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a18bee8:::benchmark_data_smithi029_108893_object97850:head# 2024-03-24T12:48:51.775 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a1ccc81:::benchmark_data_smithi029_108893_object96541:head# 2024-03-24T12:48:51.775 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a21af8f:::benchmark_data_smithi029_108893_object105789:head# 2024-03-24T12:48:51.776 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a243c0d:::benchmark_data_smithi029_108893_object107349:head# 2024-03-24T12:48:51.776 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a28a00b:::benchmark_data_smithi029_108893_object100471:head# 2024-03-24T12:48:51.777 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a2a6534:::benchmark_data_smithi029_108893_object113330:head# 2024-03-24T12:48:51.777 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a2bb749:::benchmark_data_smithi029_108893_object108331:head# 2024-03-24T12:48:51.777 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a2d677a:::benchmark_data_smithi029_108893_object106255:head# 2024-03-24T12:48:51.778 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a2e2c5b:::benchmark_data_smithi029_108893_object110866:head# 2024-03-24T12:48:51.778 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a2eb724:::benchmark_data_smithi029_108893_object103780:head# 2024-03-24T12:48:51.779 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a313882:::benchmark_data_smithi029_108893_object103157:head# 2024-03-24T12:48:51.779 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a334cc8:::benchmark_data_smithi029_108893_object106415:head# 2024-03-24T12:48:51.780 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a372682:::benchmark_data_smithi029_108893_object99345:head# 2024-03-24T12:48:51.780 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:51.781 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a3ad9a0:::benchmark_data_smithi029_108893_object94552:head# 2024-03-24T12:48:51.781 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a3d657d:::benchmark_data_smithi029_108893_object113291:head# 2024-03-24T12:48:51.784 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a482c6c:::benchmark_data_smithi029_108893_object96593:head# 2024-03-24T12:48:51.784 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a49faf0:::benchmark_data_smithi029_108893_object104251:head# 2024-03-24T12:48:51.784 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a4dbad8:::benchmark_data_smithi029_108893_object95404:head# 2024-03-24T12:48:51.784 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a4ed4ea:::benchmark_data_smithi029_108893_object94010:head# 2024-03-24T12:48:51.784 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a4f5f7b:::benchmark_data_smithi029_108893_object106698:head# 2024-03-24T12:48:51.785 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a50205f:::benchmark_data_smithi029_108893_object103886:head# 2024-03-24T12:48:51.785 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a54bfc9:::benchmark_data_smithi029_108893_object96330:head# 2024-03-24T12:48:51.786 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a5535b9:::benchmark_data_smithi029_108893_object100067:head# 2024-03-24T12:48:51.786 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a5812f4:::benchmark_data_smithi029_108893_object97252:head# 2024-03-24T12:48:51.787 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a629acc:::benchmark_data_smithi029_108893_object93134:head# 2024-03-24T12:48:51.788 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a6a1e9d:::benchmark_data_smithi029_108893_object112877:head# 2024-03-24T12:48:51.788 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a6a8646:::benchmark_data_smithi029_108893_object93081:head# 2024-03-24T12:48:51.789 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a6d77b6:::benchmark_data_smithi029_108893_object97644:head# 2024-03-24T12:48:51.789 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a6e5ddf:::benchmark_data_smithi029_108893_object104575:head# 2024-03-24T12:48:51.790 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a6e8fd1:::benchmark_data_smithi029_108893_object111228:head# 2024-03-24T12:48:51.790 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a7066e9:::benchmark_data_smithi029_108893_object102792:head# 2024-03-24T12:48:51.791 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a70c7bc:::benchmark_data_smithi029_108893_object113670:head# 2024-03-24T12:48:51.791 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a7105ed:::benchmark_data_smithi029_108893_object109072:head# 2024-03-24T12:48:51.792 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a74e091:::benchmark_data_smithi029_108893_object98258:head# 2024-03-24T12:48:51.792 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a75d554:::benchmark_data_smithi029_108893_object93368:head# 2024-03-24T12:48:51.793 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a78d13a:::benchmark_data_smithi029_108893_object106325:head# 2024-03-24T12:48:51.793 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a7947d5:::benchmark_data_smithi029_108893_object96115:head# 2024-03-24T12:48:51.794 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a7fbd1c:::benchmark_data_smithi029_108893_object99252:head# 2024-03-24T12:48:51.795 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a83348a:::benchmark_data_smithi029_108893_object105942:head# 2024-03-24T12:48:51.795 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a847dba:::benchmark_data_smithi029_108893_object98092:head# 2024-03-24T12:48:51.795 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a886d13:::benchmark_data_smithi029_108893_object94468:head# 2024-03-24T12:48:51.796 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a8d9077:::benchmark_data_smithi029_108893_object93586:head# 2024-03-24T12:48:51.796 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a8f0e16:::benchmark_data_smithi029_108893_object95834:head# 2024-03-24T12:48:51.797 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a971d6d:::benchmark_data_smithi029_108893_object101802:head# 2024-03-24T12:48:51.797 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a98c1b2:::benchmark_data_smithi029_108893_object105071:head# 2024-03-24T12:48:51.798 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a9c3284:::benchmark_data_smithi029_108893_object111004:head# 2024-03-24T12:48:51.799 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9a9e743b:::benchmark_data_smithi029_108893_object112109:head# 2024-03-24T12:48:51.801 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9aaa6920:::benchmark_data_smithi029_108893_object107920:head# 2024-03-24T12:48:51.801 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ab68d8c:::benchmark_data_smithi029_108893_object111322:head# 2024-03-24T12:48:51.801 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ab859e5:::benchmark_data_smithi029_108893_object113509:head# 2024-03-24T12:48:51.802 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9aba94c1:::benchmark_data_smithi029_108893_object94471:head# 2024-03-24T12:48:51.802 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9abc0c61:::benchmark_data_smithi029_108893_object107264:head# 2024-03-24T12:48:51.803 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9abe2fe8:::benchmark_data_smithi029_108893_object99426:head# 2024-03-24T12:48:51.803 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9abf5bec:::benchmark_data_smithi029_108893_object95436:head# 2024-03-24T12:48:51.804 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9abfdb29:::benchmark_data_smithi029_108893_object98842:head# 2024-03-24T12:48:51.804 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ac31ec2:::benchmark_data_smithi029_108893_object102496:head# 2024-03-24T12:48:51.805 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ac43bf5:::benchmark_data_smithi029_108893_object103542:head# 2024-03-24T12:48:51.805 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ac82f11:::benchmark_data_smithi029_108893_object102706:head# 2024-03-24T12:48:51.806 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9acb5d2c:::benchmark_data_smithi029_108893_object104168:head# 2024-03-24T12:48:51.806 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9acce288:::benchmark_data_smithi029_108893_object98611:head# 2024-03-24T12:48:51.806 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ace567e:::benchmark_data_smithi029_108893_object101367:head# 2024-03-24T12:48:51.807 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9acfae71:::benchmark_data_smithi029_108893_object113259:head# 2024-03-24T12:48:51.808 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ad10779:::benchmark_data_smithi029_108893_object104554:head# 2024-03-24T12:48:51.808 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ad64d85:::benchmark_data_smithi029_108893_object105159:head# 2024-03-24T12:48:51.809 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ad6a30d:::benchmark_data_smithi029_108893_object98633:head# 2024-03-24T12:48:51.809 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ad9cc59:::benchmark_data_smithi029_108893_object108254:head# 2024-03-24T12:48:51.810 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:51.810 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ada79fe:::benchmark_data_smithi029_108893_object103933:head# 2024-03-24T12:48:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:51.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:51.812 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9add4cfd:::benchmark_data_smithi029_108893_object99720:head# 2024-03-24T12:48:51.812 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ae130c4:::benchmark_data_smithi029_108893_object99747:head# 2024-03-24T12:48:51.812 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ae53c66:::benchmark_data_smithi029_108893_object95759:head# 2024-03-24T12:48:51.812 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ae8bf46:::benchmark_data_smithi029_108893_object106856:head# 2024-03-24T12:48:51.813 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9aea1409:::benchmark_data_smithi029_108893_object92945:head# 2024-03-24T12:48:51.813 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9aea7525:::benchmark_data_smithi029_108893_object105930:head# 2024-03-24T12:48:51.813 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9aec1a19:::benchmark_data_smithi029_108893_object93281:head# 2024-03-24T12:48:51.814 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9af2046a:::benchmark_data_smithi029_108893_object99527:head# 2024-03-24T12:48:51.814 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9af6e2df:::benchmark_data_smithi029_108893_object112692:head# 2024-03-24T12:48:51.815 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9afb6cea:::benchmark_data_smithi029_108893_object111714:head# 2024-03-24T12:48:51.815 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9afc5ab7:::benchmark_data_smithi029_108893_object102382:head# 2024-03-24T12:48:51.815 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9afef720:::benchmark_data_smithi029_108893_object108851:head# 2024-03-24T12:48:51.816 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b012c35:::benchmark_data_smithi029_108893_object106596:head# 2024-03-24T12:48:51.816 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b014805:::benchmark_data_smithi029_108893_object102903:head# 2024-03-24T12:48:51.817 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b024601:::benchmark_data_smithi029_108893_object110706:head# 2024-03-24T12:48:51.817 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b025592:::benchmark_data_smithi029_108893_object112645:head# 2024-03-24T12:48:51.818 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b04f807:::benchmark_data_smithi029_108893_object94797:head# 2024-03-24T12:48:51.818 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b05596f:::benchmark_data_smithi029_108893_object100532:head# 2024-03-24T12:48:51.819 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b08dec1:::benchmark_data_smithi029_108893_object106728:head# 2024-03-24T12:48:51.820 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b09a32d:::benchmark_data_smithi029_108893_object99977:head# 2024-03-24T12:48:51.820 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b0af71f:::benchmark_data_smithi029_108893_object104519:head# 2024-03-24T12:48:51.821 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b0be935:::benchmark_data_smithi029_108893_object94044:head# 2024-03-24T12:48:51.822 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b10f2d1:::benchmark_data_smithi029_108893_object110985:head# 2024-03-24T12:48:51.829 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b16b597:::benchmark_data_smithi029_108893_object111151:head# 2024-03-24T12:48:51.829 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b1c58a6:::benchmark_data_smithi029_108893_object109368:head# 2024-03-24T12:48:51.830 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b1cc9a5:::benchmark_data_smithi029_108893_object107079:head# 2024-03-24T12:48:51.830 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b1e60ca:::benchmark_data_smithi029_108893_object104445:head# 2024-03-24T12:48:51.830 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b2071a2:::benchmark_data_smithi029_108893_object94891:head# 2024-03-24T12:48:51.831 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b21c5ec:::benchmark_data_smithi029_108893_object103618:head# 2024-03-24T12:48:51.831 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b239c38:::benchmark_data_smithi029_108893_object108151:head# 2024-03-24T12:48:51.832 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b28787a:::benchmark_data_smithi029_108893_object102900:head# 2024-03-24T12:48:51.832 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b29b3dd:::benchmark_data_smithi029_108893_object107258:head# 2024-03-24T12:48:51.833 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b2ba856:::benchmark_data_smithi029_108893_object102425:head# 2024-03-24T12:48:51.833 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b32eaa1:::benchmark_data_smithi029_108893_object109869:head# 2024-03-24T12:48:51.834 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b35831d:::benchmark_data_smithi029_108893_object95084:head# 2024-03-24T12:48:51.835 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b41ade0:::benchmark_data_smithi029_108893_object102525:head# 2024-03-24T12:48:51.835 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b4295f1:::benchmark_data_smithi029_108893_object110053:head# 2024-03-24T12:48:51.836 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b44435d:::benchmark_data_smithi029_108893_object109293:head# 2024-03-24T12:48:51.837 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b46bfef:::benchmark_data_smithi029_108893_object106193:head# 2024-03-24T12:48:51.837 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b499ea5:::benchmark_data_smithi029_108893_object104367:head# 2024-03-24T12:48:51.838 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b4fe17c:::benchmark_data_smithi029_108893_object101830:head# 2024-03-24T12:48:51.838 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b501f8a:::benchmark_data_smithi029_108893_object103262:head# 2024-03-24T12:48:51.839 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b5393e5:::benchmark_data_smithi029_108893_object99348:head# 2024-03-24T12:48:51.840 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b5401b9:::benchmark_data_smithi029_108893_object111193:head# 2024-03-24T12:48:51.840 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b5746e4:::benchmark_data_smithi029_108893_object109074:head# 2024-03-24T12:48:51.840 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b5a366e:::benchmark_data_smithi029_108893_object98636:head# 2024-03-24T12:48:51.840 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b5a6441:::benchmark_data_smithi029_108893_object107602:head# 2024-03-24T12:48:51.841 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b5b0474:::benchmark_data_smithi029_108893_object99827:head# 2024-03-24T12:48:51.841 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b5b86be:::benchmark_data_smithi029_108893_object99138:head# 2024-03-24T12:48:51.842 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b6043ba:::benchmark_data_smithi029_108893_object93863:head# 2024-03-24T12:48:51.843 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b63dabc:::benchmark_data_smithi029_108893_object97059:head# 2024-03-24T12:48:51.843 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b662939:::benchmark_data_smithi029_108893_object93559:head# 2024-03-24T12:48:51.844 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b68652d:::benchmark_data_smithi029_108893_object95439:head# 2024-03-24T12:48:51.845 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b6d38d4:::benchmark_data_smithi029_108893_object96757:head# 2024-03-24T12:48:51.845 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b6d9317:::benchmark_data_smithi029_108893_object100609:head# 2024-03-24T12:48:51.846 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b82af0e:::benchmark_data_smithi029_108893_object97713:head# 2024-03-24T12:48:51.846 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b831d6b:::benchmark_data_smithi029_108893_object98444:head# 2024-03-24T12:48:51.847 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b84c6a8:::benchmark_data_smithi029_108893_object109532:head# 2024-03-24T12:48:51.847 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b84ddb1:::benchmark_data_smithi029_108893_object92880:head# 2024-03-24T12:48:51.848 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b872c1c:::benchmark_data_smithi029_108893_object92738:head# 2024-03-24T12:48:51.848 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b90cf83:::benchmark_data_smithi029_108893_object99937:head# 2024-03-24T12:48:51.849 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b926124:::benchmark_data_smithi029_108893_object99342:head# 2024-03-24T12:48:51.850 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b944d41:::benchmark_data_smithi029_108893_object108804:head# 2024-03-24T12:48:51.850 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b946557:::benchmark_data_smithi029_108893_object100892:head# 2024-03-24T12:48:51.851 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b94e31c:::benchmark_data_smithi029_108893_object106077:head# 2024-03-24T12:48:51.852 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b95e456:::benchmark_data_smithi029_108893_object95854:head# 2024-03-24T12:48:51.852 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b9896e4:::benchmark_data_smithi029_108893_object103801:head# 2024-03-24T12:48:51.853 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b98d0dc:::benchmark_data_smithi029_108893_object110396:head# 2024-03-24T12:48:51.854 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b99e818:::benchmark_data_smithi029_108893_object94913:head# 2024-03-24T12:48:51.854 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b9bd9a3:::benchmark_data_smithi029_108893_object111031:head# 2024-03-24T12:48:51.855 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b9d30e0:::benchmark_data_smithi029_108893_object101015:head# 2024-03-24T12:48:51.855 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b9dce94:::benchmark_data_smithi029_108893_object93843:head# 2024-03-24T12:48:51.856 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b9e1103:::benchmark_data_smithi029_108893_object108245:head# 2024-03-24T12:48:51.856 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9b9f99ae:::benchmark_data_smithi029_108893_object103055:head# 2024-03-24T12:48:51.857 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ba0b23b:::benchmark_data_smithi029_108893_object107016:head# 2024-03-24T12:48:51.859 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ba5c517:::benchmark_data_smithi029_108893_object98898:head# 2024-03-24T12:48:51.859 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ba654ea:::benchmark_data_smithi029_108893_object98929:head# 2024-03-24T12:48:51.859 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bac1aca:::benchmark_data_smithi029_108893_object94369:head# 2024-03-24T12:48:51.859 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bb2e755:::benchmark_data_smithi029_108893_object102336:head# 2024-03-24T12:48:51.860 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bb58c62:::benchmark_data_smithi029_108893_object102323:head# 2024-03-24T12:48:51.863 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bb80885:::benchmark_data_smithi029_108893_object103624:head# 2024-03-24T12:48:51.863 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bbf8d78:::benchmark_data_smithi029_108893_object113072:head# 2024-03-24T12:48:51.863 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bc7c5a6:::benchmark_data_smithi029_108893_object98622:head# 2024-03-24T12:48:51.864 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bcf1811:::benchmark_data_smithi029_108893_object97399:head# 2024-03-24T12:48:51.864 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bd42bb8:::benchmark_data_smithi029_108893_object100521:head# 2024-03-24T12:48:51.864 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bd5258b:::benchmark_data_smithi029_108893_object112059:head# 2024-03-24T12:48:51.866 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bd5e140:::benchmark_data_smithi029_108893_object93525:head# 2024-03-24T12:48:51.866 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bd8e751:::benchmark_data_smithi029_108893_object103682:head# 2024-03-24T12:48:51.869 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bd9d442:::benchmark_data_smithi029_108893_object109442:head# 2024-03-24T12:48:51.871 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bdb082c:::benchmark_data_smithi029_108893_object103457:head# 2024-03-24T12:48:51.871 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9be2ced2:::benchmark_data_smithi029_108893_object111642:head# 2024-03-24T12:48:51.872 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9be5c82d:::benchmark_data_smithi029_108893_object102124:head# 2024-03-24T12:48:51.874 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9be614ae:::benchmark_data_smithi029_108893_object104589:head# 2024-03-24T12:48:51.874 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bedef74:::benchmark_data_smithi029_108893_object108470:head# 2024-03-24T12:48:51.875 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9bfbf763:::benchmark_data_smithi029_108893_object101425:head# 2024-03-24T12:48:51.875 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c0104cb:::benchmark_data_smithi029_108893_object96921:head# 2024-03-24T12:48:51.876 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c02a809:::benchmark_data_smithi029_108893_object104790:head# 2024-03-24T12:48:51.877 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c0322f0:::benchmark_data_smithi029_108893_object112022:head# 2024-03-24T12:48:51.877 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c07ca97:::benchmark_data_smithi029_108893_object95601:head# 2024-03-24T12:48:51.881 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c0aaafb:::benchmark_data_smithi029_108893_object109683:head# 2024-03-24T12:48:51.881 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c0bc44f:::benchmark_data_smithi029_108893_object103305:head# 2024-03-24T12:48:51.882 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c0cc0e3:::benchmark_data_smithi029_108893_object106048:head# 2024-03-24T12:48:51.883 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c0d721a:::benchmark_data_smithi029_108893_object95158:head# 2024-03-24T12:48:51.884 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c12a795:::benchmark_data_smithi029_108893_object97358:head# 2024-03-24T12:48:51.884 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c144023:::benchmark_data_smithi029_108893_object103500:head# 2024-03-24T12:48:51.887 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c150cc3:::benchmark_data_smithi029_108893_object97948:head# 2024-03-24T12:48:51.887 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c155895:::benchmark_data_smithi029_108893_object113420:head# 2024-03-24T12:48:51.888 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c180234:::benchmark_data_smithi029_108893_object106765:head# 2024-03-24T12:48:51.891 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c197345:::benchmark_data_smithi029_108893_object103937:head# 2024-03-24T12:48:51.892 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c1edf3e:::benchmark_data_smithi029_108893_object102918:head# 2024-03-24T12:48:51.892 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c21244d:::benchmark_data_smithi029_108893_object113203:head# 2024-03-24T12:48:51.893 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c217f57:::benchmark_data_smithi029_108893_object105170:head# 2024-03-24T12:48:51.893 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c246e2e:::benchmark_data_smithi029_108893_object104262:head# 2024-03-24T12:48:51.895 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c24ca29:::benchmark_data_smithi029_108893_object103976:head# 2024-03-24T12:48:51.897 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c25c19e:::benchmark_data_smithi029_108893_object95729:head# 2024-03-24T12:48:51.897 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c2873c1:::benchmark_data_smithi029_108893_object108264:head# 2024-03-24T12:48:51.900 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c2a658a:::benchmark_data_smithi029_108893_object106875:head# 2024-03-24T12:48:51.900 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c2eba5b:::benchmark_data_smithi029_108893_object100205:head# 2024-03-24T12:48:51.901 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c378648:::benchmark_data_smithi029_108893_object99429:head# 2024-03-24T12:48:51.902 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c399114:::benchmark_data_smithi029_108893_object102619:head# 2024-03-24T12:48:51.902 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c39a422:::benchmark_data_smithi029_108893_object101849:head# 2024-03-24T12:48:51.905 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c426300:::benchmark_data_smithi029_108893_object97004:head# 2024-03-24T12:48:51.905 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c482eaf:::benchmark_data_smithi029_108893_object102274:head# 2024-03-24T12:48:51.906 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c48db0f:::benchmark_data_smithi029_108893_object96318:head# 2024-03-24T12:48:51.906 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c4a0642:::benchmark_data_smithi029_108893_object99418:head# 2024-03-24T12:48:51.907 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c547ddd:::benchmark_data_smithi029_108893_object106199:head# 2024-03-24T12:48:51.907 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c55a8f1:::benchmark_data_smithi029_108893_object100963:head# 2024-03-24T12:48:51.908 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c5d1fd0:::benchmark_data_smithi029_108893_object95808:head# 2024-03-24T12:48:51.908 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c5e37c2:::benchmark_data_smithi029_108893_object112691:head# 2024-03-24T12:48:51.909 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c6167df:::benchmark_data_smithi029_108893_object111299:head# 2024-03-24T12:48:51.910 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c618860:::benchmark_data_smithi029_108893_object98296:head# 2024-03-24T12:48:51.910 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c64d561:::benchmark_data_smithi029_108893_object105072:head# 2024-03-24T12:48:51.910 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c657dc5:::benchmark_data_smithi029_108893_object112716:head# 2024-03-24T12:48:51.911 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c68890f:::benchmark_data_smithi029_108893_object105215:head# 2024-03-24T12:48:51.911 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c6fe313:::benchmark_data_smithi029_108893_object104190:head# 2024-03-24T12:48:51.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:51.913 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c74eb4c:::benchmark_data_smithi029_108893_object103304:head# 2024-03-24T12:48:51.913 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c7dd17c:::benchmark_data_smithi029_108893_object95593:head# 2024-03-24T12:48:51.913 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c8173e5:::benchmark_data_smithi029_108893_object104248:head# 2024-03-24T12:48:51.914 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c81d4c1:::benchmark_data_smithi029_108893_object102318:head# 2024-03-24T12:48:51.914 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c86f55d:::benchmark_data_smithi029_108893_object103755:head# 2024-03-24T12:48:51.914 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c95fe5a:::benchmark_data_smithi029_108893_object100273:head# 2024-03-24T12:48:51.915 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c993714:::benchmark_data_smithi029_108893_object109909:head# 2024-03-24T12:48:51.916 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c9d22f8:::benchmark_data_smithi029_108893_object112508:head# 2024-03-24T12:48:51.916 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9c9e1a2c:::benchmark_data_smithi029_108893_object107729:head# 2024-03-24T12:48:51.916 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ca0e146:::benchmark_data_smithi029_108893_object106908:head# 2024-03-24T12:48:51.917 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ca2b8c1:::benchmark_data_smithi029_108893_object112097:head# 2024-03-24T12:48:51.917 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ca7495c:::benchmark_data_smithi029_108893_object97529:head# 2024-03-24T12:48:51.917 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ca75207:::benchmark_data_smithi029_108893_object108884:head# 2024-03-24T12:48:51.918 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ca905f9:::benchmark_data_smithi029_108893_object94839:head# 2024-03-24T12:48:51.921 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9caddcc3:::benchmark_data_smithi029_108893_object98048:head# 2024-03-24T12:48:51.921 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9caf6b64:::benchmark_data_smithi029_108893_object96112:head# 2024-03-24T12:48:51.921 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cafe3d9:::benchmark_data_smithi029_108893_object92720:head# 2024-03-24T12:48:51.922 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cb0dc7f:::benchmark_data_smithi029_108893_object93895:head# 2024-03-24T12:48:51.922 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cb32c6e:::benchmark_data_smithi029_108893_object108860:head# 2024-03-24T12:48:51.923 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cb3cb4d:::benchmark_data_smithi029_108893_object106511:head# 2024-03-24T12:48:51.923 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cb4f46f:::benchmark_data_smithi029_108893_object100833:head# 2024-03-24T12:48:51.924 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cb63e01:::benchmark_data_smithi029_108893_object100774:head# 2024-03-24T12:48:51.924 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cb66d88:::benchmark_data_smithi029_108893_object100911:head# 2024-03-24T12:48:51.924 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cb6a189:::benchmark_data_smithi029_108893_object95801:head# 2024-03-24T12:48:51.925 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cbb0487:::benchmark_data_smithi029_108893_object93649:head# 2024-03-24T12:48:51.926 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cc0bc03:::benchmark_data_smithi029_108893_object109955:head# 2024-03-24T12:48:51.926 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cc77923:::benchmark_data_smithi029_108893_object109695:head# 2024-03-24T12:48:51.926 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ccd05bf:::benchmark_data_smithi029_108893_object113302:head# 2024-03-24T12:48:51.928 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cce91df:::benchmark_data_smithi029_108893_object113041:head# 2024-03-24T12:48:51.929 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cd18a18:::benchmark_data_smithi029_108893_object93344:head# 2024-03-24T12:48:51.929 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cd91276:::benchmark_data_smithi029_108893_object101715:head# 2024-03-24T12:48:51.930 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cd9c52c:::benchmark_data_smithi029_108893_object109963:head# 2024-03-24T12:48:51.930 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cdba98e:::benchmark_data_smithi029_108893_object94105:head# 2024-03-24T12:48:51.931 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cdd249e:::benchmark_data_smithi029_108893_object101221:head# 2024-03-24T12:48:51.934 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cdfeee1:::benchmark_data_smithi029_108893_object111258:head# 2024-03-24T12:48:51.935 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ce00628:::benchmark_data_smithi029_108893_object95733:head# 2024-03-24T12:48:51.935 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ce2342e:::benchmark_data_smithi029_108893_object103508:head# 2024-03-24T12:48:51.936 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ce78956:::benchmark_data_smithi029_108893_object101262:head# 2024-03-24T12:48:51.936 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ce9a32c:::benchmark_data_smithi029_108893_object99318:head# 2024-03-24T12:48:51.937 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cf1b354:::benchmark_data_smithi029_108893_object95836:head# 2024-03-24T12:48:51.941 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cf4222a:::benchmark_data_smithi029_108893_object98694:head# 2024-03-24T12:48:51.942 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cf6d329:::benchmark_data_smithi029_108893_object92959:head# 2024-03-24T12:48:51.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cf8833c:::benchmark_data_smithi029_108893_object108683:head# 2024-03-24T12:48:51.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cf8a001:::benchmark_data_smithi029_108893_object95144:head# 2024-03-24T12:48:51.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9cfe8710:::benchmark_data_smithi029_108893_object99700:head# 2024-03-24T12:48:51.943 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d02de5b:::benchmark_data_smithi029_108893_object102886:head# 2024-03-24T12:48:51.944 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d075689:::benchmark_data_smithi029_108893_object112155:head# 2024-03-24T12:48:51.945 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d16d268:::benchmark_data_smithi029_108893_object104430:head# 2024-03-24T12:48:51.945 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d1bf6f6:::benchmark_data_smithi029_108893_object102890:head# 2024-03-24T12:48:51.948 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d24102f:::benchmark_data_smithi029_108893_object103748:head# 2024-03-24T12:48:51.949 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d26747a:::benchmark_data_smithi029_108893_object99869:head# 2024-03-24T12:48:51.949 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d2d5b5a:::benchmark_data_smithi029_108893_object109100:head# 2024-03-24T12:48:51.950 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d2f09f8:::benchmark_data_smithi029_108893_object97700:head# 2024-03-24T12:48:51.950 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d3103cf:::benchmark_data_smithi029_108893_object108053:head# 2024-03-24T12:48:51.951 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d336d19:::benchmark_data_smithi029_108893_object112042:head# 2024-03-24T12:48:51.951 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d34a51c:::benchmark_data_smithi029_108893_object104451:head# 2024-03-24T12:48:51.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:51.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:51.953 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d3647ea:::benchmark_data_smithi029_108893_object94283:head# 2024-03-24T12:48:51.954 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d36c1f6:::benchmark_data_smithi029_108893_object98691:head# 2024-03-24T12:48:51.955 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d3b170a:::benchmark_data_smithi029_108893_object110611:head# 2024-03-24T12:48:51.955 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d40d03c:::benchmark_data_smithi029_108893_object104448:head# 2024-03-24T12:48:51.956 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d48a3bd:::benchmark_data_smithi029_108893_object93813:head# 2024-03-24T12:48:51.956 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d4be88d:::benchmark_data_smithi029_108893_object93391:head# 2024-03-24T12:48:51.957 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d53818f:::benchmark_data_smithi029_108893_object99662:head# 2024-03-24T12:48:51.960 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d59ff84:::benchmark_data_smithi029_108893_object106093:head# 2024-03-24T12:48:51.960 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d5a4467:::benchmark_data_smithi029_108893_object93484:head# 2024-03-24T12:48:51.960 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d5ef9e5:::benchmark_data_smithi029_108893_object110898:head# 2024-03-24T12:48:51.960 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d6019a8:::benchmark_data_smithi029_108893_object105013:head# 2024-03-24T12:48:51.960 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d61528b:::benchmark_data_smithi029_108893_object113023:head# 2024-03-24T12:48:51.961 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d628222:::benchmark_data_smithi029_108893_object98923:head# 2024-03-24T12:48:51.961 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d64367d:::benchmark_data_smithi029_108893_object93948:head# 2024-03-24T12:48:51.962 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d71180c:::benchmark_data_smithi029_108893_object97962:head# 2024-03-24T12:48:51.964 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d728eeb:::benchmark_data_smithi029_108893_object106837:head# 2024-03-24T12:48:51.964 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d7800e0:::benchmark_data_smithi029_108893_object96273:head# 2024-03-24T12:48:51.964 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d7b9f36:::benchmark_data_smithi029_108893_object107030:head# 2024-03-24T12:48:51.964 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d7c5cce:::benchmark_data_smithi029_108893_object107489:head# 2024-03-24T12:48:51.966 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d8145d4:::benchmark_data_smithi029_108893_object93742:head# 2024-03-24T12:48:51.966 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d857449:::benchmark_data_smithi029_108893_object108951:head# 2024-03-24T12:48:51.966 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d8871dc:::benchmark_data_smithi029_108893_object110725:head# 2024-03-24T12:48:51.967 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d897002:::benchmark_data_smithi029_108893_object93941:head# 2024-03-24T12:48:51.967 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d8dea17:::benchmark_data_smithi029_108893_object109986:head# 2024-03-24T12:48:51.968 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d8e24a4:::benchmark_data_smithi029_108893_object101639:head# 2024-03-24T12:48:51.968 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d9622c6:::benchmark_data_smithi029_108893_object95627:head# 2024-03-24T12:48:51.968 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d969512:::benchmark_data_smithi029_108893_object96820:head# 2024-03-24T12:48:51.969 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d9807d9:::benchmark_data_smithi029_108893_object110632:head# 2024-03-24T12:48:51.970 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d9a85d3:::benchmark_data_smithi029_108893_object113778:head# 2024-03-24T12:48:51.971 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9d9b96c4:::benchmark_data_smithi029_108893_object95748:head# 2024-03-24T12:48:51.971 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9da5ddff:::benchmark_data_smithi029_108893_object98965:head# 2024-03-24T12:48:51.972 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dad612c:::benchmark_data_smithi029_108893_object107256:head# 2024-03-24T12:48:51.972 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9db5d5cf:::benchmark_data_smithi029_108893_object111337:head# 2024-03-24T12:48:51.972 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9db6f066:::benchmark_data_smithi029_108893_object111794:head# 2024-03-24T12:48:51.973 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dba815a:::benchmark_data_smithi029_108893_object96465:head# 2024-03-24T12:48:51.973 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dc05cd3:::benchmark_data_smithi029_108893_object103867:head# 2024-03-24T12:48:51.974 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dc200fc:::benchmark_data_smithi029_108893_object112308:head# 2024-03-24T12:48:51.974 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dc4e7f2:::benchmark_data_smithi029_108893_object108847:head# 2024-03-24T12:48:51.975 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dc53e91:::benchmark_data_smithi029_108893_object94444:head# 2024-03-24T12:48:51.975 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dc77b7f:::benchmark_data_smithi029_108893_object104611:head# 2024-03-24T12:48:51.976 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dc8e375:::benchmark_data_smithi029_108893_object107160:head# 2024-03-24T12:48:51.976 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dcdf4e5:::benchmark_data_smithi029_108893_object104874:head# 2024-03-24T12:48:51.978 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dce2304:::benchmark_data_smithi029_108893_object101960:head# 2024-03-24T12:48:51.978 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dd0253d:::benchmark_data_smithi029_108893_object99924:head# 2024-03-24T12:48:51.978 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dd07418:::benchmark_data_smithi029_108893_object100780:head# 2024-03-24T12:48:51.979 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dd17470:::benchmark_data_smithi029_108893_object99068:head# 2024-03-24T12:48:51.979 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dd2d10b:::benchmark_data_smithi029_108893_object95118:head# 2024-03-24T12:48:51.980 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dd385d0:::benchmark_data_smithi029_108893_object112741:head# 2024-03-24T12:48:51.980 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dd54f3f:::benchmark_data_smithi029_108893_object109888:head# 2024-03-24T12:48:51.980 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dd6a938:::benchmark_data_smithi029_108893_object113472:head# 2024-03-24T12:48:51.981 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9dd74fc6:::benchmark_data_smithi029_108893_object103602:head# 2024-03-24T12:48:51.981 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9de440e0:::benchmark_data_smithi029_108893_object94980:head# 2024-03-24T12:48:51.982 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9de9b39b:::benchmark_data_smithi029_108893_object108057:head# 2024-03-24T12:48:51.982 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9deb56c6:::benchmark_data_smithi029_108893_object104343:head# 2024-03-24T12:48:51.983 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9defbaff:::benchmark_data_smithi029_108893_object113707:head# 2024-03-24T12:48:51.983 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9df0c94d:::benchmark_data_smithi029_108893_object105843:head# 2024-03-24T12:48:51.983 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9df2491b:::benchmark_data_smithi029_108893_object100782:head# 2024-03-24T12:48:51.984 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9df35787:::benchmark_data_smithi029_108893_object97574:head# 2024-03-24T12:48:51.984 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9df4bbcb:::benchmark_data_smithi029_108893_object101403:head# 2024-03-24T12:48:51.985 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9df61c7b:::benchmark_data_smithi029_108893_object101926:head# 2024-03-24T12:48:51.985 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9df6daaa:::benchmark_data_smithi029_108893_object94147:head# 2024-03-24T12:48:51.986 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9df8502a:::benchmark_data_smithi029_108893_object92879:head# 2024-03-24T12:48:51.986 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e015007:::benchmark_data_smithi029_108893_object94603:head# 2024-03-24T12:48:51.987 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e034dee:::benchmark_data_smithi029_108893_object106508:head# 2024-03-24T12:48:51.989 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e059bd4:::benchmark_data_smithi029_108893_object109712:head# 2024-03-24T12:48:51.989 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e0a32c7:::benchmark_data_smithi029_108893_object111192:head# 2024-03-24T12:48:51.989 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e0aaa2d:::benchmark_data_smithi029_108893_object112083:head# 2024-03-24T12:48:51.989 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e0bc3dd:::benchmark_data_smithi029_108893_object102280:head# 2024-03-24T12:48:51.990 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e0fb230:::benchmark_data_smithi029_108893_object95961:head# 2024-03-24T12:48:51.990 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e1200aa:::benchmark_data_smithi029_108893_object105544:head# 2024-03-24T12:48:51.990 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e1249fd:::benchmark_data_smithi029_108893_object104144:head# 2024-03-24T12:48:51.991 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e19fc0b:::benchmark_data_smithi029_108893_object99044:head# 2024-03-24T12:48:51.993 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e1a6579:::benchmark_data_smithi029_108893_object99219:head# 2024-03-24T12:48:51.994 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e1c686a:::benchmark_data_smithi029_108893_object103978:head# 2024-03-24T12:48:51.994 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e20f791:::benchmark_data_smithi029_108893_object94100:head# 2024-03-24T12:48:51.994 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e2100a0:::benchmark_data_smithi029_108893_object108620:head# 2024-03-24T12:48:51.995 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e2b94f5:::benchmark_data_smithi029_108893_object98987:head# 2024-03-24T12:48:51.995 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e2dd6bc:::benchmark_data_smithi029_108893_object100802:head# 2024-03-24T12:48:51.996 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e2f71f9:::benchmark_data_smithi029_108893_object101358:head# 2024-03-24T12:48:51.997 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e3024e6:::benchmark_data_smithi029_108893_object113008:head# 2024-03-24T12:48:51.998 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e32b996:::benchmark_data_smithi029_108893_object113100:head# 2024-03-24T12:48:51.998 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e332a8d:::benchmark_data_smithi029_108893_object104070:head# 2024-03-24T12:48:51.999 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e335ca4:::benchmark_data_smithi029_108893_object110793:head# 2024-03-24T12:48:51.999 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e352d46:::benchmark_data_smithi029_108893_object101499:head# 2024-03-24T12:48:52.000 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e373111:::benchmark_data_smithi029_108893_object99506:head# 2024-03-24T12:48:52.000 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e393901:::benchmark_data_smithi029_108893_object110128:head# 2024-03-24T12:48:52.001 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e3ba2a6:::benchmark_data_smithi029_108893_object102152:head# 2024-03-24T12:48:52.001 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e3bd255:::benchmark_data_smithi029_108893_object103640:head# 2024-03-24T12:48:52.002 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e3c8230:::benchmark_data_smithi029_108893_object111406:head# 2024-03-24T12:48:52.002 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e3cd231:::benchmark_data_smithi029_108893_object93095:head# 2024-03-24T12:48:52.003 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e412726:::benchmark_data_smithi029_108893_object113070:head# 2024-03-24T12:48:52.003 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e413b98:::benchmark_data_smithi029_108893_object97389:head# 2024-03-24T12:48:52.004 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e43a03c:::benchmark_data_smithi029_108893_object106898:head# 2024-03-24T12:48:52.005 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e489516:::benchmark_data_smithi029_108893_object103060:head# 2024-03-24T12:48:52.008 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e4b41cd:::benchmark_data_smithi029_108893_object110509:head# 2024-03-24T12:48:52.010 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e4c9080:::benchmark_data_smithi029_108893_object95685:head# 2024-03-24T12:48:52.010 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e500de6:::benchmark_data_smithi029_108893_object96581:head# 2024-03-24T12:48:52.011 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e5057fe:::benchmark_data_smithi029_108893_object98046:head# 2024-03-24T12:48:52.011 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e5083a9:::benchmark_data_smithi029_108893_object103200:head# 2024-03-24T12:48:52.012 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e56ff06:::benchmark_data_smithi029_108893_object96511:head# 2024-03-24T12:48:52.013 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e592fcc:::benchmark_data_smithi029_108893_object95507:head# 2024-03-24T12:48:52.013 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e5abf5f:::benchmark_data_smithi029_108893_object111976:head# 2024-03-24T12:48:52.015 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e5ae6ae:::benchmark_data_smithi029_108893_object104504:head# 2024-03-24T12:48:52.016 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e5d1f06:::benchmark_data_smithi029_108893_object93719:head# 2024-03-24T12:48:52.016 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e5dfbda:::benchmark_data_smithi029_108893_object108735:head# 2024-03-24T12:48:52.017 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e5fbd38:::benchmark_data_smithi029_108893_object107501:head# 2024-03-24T12:48:52.017 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e62684c:::benchmark_data_smithi029_108893_object100957:head# 2024-03-24T12:48:52.020 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e6363ce:::benchmark_data_smithi029_108893_object106460:head# 2024-03-24T12:48:52.020 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e6d6cbf:::benchmark_data_smithi029_108893_object96804:head# 2024-03-24T12:48:52.020 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e6fb9ca:::benchmark_data_smithi029_108893_object103939:head# 2024-03-24T12:48:52.023 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e71c765:::benchmark_data_smithi029_108893_object94271:head# 2024-03-24T12:48:52.023 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e73266c:::benchmark_data_smithi029_108893_object96712:head# 2024-03-24T12:48:52.024 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e74237d:::benchmark_data_smithi029_108893_object97275:head# 2024-03-24T12:48:52.024 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e75ad81:::benchmark_data_smithi029_108893_object104565:head# 2024-03-24T12:48:52.025 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e779cef:::benchmark_data_smithi029_108893_object102421:head# 2024-03-24T12:48:52.025 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e7a0596:::benchmark_data_smithi029_108893_object101402:head# 2024-03-24T12:48:52.026 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e7a69a2:::benchmark_data_smithi029_108893_object95758:head# 2024-03-24T12:48:52.026 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e80086a:::benchmark_data_smithi029_108893_object110722:head# 2024-03-24T12:48:52.027 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e812c71:::benchmark_data_smithi029_108893_object109790:head# 2024-03-24T12:48:52.027 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e815aad:::benchmark_data_smithi029_108893_object98455:head# 2024-03-24T12:48:52.028 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e86341f:::benchmark_data_smithi029_108893_object111547:head# 2024-03-24T12:48:52.028 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e8a4969:::benchmark_data_smithi029_108893_object98995:head# 2024-03-24T12:48:52.029 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e8c1f05:::benchmark_data_smithi029_108893_object111854:head# 2024-03-24T12:48:52.029 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e8d6a1f:::benchmark_data_smithi029_108893_object107813:head# 2024-03-24T12:48:52.029 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e8e1cb2:::benchmark_data_smithi029_108893_object109116:head# 2024-03-24T12:48:52.030 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e8fbba4:::benchmark_data_smithi029_108893_object94576:head# 2024-03-24T12:48:52.030 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e909f61:::benchmark_data_smithi029_108893_object102304:head# 2024-03-24T12:48:52.031 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e93d659:::benchmark_data_smithi029_108893_object98879:head# 2024-03-24T12:48:52.031 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e948ad3:::benchmark_data_smithi029_108893_object98019:head# 2024-03-24T12:48:52.032 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e9a4e7f:::benchmark_data_smithi029_108893_object109962:head# 2024-03-24T12:48:52.032 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e9ce001:::benchmark_data_smithi029_108893_object96608:head# 2024-03-24T12:48:52.033 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9e9e8001:::benchmark_data_smithi029_108893_object96708:head# 2024-03-24T12:48:52.034 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ea28ae8:::benchmark_data_smithi029_108893_object109184:head# 2024-03-24T12:48:52.034 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ea3e1a6:::benchmark_data_smithi029_108893_object105281:head# 2024-03-24T12:48:52.035 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ea41fd9:::benchmark_data_smithi029_108893_object104187:head# 2024-03-24T12:48:52.036 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ea603b5:::benchmark_data_smithi029_108893_object103379:head# 2024-03-24T12:48:52.036 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ea9a8d5:::benchmark_data_smithi029_108893_object102931:head# 2024-03-24T12:48:52.037 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9eaa9589:::benchmark_data_smithi029_108893_object113277:head# 2024-03-24T12:48:52.037 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9eaabd02:::benchmark_data_smithi029_108893_object104645:head# 2024-03-24T12:48:52.038 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9eaff22a:::benchmark_data_smithi029_108893_object105757:head# 2024-03-24T12:48:52.039 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9eb00c46:::benchmark_data_smithi029_108893_object112919:head# 2024-03-24T12:48:52.039 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9eb17d44:::benchmark_data_smithi029_108893_object96236:head# 2024-03-24T12:48:52.040 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9eb27bed:::benchmark_data_smithi029_108893_object107931:head# 2024-03-24T12:48:52.040 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9eb2f9a3:::benchmark_data_smithi029_108893_object96889:head# 2024-03-24T12:48:52.040 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ebbb8f9:::benchmark_data_smithi029_108893_object94063:head# 2024-03-24T12:48:52.041 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ebbbb1b:::benchmark_data_smithi029_108893_object103311:head# 2024-03-24T12:48:52.041 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ec1fc51:::benchmark_data_smithi029_108893_object98558:head# 2024-03-24T12:48:52.042 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ec22666:::benchmark_data_smithi029_108893_object103796:head# 2024-03-24T12:48:52.042 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ec324ce:::benchmark_data_smithi029_108893_object112432:head# 2024-03-24T12:48:52.043 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ec99cfd:::benchmark_data_smithi029_108893_object102057:head# 2024-03-24T12:48:52.043 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ecc6508:::benchmark_data_smithi029_108893_object92852:head# 2024-03-24T12:48:52.044 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ecd6042:::benchmark_data_smithi029_108893_object112600:head# 2024-03-24T12:48:52.044 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ed1af55:::benchmark_data_smithi029_108893_object109284:head# 2024-03-24T12:48:52.045 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ed236a1:::benchmark_data_smithi029_108893_object93949:head# 2024-03-24T12:48:52.045 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ed31442:::benchmark_data_smithi029_108893_object103727:head# 2024-03-24T12:48:52.046 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ed501b0:::benchmark_data_smithi029_108893_object99943:head# 2024-03-24T12:48:52.046 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ed57d54:::benchmark_data_smithi029_108893_object108206:head# 2024-03-24T12:48:52.047 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ed776c0:::benchmark_data_smithi029_108893_object106915:head# 2024-03-24T12:48:52.047 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9eda0ba1:::benchmark_data_smithi029_108893_object109004:head# 2024-03-24T12:48:52.048 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9edaf2a5:::benchmark_data_smithi029_108893_object98350:head# 2024-03-24T12:48:52.049 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ede6dc0:::benchmark_data_smithi029_108893_object107632:head# 2024-03-24T12:48:52.049 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ee1c08f:::benchmark_data_smithi029_108893_object99486:head# 2024-03-24T12:48:52.049 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ee26864:::benchmark_data_smithi029_108893_object95930:head# 2024-03-24T12:48:52.050 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ee2d7e4:::benchmark_data_smithi029_108893_object98212:head# 2024-03-24T12:48:52.050 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ee3f51b:::benchmark_data_smithi029_108893_object98194:head# 2024-03-24T12:48:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:48:52.051 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ee67dd7:::benchmark_data_smithi029_108893_object110926:head# 2024-03-24T12:48:52.052 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9eea6187:::benchmark_data_smithi029_108893_object99549:head# 2024-03-24T12:48:52.052 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ef03607:::benchmark_data_smithi029_108893_object106724:head# 2024-03-24T12:48:52.052 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ef237fb:::benchmark_data_smithi029_108893_object113179:head# 2024-03-24T12:48:52.053 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ef426c3:::benchmark_data_smithi029_108893_object99473:head# 2024-03-24T12:48:52.053 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:52.054 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ef4c09e:::benchmark_data_smithi029_108893_object108308:head# 2024-03-24T12:48:52.054 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ef5fe83:::benchmark_data_smithi029_108893_object96492:head# 2024-03-24T12:48:52.056 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ef7e804:::benchmark_data_smithi029_108893_object95019:head# 2024-03-24T12:48:52.056 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9efd4396:::benchmark_data_smithi029_108893_object108414:head# 2024-03-24T12:48:52.056 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f042c1f:::benchmark_data_smithi029_108893_object108134:head# 2024-03-24T12:48:52.056 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f05ad45:::benchmark_data_smithi029_108893_object112461:head# 2024-03-24T12:48:52.056 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f05b59c:::benchmark_data_smithi029_108893_object106427:head# 2024-03-24T12:48:52.057 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f066858:::benchmark_data_smithi029_108893_object102519:head# 2024-03-24T12:48:52.057 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f089666:::benchmark_data_smithi029_108893_object95152:head# 2024-03-24T12:48:52.058 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f09d5ed:::benchmark_data_smithi029_108893_object113075:head# 2024-03-24T12:48:52.058 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f0f9cc7:::benchmark_data_smithi029_108893_object108837:head# 2024-03-24T12:48:52.059 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f1370ef:::benchmark_data_smithi029_108893_object99286:head# 2024-03-24T12:48:52.059 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f142973:::benchmark_data_smithi029_108893_object109383:head# 2024-03-24T12:48:52.061 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f17b289:::benchmark_data_smithi029_108893_object94730:head# 2024-03-24T12:48:52.061 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f18434e:::benchmark_data_smithi029_108893_object97411:head# 2024-03-24T12:48:52.062 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f19d650:::benchmark_data_smithi029_108893_object95826:head# 2024-03-24T12:48:52.062 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f19fae6:::benchmark_data_smithi029_108893_object110103:head# 2024-03-24T12:48:52.063 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f1a3111:::benchmark_data_smithi029_108893_object109903:head# 2024-03-24T12:48:52.064 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f1a8a54:::benchmark_data_smithi029_108893_object105563:head# 2024-03-24T12:48:52.064 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f1b4939:::benchmark_data_smithi029_108893_object93148:head# 2024-03-24T12:48:52.065 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f1b8b38:::benchmark_data_smithi029_108893_object97596:head# 2024-03-24T12:48:52.065 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f1cac3c:::benchmark_data_smithi029_108893_object99150:head# 2024-03-24T12:48:52.066 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f1db002:::benchmark_data_smithi029_108893_object105996:head# 2024-03-24T12:48:52.066 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f2063c8:::benchmark_data_smithi029_108893_object102731:head# 2024-03-24T12:48:52.067 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f232b81:::benchmark_data_smithi029_108893_object111706:head# 2024-03-24T12:48:52.067 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f35562f:::benchmark_data_smithi029_108893_object103986:head# 2024-03-24T12:48:52.068 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f3c24ee:::benchmark_data_smithi029_108893_object102540:head# 2024-03-24T12:48:52.068 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f3c9122:::benchmark_data_smithi029_108893_object101588:head# 2024-03-24T12:48:52.069 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f49586d:::benchmark_data_smithi029_108893_object95646:head# 2024-03-24T12:48:52.069 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f49ae46:::benchmark_data_smithi029_108893_object107637:head# 2024-03-24T12:48:52.071 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f4a14f0:::benchmark_data_smithi029_108893_object104032:head# 2024-03-24T12:48:52.071 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f4ac280:::benchmark_data_smithi029_108893_object111427:head# 2024-03-24T12:48:52.071 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f4c751a:::benchmark_data_smithi029_108893_object93861:head# 2024-03-24T12:48:52.072 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f5bb997:::benchmark_data_smithi029_108893_object111653:head# 2024-03-24T12:48:52.072 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f5d3699:::benchmark_data_smithi029_108893_object108653:head# 2024-03-24T12:48:52.073 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f623dce:::benchmark_data_smithi029_108893_object93288:head# 2024-03-24T12:48:52.073 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f668c82:::benchmark_data_smithi029_108893_object93179:head# 2024-03-24T12:48:52.074 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f672166:::benchmark_data_smithi029_108893_object99686:head# 2024-03-24T12:48:52.074 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f69e60e:::benchmark_data_smithi029_108893_object96788:head# 2024-03-24T12:48:52.074 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f6ca2b3:::benchmark_data_smithi029_108893_object99632:head# 2024-03-24T12:48:52.075 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f71f05b:::benchmark_data_smithi029_108893_object113661:head# 2024-03-24T12:48:52.075 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f73ea00:::benchmark_data_smithi029_108893_object97911:head# 2024-03-24T12:48:52.076 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f74bcbf:::benchmark_data_smithi029_108893_object104450:head# 2024-03-24T12:48:52.076 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f7f5c2d:::benchmark_data_smithi029_108893_object103614:head# 2024-03-24T12:48:52.077 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f7fc9b0:::benchmark_data_smithi029_108893_object94246:head# 2024-03-24T12:48:52.077 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f81b87d:::benchmark_data_smithi029_108893_object112159:head# 2024-03-24T12:48:52.079 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f87a24e:::benchmark_data_smithi029_108893_object101909:head# 2024-03-24T12:48:52.079 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f89ecbc:::benchmark_data_smithi029_108893_object99497:head# 2024-03-24T12:48:52.079 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f8be8cc:::benchmark_data_smithi029_108893_object97313:head# 2024-03-24T12:48:52.080 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f8f1eaa:::benchmark_data_smithi029_108893_object103539:head# 2024-03-24T12:48:52.080 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9f9b27bc:::benchmark_data_smithi029_108893_object95949:head# 2024-03-24T12:48:52.080 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fa442cc:::benchmark_data_smithi029_108893_object97439:head# 2024-03-24T12:48:52.081 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fa4c651:::benchmark_data_smithi029_108893_object113222:head# 2024-03-24T12:48:52.081 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fa96145:::benchmark_data_smithi029_108893_object99063:head# 2024-03-24T12:48:52.082 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9facc4e2:::benchmark_data_smithi029_108893_object104949:head# 2024-03-24T12:48:52.082 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:52.083 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fb1f65b:::benchmark_data_smithi029_108893_object93969:head# 2024-03-24T12:48:52.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:52.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:52.084 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fb5e834:::benchmark_data_smithi029_108893_object107895:head# 2024-03-24T12:48:52.085 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fb6f795:::benchmark_data_smithi029_108893_object102040:head# 2024-03-24T12:48:52.085 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fb7f112:::benchmark_data_smithi029_108893_object97286:head# 2024-03-24T12:48:52.085 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fbb1b7f:::benchmark_data_smithi029_108893_object112799:head# 2024-03-24T12:48:52.085 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fbcb980:::benchmark_data_smithi029_108893_object93618:head# 2024-03-24T12:48:52.086 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fc1c9d0:::benchmark_data_smithi029_108893_object111115:head# 2024-03-24T12:48:52.086 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fc20539:::benchmark_data_smithi029_108893_object105416:head# 2024-03-24T12:48:52.086 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fc4c7ed:::benchmark_data_smithi029_108893_object105204:head# 2024-03-24T12:48:52.087 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fc7304e:::benchmark_data_smithi029_108893_object105053:head# 2024-03-24T12:48:52.087 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fcbc86e:::benchmark_data_smithi029_108893_object98010:head# 2024-03-24T12:48:52.088 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fd06d1a:::benchmark_data_smithi029_108893_object104981:head# 2024-03-24T12:48:52.088 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fd417bc:::benchmark_data_smithi029_108893_object104609:head# 2024-03-24T12:48:52.089 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fd5570a:::benchmark_data_smithi029_108893_object111237:head# 2024-03-24T12:48:52.089 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fd98901:::benchmark_data_smithi029_108893_object104015:head# 2024-03-24T12:48:52.090 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fdba89c:::benchmark_data_smithi029_108893_object103418:head# 2024-03-24T12:48:52.090 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fdc07be:::benchmark_data_smithi029_108893_object102093:head# 2024-03-24T12:48:52.090 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fe05f78:::benchmark_data_smithi029_108893_object101274:head# 2024-03-24T12:48:52.092 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fe74451:::benchmark_data_smithi029_108893_object94954:head# 2024-03-24T12:48:52.092 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fea20d7:::benchmark_data_smithi029_108893_object112951:head# 2024-03-24T12:48:52.093 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9fede457:::benchmark_data_smithi029_108893_object99666:head# 2024-03-24T12:48:52.093 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ff14ae8:::benchmark_data_smithi029_108893_object108972:head# 2024-03-24T12:48:52.094 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ff3e17f:::benchmark_data_smithi029_108893_object102379:head# 2024-03-24T12:48:52.096 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ff66964:::benchmark_data_smithi029_108893_object97028:head# 2024-03-24T12:48:52.096 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ffa1843:::benchmark_data_smithi029_108893_object93869:head# 2024-03-24T12:48:52.097 INFO:teuthology.orchestra.run.smithi190.stderr:Read #8:9ffc3b76:::benchmark_data_smithi029_108893_object95243:head# 2024-03-24T12:48:52.100 INFO:teuthology.orchestra.run.smithi190.stderr:Export successful 2024-03-24T12:48:52.185 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:52.214 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:52.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:52.315 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:52.348 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:52.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:52.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:52.450 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:52.479 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:52.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:52.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:52.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:52 smithi029 ceph-mon[41500]: pgmap v1003: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.1 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 9893/53706 objects degraded (18.421%) 2024-03-24T12:48:52.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:52 smithi029 ceph-mon[29919]: pgmap v1003: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.1 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 9893/53706 objects degraded (18.421%) 2024-03-24T12:48:52.571 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:52 smithi190 ceph-mon[33748]: pgmap v1003: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.1 GiB data, 9.2 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.44k op/s; 9893/53706 objects degraded (18.421%) 2024-03-24T12:48:52.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:52.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:52.625 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:52.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:52.726 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:52.763 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:52.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:52.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:52.865 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:52.896 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:52.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:52.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:52.997 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:53.027 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:53.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:53.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:53.129 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:53.166 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:53.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:53.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:53.268 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:53.303 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:53.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:53.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:53.428 INFO:teuthology.orchestra.run.smithi190.stdout:ERROR: (22) Invalid argument 2024-03-24T12:48:53.428 INFO:teuthology.orchestra.run.smithi190.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:48:53.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:53.451 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:53.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:53.552 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:53.581 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:53.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:53.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:53.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:53 smithi029 ceph-mon[29919]: pgmap v1004: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.0 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 577 B/s rd, 0 B/s wr, 1.03k op/s; 9476/51147 objects degraded (18.527%) 2024-03-24T12:48:53.618 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:53 smithi029 ceph-mon[29919]: osdmap e422: 8 total, 6 up, 6 in 2024-03-24T12:48:53.618 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:53 smithi029 ceph-mon[41500]: pgmap v1004: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.0 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 577 B/s rd, 0 B/s wr, 1.03k op/s; 9476/51147 objects degraded (18.527%) 2024-03-24T12:48:53.618 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:53 smithi029 ceph-mon[41500]: osdmap e422: 8 total, 6 up, 6 in 2024-03-24T12:48:53.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:53 smithi190 ceph-mon[33748]: pgmap v1004: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 1.0 GiB data, 9.0 GiB used, 527 GiB / 536 GiB avail; 577 B/s rd, 0 B/s wr, 1.03k op/s; 9476/51147 objects degraded (18.527%) 2024-03-24T12:48:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:53 smithi190 ceph-mon[33748]: osdmap e422: 8 total, 6 up, 6 in 2024-03-24T12:48:53.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:53.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:53.725 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:53.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:53.826 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:53.852 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:53.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:53.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:53.954 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:53.981 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:53.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:53.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:54.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:48:54.083 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:54.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-24T12:48:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:54.110 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:54.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:54.205 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:48:54.211 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:54.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:54.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:54.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:54.252 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:48:54.299 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:48:54.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:54.348 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:48:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:54.387 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:54.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:54.389 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:48:54.438 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:48:54.438 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-24T12:48:54.438 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:48:54.438 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:48:54.438 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:48:54.438 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:48:54.439 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:48:54.439 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:48:54.439 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:48:54.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:54.528 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:54.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:54.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:54 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:48:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:54 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:48:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:54 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2171697018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:48:54.653 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:54.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:54.656 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:48:54.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:54 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:48:54.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:54 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:48:54.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:54 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2171697018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:48:54.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:54 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:48:54.669 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:54 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:48:54.670 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:54 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2171697018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:48:54.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:54.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:54.803 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:54.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:54.905 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:54.946 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:54.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:55.048 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:55.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:55.078 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:55.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:55.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:55.214 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:55.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:48:55.262 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:48:55.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:55.357 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:55.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:55.459 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:55.489 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:55.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:55.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:55.536 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:55 smithi029 ceph-mon[29919]: pgmap v1006: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 874 MiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 8224/42522 objects degraded (19.341%) 2024-03-24T12:48:55.536 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:55 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 9476/51147 objects degraded (18.527%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:55.536 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:55 smithi029 ceph-mon[41500]: pgmap v1006: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 874 MiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 8224/42522 objects degraded (19.341%) 2024-03-24T12:48:55.536 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:55 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 9476/51147 objects degraded (18.527%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:55.591 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:55.610 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:55 smithi190 ceph-mon[33748]: pgmap v1006: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 874 MiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.17k op/s; 8224/42522 objects degraded (19.341%) 2024-03-24T12:48:55.610 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:55 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 9476/51147 objects degraded (18.527%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:48:55.616 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:55.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:55.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.5 --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --force --op remove --pgid 8.9 2024-03-24T12:48:55.718 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:55.760 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:55.760 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:55.861 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:55.891 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:55.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:55.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:55.918 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/osd.5/config 2024-03-24T12:48:55.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:56.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:56.031 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:56.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:56.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:56.168 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:56.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:56.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:56.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:56.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:56.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:56.442 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:56.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:56.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:56.545 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:56.575 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:56.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:56.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:56.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:56.712 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:56.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:56.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:56.814 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:56.842 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:56.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:56.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:56.944 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:56.990 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:56.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:56.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:57.092 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:57.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:57.131 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:57.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:57.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:57.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-24T12:48:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:57.269 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:57.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:57.370 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:57.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:57.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:57.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:57.421 INFO:teuthology.orchestra.run.smithi190.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:48:57.465 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:48:57.499 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:57.529 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:57.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:57.630 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:57.660 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:57.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:57.761 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:57.785 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:57.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:57.887 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:57.914 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:57.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:57.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:57.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:48:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:48:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:48:58.013 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:48:58.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:58.059 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:58.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:58.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:58.131 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:57 smithi190 ceph-mon[33748]: pgmap v1007: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 670 MiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 579 B/s rd, 0 B/s wr, 1.08k op/s; 5920/32742 objects degraded (18.081%) 2024-03-24T12:48:58.131 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:57 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3986688409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:48:58.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:57 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:48:58.161 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:58.201 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:58.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:58.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:58.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:48:58.344 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:58.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:58.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:48:58.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:57 smithi029 ceph-mon[29919]: pgmap v1007: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 670 MiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 579 B/s rd, 0 B/s wr, 1.08k op/s; 5920/32742 objects degraded (18.081%) 2024-03-24T12:48:58.359 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:57 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3986688409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:48:58.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:57 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:48:58.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:57 smithi029 ceph-mon[41500]: pgmap v1007: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 670 MiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 579 B/s rd, 0 B/s wr, 1.08k op/s; 5920/32742 objects degraded (18.081%) 2024-03-24T12:48:58.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:57 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3986688409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:48:58.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:57 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:48:58.446 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:58.454 INFO:teuthology.orchestra.run.smithi190.stdout: marking collection for removal 2024-03-24T12:48:58.455 INFO:teuthology.orchestra.run.smithi190.stdout:setting '_remove' omap key 2024-03-24T12:48:58.455 INFO:teuthology.orchestra.run.smithi190.stdout:finish_remove_pgs 8.9_head removing 8.9 2024-03-24T12:48:58.478 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:58.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:58.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:58.530 INFO:teuthology.orchestra.run.smithi190.stdout:Remove successful 2024-03-24T12:48:58.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:58.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:58.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:58.728 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:58.755 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:58.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:58.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:48:58.882 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:48:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:48:58.898 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:58.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:58.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:48:59.000 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:59.031 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:59.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:59.132 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:59.161 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:59.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:59.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:59.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:48:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:59.304 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:48:59.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:48:59.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:58 smithi190 ceph-mon[33748]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:48:59.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:58 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:48:59.320 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:48:58 smithi190 ceph-mon[33748]: osdmap e423: 8 total, 6 up, 6 in 2024-03-24T12:48:59.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:59.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:58 smithi029 ceph-mon[29919]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:48:59.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:58 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:48:59.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:48:58 smithi029 ceph-mon[29919]: osdmap e423: 8 total, 6 up, 6 in 2024-03-24T12:48:59.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:58 smithi029 ceph-mon[41500]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:48:59.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:58 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:48:59.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:48:58 smithi029 ceph-mon[41500]: osdmap e423: 8 total, 6 up, 6 in 2024-03-24T12:48:59.435 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:59.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:59.537 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:48:59.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:59.570 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:59.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:48:59.656 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:48:59.671 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:48:59.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.0 --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-03-24T12:48:59.713 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:59.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:59.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:48:59.817 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:48:59.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:48:59.851 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:48:59.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:48:59.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:00.028 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:00.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:00.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:00.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:00.175 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:00.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:00.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:49:00.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:49:00.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[29919]: 11.c scrub starts 2024-03-24T12:49:00.213 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:00.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[29919]: 11.c scrub ok 2024-03-24T12:49:00.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[29919]: pgmap v1009: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 415 MiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 4074/20505 objects degraded (19.868%) 2024-03-24T12:49:00.214 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[29919]: osdmap e424: 8 total, 6 up, 6 in 2024-03-24T12:49:00.214 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:49:00.214 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:49:00.214 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[41500]: 11.c scrub starts 2024-03-24T12:49:00.214 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:00.214 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[41500]: 11.c scrub ok 2024-03-24T12:49:00.214 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[41500]: pgmap v1009: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 415 MiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 4074/20505 objects degraded (19.868%) 2024-03-24T12:49:00.215 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:00 smithi029 ceph-mon[41500]: osdmap e424: 8 total, 6 up, 6 in 2024-03-24T12:49:00.218 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/osd.0/config 2024-03-24T12:49:00.276 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:00.304 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:00.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:00.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:00 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:49:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:00 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:49:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:00 smithi190 ceph-mon[33748]: 11.c scrub starts 2024-03-24T12:49:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:00 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:00 smithi190 ceph-mon[33748]: 11.c scrub ok 2024-03-24T12:49:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:00 smithi190 ceph-mon[33748]: pgmap v1009: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 415 MiB data, 7.8 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s; 4074/20505 objects degraded (19.868%) 2024-03-24T12:49:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:00 smithi190 ceph-mon[33748]: osdmap e424: 8 total, 6 up, 6 in 2024-03-24T12:49:00.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:00.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:00.449 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:00.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:00.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:00.576 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:49:00.576 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:49:00.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.576 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:208858 11.5 e424/418)", 2024-03-24T12:49:00.576 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:00.573739+0000", 2024-03-24T12:49:00.576 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.00062597200000000003, 2024-03-24T12:49:00.576 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00065011799999999998, 2024-03-24T12:49:00.576 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573739+0000", 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573739+0000", 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.577 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573740+0000", 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.37e-07 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573747+0000", 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4100000000000002e-06 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573757+0000", 2024-03-24T12:49:00.578 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4429999999999992e-06 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573764+0000", 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0360000000000001e-06 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:208857 11.3 e424/382 11:c67e2a81:::benchmark_data_smithi029_192894_object103922:head v 424'12837, mlcod=424'12834)", 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:00.573776+0000", 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.000589048, 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00067678499999999995, 2024-03-24T12:49:00.579 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573776+0000", 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573776+0000", 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573776+0000", 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.53e-07 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573780+0000", 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6509999999999999e-06 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573786+0000", 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6450000000000002e-06 2024-03-24T12:49:00.581 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573792+0000", 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9259999999999997e-06 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573858+0000", 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6027999999999999e-05 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:00.582 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573877+0000", 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9572000000000002e-05 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:208843 11.d e424/382)", 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:00.574478+0000", 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 4294967295.999887, 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4941999999999999e-05, 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:00.583 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574478+0000", 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574478+0000", 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574479+0000", 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9299999999999997e-07 2024-03-24T12:49:00.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574485+0000", 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8669999999999999e-06 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574494+0000", 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1300000000000007e-06 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574501+0000", 2024-03-24T12:49:00.585 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0979999999999996e-06 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:208859 11.6 e424/418)", 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:00.574525+0000", 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 4294967295.9998398, 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6285999999999998e-05, 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:00.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574525+0000", 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574525+0000", 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574526+0000", 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3000000000000001e-07 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.587 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574531+0000", 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8969999999999999e-06 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574538+0000", 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8800000000000002e-06 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.574543+0000", 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7230000000000001e-06 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.588 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:208837 11.d e424/382 11:bbce7935:::benchmark_data_smithi029_192894_object103902:head v 424'12985, mlcod=424'12980)", 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:00.563029+0000", 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.011336214000000001, 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.011584074999999999, 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.563029+0000", 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.589 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.563029+0000", 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.563029+0000", 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.89e-07 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:00.590 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.563033+0000", 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.512e-06 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.563038+0000", 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6130000000000003e-06 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.563043+0000", 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7650000000000001e-06 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:00.591 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.566121+0000", 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0030774449999999998 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.566139+0000", 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8403999999999999e-05 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:208839 11.d e424/382 11:bebbeee6:::benchmark_data_smithi029_192894_object103904:head v 424'12986, mlcod=424'12982)", 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:00.565183+0000", 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0091817410000000002, 2024-03-24T12:49:00.592 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0094923939999999995, 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565183+0000", 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565183+0000", 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565184+0000", 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6300000000000001e-07 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565188+0000", 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.048e-06 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565193+0000", 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8319999999999996e-06 2024-03-24T12:49:00.594 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565196+0000", 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.3560000000000001e-06 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.569320+0000", 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.004124295 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.569341+0000", 2024-03-24T12:49:00.595 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0543999999999998e-05 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:208845 11.5 e424/418 11:af80a15d:::benchmark_data_smithi029_192894_object103910:head v 424'13170, mlcod=424'13168)", 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:00.565707+0000", 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0086578079999999995, 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0090294510000000008, 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:00.596 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565707+0000", 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565707+0000", 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565708+0000", 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1099999999999996e-07 2024-03-24T12:49:00.597 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565713+0000", 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0289999999999996e-06 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565722+0000", 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3319999999999998e-06 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.598 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.565729+0000", 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4779999999999996e-06 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.568865+0000", 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0031359560000000001 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.568885+0000", 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9856000000000001e-05 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.599 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:208841 11.d e424/382)", 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:00.570709+0000", 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.003655789, 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0040846859999999997, 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "queued for pg", 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570709+0000", 2024-03-24T12:49:00.600 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570709+0000", 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570710+0000", 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9700000000000003e-07 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570715+0000", 2024-03-24T12:49:00.601 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8489999999999997e-06 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570721+0000", 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8059999999999997e-06 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570727+0000", 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9939999999999997e-06 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.602 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:208855 11.f e424/382 11:fedb3bf0:::benchmark_data_smithi029_192894_object103920:head v 424'12609, mlcod=424'12607)", 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:00.570832+0000", 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.003533563, 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0040123509999999999, 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570832+0000", 2024-03-24T12:49:00.603 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570832+0000", 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570832+0000", 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6100000000000001e-07 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:00.604 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570838+0000", 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.677e-06 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570846+0000", 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8809999999999996e-06 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570853+0000", 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7090000000000002e-06 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.605 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573170+0000", 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0023167019999999999 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.573193+0000", 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2566e-05 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.606 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:208853 11.6 e424/418 11:6aae618e:::benchmark_data_smithi029_192894_object103918:head v 424'13151, mlcod=424'13149)", 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:00.570890+0000", 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.0034752630000000001, 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0040169289999999998, 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570890+0000", 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570890+0000", 2024-03-24T12:49:00.607 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570890+0000", 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0900000000000002e-07 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570895+0000", 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2420000000000002e-06 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:00.608 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570902+0000", 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4089999999999997e-06 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.570908+0000", 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5639999999999996e-06 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.572577+0000", 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0016698189999999999 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:00.609 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:00.572600+0000", 2024-03-24T12:49:00.610 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2285000000000001e-05 2024-03-24T12:49:00.610 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.610 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:00.610 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.610 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:00.610 INFO:teuthology.orchestra.run.smithi029.stdout: ], 2024-03-24T12:49:00.610 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 10 2024-03-24T12:49:00.610 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:49:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:00.611 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:00.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:00.712 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:00.742 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:00.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:00.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:00.844 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:00.874 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:00.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:00.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:00.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:01.017 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:01.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:01.116 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:49:01.119 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:01.146 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:01.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:01.164 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[41500]: 6.4 scrub starts 2024-03-24T12:49:01.164 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[41500]: 6.4 scrub ok 2024-03-24T12:49:01.164 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[41500]: 6.6 scrub starts 2024-03-24T12:49:01.164 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[41500]: 6.6 scrub ok 2024-03-24T12:49:01.164 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[41500]: 4.14 scrub starts 2024-03-24T12:49:01.164 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[41500]: 4.14 scrub ok 2024-03-24T12:49:01.165 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 4074/20505 objects degraded (19.868%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:01.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[29919]: 6.4 scrub starts 2024-03-24T12:49:01.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[29919]: 6.4 scrub ok 2024-03-24T12:49:01.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[29919]: 6.6 scrub starts 2024-03-24T12:49:01.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[29919]: 6.6 scrub ok 2024-03-24T12:49:01.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[29919]: 4.14 scrub starts 2024-03-24T12:49:01.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[29919]: 4.14 scrub ok 2024-03-24T12:49:01.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:01 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 4074/20505 objects degraded (19.868%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:01.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:01.259 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Removed 104932 objects 2024-03-24T12:49:01.259 INFO:tasks.radosbench.radosbench.0.smithi029.stdout:Clean up completed and total clean up time :95.3306 2024-03-24T12:49:01.273 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:01.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:01.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:01.375 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:01.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:01 smithi190 ceph-mon[33748]: 6.4 scrub starts 2024-03-24T12:49:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:01 smithi190 ceph-mon[33748]: 6.4 scrub ok 2024-03-24T12:49:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:01 smithi190 ceph-mon[33748]: 6.6 scrub starts 2024-03-24T12:49:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:01 smithi190 ceph-mon[33748]: 6.6 scrub ok 2024-03-24T12:49:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:01 smithi190 ceph-mon[33748]: 4.14 scrub starts 2024-03-24T12:49:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:01 smithi190 ceph-mon[33748]: 4.14 scrub ok 2024-03-24T12:49:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:01 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 4074/20505 objects degraded (19.868%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:01.399 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:01.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:01.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:01.524 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:01.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:01.625 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:01.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:01.650 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:01.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:01.751 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:01.775 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:01.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:01.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:01.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:01.907 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:01.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:01.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:02.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:02.032 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:02.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:02.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:02.157 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:02.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:49:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[29919]: 3.13 scrub ok 2024-03-24T12:49:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[29919]: pgmap v1011: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 211 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 671 B/s rd, 0 B/s wr, 1.39k op/s; 1775/10692 objects degraded (16.601%) 2024-03-24T12:49:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[29919]: 11.e scrub starts 2024-03-24T12:49:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[29919]: 11.e scrub ok 2024-03-24T12:49:02.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[29919]: 6.b scrub starts 2024-03-24T12:49:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:49:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[41500]: 3.13 scrub ok 2024-03-24T12:49:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[41500]: pgmap v1011: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 211 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 671 B/s rd, 0 B/s wr, 1.39k op/s; 1775/10692 objects degraded (16.601%) 2024-03-24T12:49:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[41500]: 11.e scrub starts 2024-03-24T12:49:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[41500]: 11.e scrub ok 2024-03-24T12:49:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:02 smithi029 ceph-mon[41500]: 6.b scrub starts 2024-03-24T12:49:02.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:02.283 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:02.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:02 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:49:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:02 smithi190 ceph-mon[33748]: 3.13 scrub ok 2024-03-24T12:49:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:02 smithi190 ceph-mon[33748]: pgmap v1011: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 211 MiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 671 B/s rd, 0 B/s wr, 1.39k op/s; 1775/10692 objects degraded (16.601%) 2024-03-24T12:49:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:02 smithi190 ceph-mon[33748]: 11.e scrub starts 2024-03-24T12:49:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:02 smithi190 ceph-mon[33748]: 11.e scrub ok 2024-03-24T12:49:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:02 smithi190 ceph-mon[33748]: 6.b scrub starts 2024-03-24T12:49:02.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-24T12:49:02.384 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:02.409 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:02.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:02.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:02.511 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:02.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:02.535 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:02.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:02.636 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:02.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:02.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:02.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:02.764 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:02.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:02.790 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:02.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:02.891 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:02.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:03.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:03.048 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:03.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:03.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:03.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:03 smithi029 ceph-mon[29919]: 6.b scrub ok 2024-03-24T12:49:03.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:03 smithi029 ceph-mon[29919]: 6.3 scrub starts 2024-03-24T12:49:03.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:03 smithi029 ceph-mon[29919]: 6.3 scrub ok 2024-03-24T12:49:03.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:03 smithi029 ceph-mon[41500]: 6.b scrub ok 2024-03-24T12:49:03.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:03 smithi029 ceph-mon[41500]: 6.3 scrub starts 2024-03-24T12:49:03.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:03 smithi029 ceph-mon[41500]: 6.3 scrub ok 2024-03-24T12:49:03.174 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:03.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:03.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:03.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:03.303 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:03.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:03.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:03.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:03 smithi190 ceph-mon[33748]: 6.b scrub ok 2024-03-24T12:49:03.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:03 smithi190 ceph-mon[33748]: 6.3 scrub starts 2024-03-24T12:49:03.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:03 smithi190 ceph-mon[33748]: 6.3 scrub ok 2024-03-24T12:49:03.404 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:03.433 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:03.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:03.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:03.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:03.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:03.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:03.690 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:03.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:03.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:03.815 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:03.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:03.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:03.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:49:03 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:49:03.644+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:49:03.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:03.942 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:03.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:04.043 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:04.072 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:04.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:04.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:04.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:04.198 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:04.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:04.300 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:04.328 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:04.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:04.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:04 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:49:04.351 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:04 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:49:04.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:04 smithi029 ceph-mon[29919]: pgmap v1012: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 169 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.41k op/s; 1455/8676 objects degraded (16.770%) 2024-03-24T12:49:04.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:04 smithi029 ceph-mon[29919]: 6.7 scrub starts 2024-03-24T12:49:04.352 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:04.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:04 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:49:04.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:04 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:49:04.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:04 smithi029 ceph-mon[41500]: pgmap v1012: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 169 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.41k op/s; 1455/8676 objects degraded (16.770%) 2024-03-24T12:49:04.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:04 smithi029 ceph-mon[41500]: 6.7 scrub starts 2024-03-24T12:49:04.352 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:04 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:49:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:04 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:49:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:04 smithi190 ceph-mon[33748]: pgmap v1012: 158 pgs: 51 active+undersized, 43 active+undersized+degraded, 64 active+clean; 169 MiB data, 7.0 GiB used, 529 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 1.41k op/s; 1455/8676 objects degraded (16.770%) 2024-03-24T12:49:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:04 smithi190 ceph-mon[33748]: 6.7 scrub starts 2024-03-24T12:49:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:04.430 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:04.459 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:04.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:04.495 INFO:teuthology.orchestra.run.smithi029.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:49:04.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:04.584 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:04.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:04.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:04.709 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:04.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:04.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:04.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:04.813 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:49:04.813 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:49:04.813 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:49:04.813 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:49:04.813 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:49:04.813 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:49:04.814 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-24T12:49:04.814 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:49:04.814 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:49:04.814 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:49:04.814 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:49:04.814 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:49:04.814 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:49:04.814 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:49:04.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:04.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:04.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:04.932 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-24T12:49:04.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-24T12:49:04.940 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:04.968 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:04.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:04.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:05.023 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:05.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:05.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:05.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:05.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:05.149 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:05 smithi029 ceph-mon[29919]: 6.7 scrub ok 2024-03-24T12:49:05.149 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:05 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3908569826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:49:05.149 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:05 smithi029 ceph-mon[41500]: 6.7 scrub ok 2024-03-24T12:49:05.149 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:05 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3908569826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:49:05.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:05.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:05.222 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:05.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:05.323 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:05.350 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:05.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:05.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:05 smithi190 ceph-mon[33748]: 6.7 scrub ok 2024-03-24T12:49:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:05 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3908569826' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:49:05.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:05.475 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:05.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:05.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:05.577 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:05.605 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:05.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:05.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:05.707 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:05.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:05.833 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:05.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:05.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:05.958 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:05.982 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:05.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:05.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:06.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:06.107 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:06.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:06.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:06.209 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:06.237 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:06.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:06.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:06.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[41500]: 4.1d scrub starts 2024-03-24T12:49:06.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[41500]: 4.1d scrub ok 2024-03-24T12:49:06.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[41500]: 1.9 scrub starts 2024-03-24T12:49:06.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[41500]: 1.9 scrub ok 2024-03-24T12:49:06.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[41500]: pgmap v1013: 158 pgs: 1 active+clean+scrubbing, 51 active+undersized, 43 active+undersized+degraded, 63 active+clean; 35 MiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 524/2232 objects degraded (23.477%) 2024-03-24T12:49:06.284 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 524/2232 objects degraded (23.477%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:06.285 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:49:06.285 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:49:06.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[29919]: 4.1d scrub starts 2024-03-24T12:49:06.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[29919]: 4.1d scrub ok 2024-03-24T12:49:06.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[29919]: 1.9 scrub starts 2024-03-24T12:49:06.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[29919]: 1.9 scrub ok 2024-03-24T12:49:06.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[29919]: pgmap v1013: 158 pgs: 1 active+clean+scrubbing, 51 active+undersized, 43 active+undersized+degraded, 63 active+clean; 35 MiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 524/2232 objects degraded (23.477%) 2024-03-24T12:49:06.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 524/2232 objects degraded (23.477%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:06.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:49:06.285 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:06 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:49:06.339 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:06.363 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:06.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:06.364 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:06 smithi190 ceph-mon[33748]: 4.1d scrub starts 2024-03-24T12:49:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:06 smithi190 ceph-mon[33748]: 4.1d scrub ok 2024-03-24T12:49:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:06 smithi190 ceph-mon[33748]: 1.9 scrub starts 2024-03-24T12:49:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:06 smithi190 ceph-mon[33748]: 1.9 scrub ok 2024-03-24T12:49:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:06 smithi190 ceph-mon[33748]: pgmap v1013: 158 pgs: 1 active+clean+scrubbing, 51 active+undersized, 43 active+undersized+degraded, 63 active+clean; 35 MiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.27k op/s; 524/2232 objects degraded (23.477%) 2024-03-24T12:49:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:06 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 524/2232 objects degraded (23.477%), 43 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:06 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:49:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:06 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:49:06.465 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:06.493 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:06.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:06.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:06.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:06.619 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:06.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:06.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:06.707 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:06.721 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:06.736 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:49:06.737 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:49:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:06.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:06.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:06.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:06.851 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:06.879 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:06.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:06.980 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:07.007 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:07.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:07.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:07.109 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:07.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:07.138 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:07.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:07.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:07.265 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:07.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:07.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:07.274 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:07 smithi029 ceph-mon[29919]: 3.8 scrub starts 2024-03-24T12:49:07.274 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:07 smithi029 ceph-mon[29919]: 3.8 scrub ok 2024-03-24T12:49:07.274 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:07 smithi029 ceph-mon[41500]: 3.8 scrub starts 2024-03-24T12:49:07.274 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:07 smithi029 ceph-mon[41500]: 3.8 scrub ok 2024-03-24T12:49:07.367 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:07.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:07 smithi190 ceph-mon[33748]: 3.8 scrub starts 2024-03-24T12:49:07.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:07 smithi190 ceph-mon[33748]: 3.8 scrub ok 2024-03-24T12:49:07.396 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:07.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:07.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:07.498 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:07.525 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:07.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:07.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:07.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:07.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:07.651 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:07.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:07.752 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:07.780 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:07.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:07.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:07.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:07.908 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:07.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:07.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:49:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:49:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:49:08.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:08.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:08.034 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:08.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:08.135 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:08.160 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:08.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:08.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:08.286 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:08.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:08.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:08 smithi190 ceph-mon[33748]: pgmap v1014: 158 pgs: 1 active+clean+scrubbing, 54 active+undersized, 40 active+undersized+degraded, 63 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 573 B/s rd, 0 B/s wr, 789 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:08.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:08 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:49:08.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:08 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:49:08.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:08.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:08.413 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:08.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:08.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:08 smithi029 ceph-mon[41500]: pgmap v1014: 158 pgs: 1 active+clean+scrubbing, 54 active+undersized, 40 active+undersized+degraded, 63 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 573 B/s rd, 0 B/s wr, 789 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:08.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:08 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:49:08.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:08 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:49:08.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:08 smithi029 ceph-mon[29919]: pgmap v1014: 158 pgs: 1 active+clean+scrubbing, 54 active+undersized, 40 active+undersized+degraded, 63 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 573 B/s rd, 0 B/s wr, 789 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:08.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:08 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:49:08.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:08 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:49:08.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:08.539 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:08.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:08.640 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:08.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:08.670 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:08.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:08.738 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-24T12:49:08.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:08.804 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:08.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:08.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:08.906 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:08.934 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:08.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:08.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:09.027 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:49:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:49:09.036 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:09.063 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:09.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:09.165 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:09.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:09.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:09.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:09.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:09.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:09.321 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:09.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:09.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:09 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:49:09.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:09 smithi190 ceph-mon[33748]: 4.4 scrub ok 2024-03-24T12:49:09.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:09 smithi190 ceph-mon[33748]: 11.a scrub starts 2024-03-24T12:49:09.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:09 smithi190 ceph-mon[33748]: 11.a scrub ok 2024-03-24T12:49:09.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:09 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:49:09.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:09 smithi029 ceph-mon[29919]: 4.4 scrub ok 2024-03-24T12:49:09.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:09 smithi029 ceph-mon[29919]: 11.a scrub starts 2024-03-24T12:49:09.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:09 smithi029 ceph-mon[29919]: 11.a scrub ok 2024-03-24T12:49:09.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:09 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:49:09.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:09 smithi029 ceph-mon[41500]: 4.4 scrub ok 2024-03-24T12:49:09.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:09 smithi029 ceph-mon[41500]: 11.a scrub starts 2024-03-24T12:49:09.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:09 smithi029 ceph-mon[41500]: 11.a scrub ok 2024-03-24T12:49:09.422 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:09.448 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:09.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:09.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:09.550 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:09.580 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:09.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:09.682 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:09.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:09.711 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:09.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:09.812 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:09.839 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:09.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:09.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:09.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:09.967 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:09.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:09.968 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:10.069 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:10.097 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:10.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:10.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:10 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:10 smithi029 ceph-mon[41500]: pgmap v1015: 158 pgs: 54 active+undersized, 40 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 664 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:10.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:10 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:10.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:10 smithi029 ceph-mon[29919]: pgmap v1015: 158 pgs: 54 active+undersized, 40 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 664 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:10.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:10.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:10.326 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:10.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:10.360 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:10.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:10 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:10 smithi190 ceph-mon[33748]: pgmap v1015: 158 pgs: 54 active+undersized, 40 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 664 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:10.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:10.488 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:10.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:10.589 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:10.619 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:10.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:10.720 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:10.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:10.749 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:10.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:10.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:10.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:10.880 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:10.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:10.981 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:11.007 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:11.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:11.108 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:11.132 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:11.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:11.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:11.234 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:11.258 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:11.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:11.360 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:11.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:11 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 40 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:11.389 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:11.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:11.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:11.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:11 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 40 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:11.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:11 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 40 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:11.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:11.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:11.520 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:11.521 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:11.621 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:11.646 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:11.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:11.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:11.748 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:11.773 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:11.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:11.874 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:11.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:11.900 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:11.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:12.001 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:12.029 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:12.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:12.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:12.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:12.157 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:12.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:12.259 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:12.286 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:12.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:12 smithi190 ceph-mon[33748]: pgmap v1016: 158 pgs: 54 active+undersized, 40 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 858 B/s rd, 0 B/s wr, 557 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:12 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:49:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:12 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:49:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:12 smithi190 ceph-mon[33748]: osdmap e425: 8 total, 6 up, 6 in 2024-03-24T12:49:12.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:12.414 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:12.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:12.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:12 smithi029 ceph-mon[29919]: pgmap v1016: 158 pgs: 54 active+undersized, 40 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 858 B/s rd, 0 B/s wr, 557 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:12 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:49:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:12 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:49:12.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:12 smithi029 ceph-mon[29919]: osdmap e425: 8 total, 6 up, 6 in 2024-03-24T12:49:12.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:12 smithi029 ceph-mon[41500]: pgmap v1016: 158 pgs: 54 active+undersized, 40 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 858 B/s rd, 0 B/s wr, 557 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:12.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:12 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:49:12.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:12 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:49:12.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:12 smithi029 ceph-mon[41500]: osdmap e425: 8 total, 6 up, 6 in 2024-03-24T12:49:12.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:12.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:12.545 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:12.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:12.646 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:12.675 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:12.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:12.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:12.804 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:12.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:12.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:12.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:12.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:13.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:13.061 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:13.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:13.163 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:13.191 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:13.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:13.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:13.293 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:13.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:13.322 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:13.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:13 smithi190 ceph-mon[33748]: 6.c scrub starts 2024-03-24T12:49:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:13 smithi190 ceph-mon[33748]: 6.c scrub ok 2024-03-24T12:49:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:13 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:49:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:13 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:49:13.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:13 smithi029 ceph-mon[41500]: 6.c scrub starts 2024-03-24T12:49:13.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:13 smithi029 ceph-mon[41500]: 6.c scrub ok 2024-03-24T12:49:13.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:13 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:49:13.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:13 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:49:13.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:13 smithi029 ceph-mon[29919]: 6.c scrub starts 2024-03-24T12:49:13.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:13 smithi029 ceph-mon[29919]: 6.c scrub ok 2024-03-24T12:49:13.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:13 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:49:13.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:13 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:49:13.423 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:13.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:13.452 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:13.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:13.553 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:13.578 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:13.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:13.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:13.680 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:13.706 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:13.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:13.807 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:13.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:13.833 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:13.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:13.934 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:13.959 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:13.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:13.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:14.061 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:14.085 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:14.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:14.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:14.210 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:14.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:14.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:14.312 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:14.336 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:14.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:14.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[29919]: 3.e scrub starts 2024-03-24T12:49:14.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[29919]: 3.e scrub ok 2024-03-24T12:49:14.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[29919]: pgmap v1018: 158 pgs: 55 active+undersized, 39 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 270 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:14.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[29919]: 6.1f scrub starts 2024-03-24T12:49:14.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:49:14.349 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:49:14.350 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[41500]: 3.e scrub starts 2024-03-24T12:49:14.350 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[41500]: 3.e scrub ok 2024-03-24T12:49:14.350 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[41500]: pgmap v1018: 158 pgs: 55 active+undersized, 39 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 270 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:14.350 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[41500]: 6.1f scrub starts 2024-03-24T12:49:14.350 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:49:14.350 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:14 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:49:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:14 smithi190 ceph-mon[33748]: 3.e scrub starts 2024-03-24T12:49:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:14 smithi190 ceph-mon[33748]: 3.e scrub ok 2024-03-24T12:49:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:14 smithi190 ceph-mon[33748]: pgmap v1018: 158 pgs: 55 active+undersized, 39 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 270 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:14 smithi190 ceph-mon[33748]: 6.1f scrub starts 2024-03-24T12:49:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:14 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:49:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:14 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:49:14.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:14.461 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:14.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:14.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:14.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:14.587 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:14.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:14.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:14.718 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:14.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:14.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:14.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:14.843 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:14.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:14.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:14.945 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:14.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:14.971 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:14.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:15.054 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:49:15.072 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:15.096 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:15.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:15.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:15.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:15.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:15.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:15.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:15.349 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:15.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:15.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:15 smithi190 ceph-mon[33748]: 6.1f scrub ok 2024-03-24T12:49:15.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:15 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:49:15.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:15 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:49:15.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:15 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:49:15.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:15 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:49:15.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:15 smithi190 ceph-mon[33748]: osdmap e426: 8 total, 6 up, 6 in 2024-03-24T12:49:15.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:15 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4246509894' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:49:15.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[41500]: 6.1f scrub ok 2024-03-24T12:49:15.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:49:15.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:49:15.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:49:15.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:49:15.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[41500]: osdmap e426: 8 total, 6 up, 6 in 2024-03-24T12:49:15.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4246509894' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:49:15.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[29919]: 6.1f scrub ok 2024-03-24T12:49:15.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:49:15.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:49:15.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:49:15.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:49:15.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[29919]: osdmap e426: 8 total, 6 up, 6 in 2024-03-24T12:49:15.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:15 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4246509894' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:49:15.450 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:15.477 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:15.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:15.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:15.578 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:15.602 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:15.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:15.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:15.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:15.728 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:15.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:15.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:15.854 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:15.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:15.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:15.980 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:15.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:15.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:16.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:16.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:16.107 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:16.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:16.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:16.236 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:16.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:16.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:16.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:16.361 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:16.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:16.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:16 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:49:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:16 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:49:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:16 smithi190 ceph-mon[33748]: 5.6 scrub starts 2024-03-24T12:49:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:16 smithi190 ceph-mon[33748]: 5.6 scrub ok 2024-03-24T12:49:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:16 smithi190 ceph-mon[33748]: pgmap v1020: 158 pgs: 58 active+undersized, 36 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:16 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:49:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:16 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4246509894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:49:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:16 smithi190 ceph-mon[33748]: osdmap e427: 8 total, 6 up, 6 in 2024-03-24T12:49:16.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:16 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 36 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:16.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:49:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:49:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[41500]: 5.6 scrub starts 2024-03-24T12:49:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[41500]: 5.6 scrub ok 2024-03-24T12:49:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[41500]: pgmap v1020: 158 pgs: 58 active+undersized, 36 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:49:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4246509894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:49:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[41500]: osdmap e427: 8 total, 6 up, 6 in 2024-03-24T12:49:16.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 36 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:16.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:49:16.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:49:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[29919]: 5.6 scrub starts 2024-03-24T12:49:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[29919]: 5.6 scrub ok 2024-03-24T12:49:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[29919]: pgmap v1020: 158 pgs: 58 active+undersized, 36 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-24T12:49:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4246509894' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:49:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[29919]: osdmap e427: 8 total, 6 up, 6 in 2024-03-24T12:49:16.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:16 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 36 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:16.463 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:16.492 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:16.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:16.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:16.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:16.621 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:16.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:16.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:16.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:16.747 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:16.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:16.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:16.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:16.873 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:16.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:16.975 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:17.003 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:17.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:17.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:17.128 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:17.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:17.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:17.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:17.254 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:17.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:17.356 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:17.384 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:17.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:17.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:17.510 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:17.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:17.612 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:17.640 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:17.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:17.742 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:17.769 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:17.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:17.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:17.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:17.897 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:17.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:17.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:49:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:49:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:49:17.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:18.023 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:18.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:18.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:18.153 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:18.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:18.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:18.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:18.279 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:18.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:18.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:18.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:18 smithi029 ceph-mon[29919]: 5.19 scrub starts 2024-03-24T12:49:18.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:18 smithi029 ceph-mon[29919]: 5.19 scrub ok 2024-03-24T12:49:18.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:18 smithi029 ceph-mon[29919]: pgmap v1022: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:18.322 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:18 smithi029 ceph-mon[41500]: 5.19 scrub starts 2024-03-24T12:49:18.322 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:18 smithi029 ceph-mon[41500]: 5.19 scrub ok 2024-03-24T12:49:18.322 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:18 smithi029 ceph-mon[41500]: pgmap v1022: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:18.330 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:18.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:18.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:18 smithi190 ceph-mon[33748]: 5.19 scrub starts 2024-03-24T12:49:18.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:18 smithi190 ceph-mon[33748]: 5.19 scrub ok 2024-03-24T12:49:18.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:18 smithi190 ceph-mon[33748]: pgmap v1022: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:18.407 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:18.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:18.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:18.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:18.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:18.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:18.634 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:18.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:18.663 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:18.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:18.765 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:18.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:18.890 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:18.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:49:18 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:49:18.659+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:49:18.919 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:18.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:19.021 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:19.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:19.049 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:19.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:19.050 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:49:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:49:19.150 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:19.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:19.178 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:19.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:19.279 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:19.303 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:19.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:19.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:19 smithi190 ceph-mon[33748]: 3.2 scrub starts 2024-03-24T12:49:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:19 smithi190 ceph-mon[33748]: 3.2 scrub ok 2024-03-24T12:49:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:19 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:49:19.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:19 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:19.405 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:19.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:19 smithi029 ceph-mon[29919]: 3.2 scrub starts 2024-03-24T12:49:19.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:19 smithi029 ceph-mon[29919]: 3.2 scrub ok 2024-03-24T12:49:19.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:19 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:49:19.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:19 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:19.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:19 smithi029 ceph-mon[41500]: 3.2 scrub starts 2024-03-24T12:49:19.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:19 smithi029 ceph-mon[41500]: 3.2 scrub ok 2024-03-24T12:49:19.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:19 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:49:19.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:19 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:19.432 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:19.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:19.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:19.534 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:19.563 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:19.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:19.664 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:19.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:19.690 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:19.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:19.792 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:19.819 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:19.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:19.921 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:19.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:19.950 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:19.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:20.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:20.075 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:20.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:20.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:20.199 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:20.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:20.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:20.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:20.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:20.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:20.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:20.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[29919]: 4.d scrub starts 2024-03-24T12:49:20.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[29919]: 4.d scrub ok 2024-03-24T12:49:20.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[29919]: 3.16 deep-scrub starts 2024-03-24T12:49:20.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[29919]: 3.16 deep-scrub ok 2024-03-24T12:49:20.345 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[29919]: pgmap v1023: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:20.346 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[29919]: 6.1 deep-scrub starts 2024-03-24T12:49:20.346 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[29919]: 6.1 deep-scrub ok 2024-03-24T12:49:20.346 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:20.346 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[41500]: 4.d scrub starts 2024-03-24T12:49:20.346 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[41500]: 4.d scrub ok 2024-03-24T12:49:20.346 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[41500]: 3.16 deep-scrub starts 2024-03-24T12:49:20.346 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[41500]: 3.16 deep-scrub ok 2024-03-24T12:49:20.346 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[41500]: pgmap v1023: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:20.346 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[41500]: 6.1 deep-scrub starts 2024-03-24T12:49:20.346 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:20 smithi029 ceph-mon[41500]: 6.1 deep-scrub ok 2024-03-24T12:49:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:20 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:20 smithi190 ceph-mon[33748]: 4.d scrub starts 2024-03-24T12:49:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:20 smithi190 ceph-mon[33748]: 4.d scrub ok 2024-03-24T12:49:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:20 smithi190 ceph-mon[33748]: 3.16 deep-scrub starts 2024-03-24T12:49:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:20 smithi190 ceph-mon[33748]: 3.16 deep-scrub ok 2024-03-24T12:49:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:20 smithi190 ceph-mon[33748]: pgmap v1023: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:20 smithi190 ceph-mon[33748]: 6.1 deep-scrub starts 2024-03-24T12:49:20.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:20 smithi190 ceph-mon[33748]: 6.1 deep-scrub ok 2024-03-24T12:49:20.430 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:20.498 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:20.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:20.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:20.600 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:20.628 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:20.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:20.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:20.724 INFO:teuthology.orchestra.run.smithi029.stdout:6.d 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.a 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.9 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.1e 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.1b 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.19 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.18 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.17 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.16 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.15 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.13 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:6.0 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:5.e 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:5.d 2024-03-24T12:49:20.725 INFO:teuthology.orchestra.run.smithi029.stdout:3.c 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:3.7 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:1.8 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:4.1a 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:11.6 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:3.5 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:1.a 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:3.1f 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:3.1b 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:3.6 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:5.8 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:3.12 2024-03-24T12:49:20.726 INFO:teuthology.orchestra.run.smithi029.stdout:3.11 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:3.10 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:4.3 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:2.2 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:2.0 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:11.8 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:11.b 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:4.15 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:4.7 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:4.9 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:1.3 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:3.17 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:4.a 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:11.1 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:5.10 2024-03-24T12:49:20.727 INFO:teuthology.orchestra.run.smithi029.stdout:4.1c 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:4.0 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:11.9 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:5.4 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:1.5 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:11.0 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:1.0 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:1.1 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:6.11 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:6.10 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:3.3 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:3.1 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:1.2 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:4.1f 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:4.19 2024-03-24T12:49:20.728 INFO:teuthology.orchestra.run.smithi029.stdout:1.6 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:2.1 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:11.2 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:3.b 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:3.4 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:11.5 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:3.18 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:5.1 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:3.f 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:4.11 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:4.16 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:11.7 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:4.17 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:4.1 2024-03-24T12:49:20.729 INFO:teuthology.orchestra.run.smithi029.stdout:5.1b 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:4.b 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:4.1e 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:4.12 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:4.6 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:4.5 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:4.2 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:5.11 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:5.13 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:5.16 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:5.1e 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:5.17 2024-03-24T12:49:20.730 INFO:teuthology.orchestra.run.smithi029.stdout:5.1d 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:5.15 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:5.2 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:5.3 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:3.d 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:5.5 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:3.0 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:5.b 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:5.0 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:5.7 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:5.a 2024-03-24T12:49:20.731 INFO:teuthology.orchestra.run.smithi029.stdout:5.c 2024-03-24T12:49:20.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:20.755 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:20.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:20.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:20.857 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:20.885 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:20.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:20.987 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:21.016 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:21.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:21.117 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:21.143 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:21.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:21.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:21.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:21.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:21.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:21.312 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:21 smithi029 ceph-mon[41500]: 11.d scrub starts 2024-03-24T12:49:21.312 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:21 smithi029 ceph-mon[41500]: 11.d scrub ok 2024-03-24T12:49:21.312 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:21 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 34 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:21.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:21 smithi029 ceph-mon[29919]: 11.d scrub starts 2024-03-24T12:49:21.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:21 smithi029 ceph-mon[29919]: 11.d scrub ok 2024-03-24T12:49:21.313 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:21 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 34 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:21.373 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:21 smithi190 ceph-mon[33748]: 11.d scrub starts 2024-03-24T12:49:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:21 smithi190 ceph-mon[33748]: 11.d scrub ok 2024-03-24T12:49:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:21 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 34 pgs degraded (PG_DEGRADED) 2024-03-24T12:49:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:21.402 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:21.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd set noscrub 2024-03-24T12:49:21.502 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:21.530 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:21.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:21.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:21.632 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:21.659 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:21.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:21.660 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:21.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:21.786 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:21.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:21.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:21.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:21.911 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:21.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:21.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:22.013 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:22.041 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:22.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:22.105 INFO:tasks.thrashosds.thrasher:Moving pg 8.9 from osd.5 to osd.0 2024-03-24T12:49:22.105 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@smithi190.front.sepia.ceph.com to ubuntu@smithi029.front.sepia.ceph.com 2024-03-24T12:49:22.105 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 777 /var/log/ceph 2024-03-24T12:49:22.142 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:22.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2024-03-24T12:49:22.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:22.174 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:22.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:22.189 DEBUG:teuthology.orchestra.run.smithi190:> mktemp 2024-03-24T12:49:22.261 INFO:teuthology.orchestra.run.smithi190.stdout:/tmp/tmp.c207w77zO6 2024-03-24T12:49:22.261 DEBUG:teuthology.orchestra.run.smithi190:> sudo cp /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/exp.8.9.5 /tmp/tmp.c207w77zO6 2024-03-24T12:49:22.275 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:22.331 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:22.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:22.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:22 smithi190 ceph-mon[33748]: pgmap v1024: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:22.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:22 smithi029 ceph-mon[29919]: pgmap v1024: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:22.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:22 smithi029 ceph-mon[41500]: pgmap v1024: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:22.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:22.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:22.458 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:22.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:22.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:22.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:22.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:22.708 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:22.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:22.799 DEBUG:teuthology.orchestra.run.smithi190:> sudo chmod 0666 /tmp/tmp.c207w77zO6 2024-03-24T12:49:22.810 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6 2024-03-24T12:49:22.892 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:22.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.6' 2024-03-24T12:49:22.898 DEBUG:teuthology.orchestra.remote:smithi190:/tmp/tmp.c207w77zO6 is 85MB 2024-03-24T12:49:23.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:23.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:23.322 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:23.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:23.431 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:23.490 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:23.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:23.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:23.599 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:23.667 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:23.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:23.754 INFO:teuthology.orchestra.run.smithi029.stdout:ERROR: (22) Invalid argument 2024-03-24T12:49:23.754 INFO:teuthology.orchestra.run.smithi029.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-24T12:49:23.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:23.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:23.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:23.914 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:23.979 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:23.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:23.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:24.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:24.090 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'unique_pool_4' removed 2024-03-24T12:49:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:24.199 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:24.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:24.305 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2 2024-03-24T12:49:24.343 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:24 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:49:24.343 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:24 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:49:24.343 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:24 smithi029 ceph-mon[41500]: pgmap v1025: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 576 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:24.343 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:24 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2307324475' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:49:24.344 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:24 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:49:24.344 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:24 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:49:24.344 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:24 smithi029 ceph-mon[29919]: pgmap v1025: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 576 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:24.344 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:24 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2307324475' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:49:24.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:24.346 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:24.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.2' 2024-03-24T12:49:24.402 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:24 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:49:24.402 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:24 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:49:24.402 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:24 smithi190 ceph-mon[33748]: pgmap v1025: 158 pgs: 60 active+undersized, 34 active+undersized+degraded, 64 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 576 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:24.402 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:24 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2307324475' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T12:49:24.447 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:24.522 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:24.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:24.628 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1 2024-03-24T12:49:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:24.671 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:24.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.1' 2024-03-24T12:49:24.775 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:24.835 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:24.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:24.835 INFO:teuthology.orchestra.run.smithi029.stdout:osd.1: {} 2024-03-24T12:49:24.847 INFO:teuthology.orchestra.run.smithi029.stdout:osd.2: {} 2024-03-24T12:49:24.871 INFO:teuthology.orchestra.run.smithi029.stdout:osd.3: {} 2024-03-24T12:49:24.872 DEBUG:teuthology.orchestra.run.smithi190:> rm -fr /tmp/tmp.c207w77zO6 2024-03-24T12:49:24.894 INFO:teuthology.orchestra.run.smithi029.stdout:osd.4: {} 2024-03-24T12:49:24.915 INFO:teuthology.orchestra.run.smithi029.stdout:osd.6: {} 2024-03-24T12:49:24.935 INFO:teuthology.orchestra.run.smithi029.stdout:osd.7: {} 2024-03-24T12:49:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-03-24T12:49:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-24T12:49:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-24T12:49:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-24T12:49:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-24T12:49:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-03-24T12:49:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-24T12:49:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-24T12:49:24.936 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7 2024-03-24T12:49:24.960 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:24.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:24.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.7' 2024-03-24T12:49:25.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3 2024-03-24T12:49:25.131 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: docker: command not found 2024-03-24T12:49:25.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:25.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi029 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.3' 2024-03-24T12:49:25.235 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:25.274 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:25.275 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:25.376 DEBUG:teuthology.orchestra.run.smithi190:> sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4 2024-03-24T12:49:25.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:25 smithi190 ceph-mon[33748]: 3.1d deep-scrub starts 2024-03-24T12:49:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:25 smithi190 ceph-mon[33748]: 3.1d deep-scrub ok 2024-03-24T12:49:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:25 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2307324475' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:49:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:25 smithi190 ceph-mon[33748]: osdmap e428: 8 total, 6 up, 6 in 2024-03-24T12:49:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:25 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/658809176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:49:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T12:49:25.412 INFO:teuthology.orchestra.run.smithi190.stderr:sudo: docker: command not found 2024-03-24T12:49:25.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi190 with status 1: 'sudo docker kill -s 1 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd.4' 2024-03-24T12:49:25.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:25 smithi029 ceph-mon[41500]: 3.1d deep-scrub starts 2024-03-24T12:49:25.435 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:25 smithi029 ceph-mon[41500]: 3.1d deep-scrub ok 2024-03-24T12:49:25.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:25 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2307324475' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:49:25.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:25 smithi029 ceph-mon[41500]: osdmap e428: 8 total, 6 up, 6 in 2024-03-24T12:49:25.436 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:25 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/658809176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:49:25.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:25 smithi029 ceph-mon[29919]: 3.1d deep-scrub starts 2024-03-24T12:49:25.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:25 smithi029 ceph-mon[29919]: 3.1d deep-scrub ok 2024-03-24T12:49:25.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:25 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2307324475' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T12:49:25.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:25 smithi029 ceph-mon[29919]: osdmap e428: 8 total, 6 up, 6 in 2024-03-24T12:49:25.436 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:25 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/658809176' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-24T12:49:25.465 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-24T12:49:25.474 INFO:tasks.thrashosds:joining thrashosds 2024-03-24T12:49:26.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-24T12:49:26.417 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:26.427 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:26 smithi029 ceph-mon[41500]: pgmap v1027: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 932 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:26.427 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:26 smithi029 ceph-mon[29919]: pgmap v1027: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 932 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:26.530 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv /var/log/ceph/exp.8.9.5 /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/exp.8.9.5 2024-03-24T12:49:26.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell -n osd.0 --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.8.9.5 2024-03-24T12:49:26.802 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:26 smithi190 ceph-mon[33748]: pgmap v1027: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 932 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-24T12:49:26.925 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-24T12:49:27.216 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/osd.0/config 2024-03-24T12:49:27.474 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:27 smithi029 ceph-mon[29919]: 3.9 scrub starts 2024-03-24T12:49:27.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:27 smithi029 ceph-mon[29919]: 3.9 scrub ok 2024-03-24T12:49:27.475 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:27 smithi029 ceph-mon[29919]: pgmap v1028: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 921 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:27.475 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:27 smithi029 ceph-mon[41500]: 3.9 scrub starts 2024-03-24T12:49:27.475 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:27 smithi029 ceph-mon[41500]: 3.9 scrub ok 2024-03-24T12:49:27.475 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:27 smithi029 ceph-mon[41500]: pgmap v1028: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 921 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:27 smithi190 ceph-mon[33748]: 3.9 scrub starts 2024-03-24T12:49:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:27 smithi190 ceph-mon[33748]: 3.9 scrub ok 2024-03-24T12:49:27.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:27 smithi190 ceph-mon[33748]: pgmap v1028: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 921 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:27.751 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:27.872 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:49:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:49:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:49:28.388 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:28 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 34 pgs degraded, 85 pgs undersized (PG_DEGRADED) 2024-03-24T12:49:28.388 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:28 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 34 pgs degraded, 85 pgs undersized (PG_DEGRADED) 2024-03-24T12:49:28.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:28 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 128/600 objects degraded (21.333%), 34 pgs degraded, 85 pgs undersized (PG_DEGRADED) 2024-03-24T12:49:29.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:49:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:49:29.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:29 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:29.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:29 smithi190 ceph-mon[33748]: 3.15 deep-scrub starts 2024-03-24T12:49:29.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:29 smithi190 ceph-mon[33748]: 3.15 deep-scrub ok 2024-03-24T12:49:29.552 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:29 smithi190 ceph-mon[33748]: pgmap v1029: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 921 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:29.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:29 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:29.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:29 smithi029 ceph-mon[29919]: 3.15 deep-scrub starts 2024-03-24T12:49:29.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:29 smithi029 ceph-mon[29919]: 3.15 deep-scrub ok 2024-03-24T12:49:29.602 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:29 smithi029 ceph-mon[29919]: pgmap v1029: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 921 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:29.603 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:29 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:29.603 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:29 smithi029 ceph-mon[41500]: 3.15 deep-scrub starts 2024-03-24T12:49:29.603 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:29 smithi029 ceph-mon[41500]: 3.15 deep-scrub ok 2024-03-24T12:49:29.603 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:29 smithi029 ceph-mon[41500]: pgmap v1029: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 921 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:30.116 INFO:teuthology.orchestra.run.smithi029.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-24T12:49:30.353 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:30 smithi029 ceph-mon[29919]: 3.14 scrub starts 2024-03-24T12:49:30.354 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:30 smithi029 ceph-mon[29919]: 3.14 scrub ok 2024-03-24T12:49:30.354 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:30 smithi029 ceph-mon[41500]: 3.14 scrub starts 2024-03-24T12:49:30.354 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:30 smithi029 ceph-mon[41500]: 3.14 scrub ok 2024-03-24T12:49:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:30 smithi190 ceph-mon[33748]: 3.14 scrub starts 2024-03-24T12:49:30.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:30 smithi190 ceph-mon[33748]: 3.14 scrub ok 2024-03-24T12:49:30.955 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:49:30.955 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:49:30.955 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:49:30.955 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:49:30.955 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172155 9.0 9:07da41af:::benchmark_data_smithi029_134815_object60089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e253)", 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.686738+0000", 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 597.27160548999996, 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.508465065, 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172155 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.956 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686738+0000", 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686738+0000", 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686739+0000", 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5209999999999999e-06 2024-03-24T12:49:30.957 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686744+0000", 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7589999999999997e-06 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686755+0000", 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0711e-05 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:30.958 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686763+0000", 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8010000000000004e-06 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686822+0000", 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9033000000000002e-05 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.686972+0000", 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000149894 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.959 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.687082+0000", 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000110303 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.689792+0000", 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027105240000000002 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.692544+0000", 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0027517980000000002 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.751725+0000", 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.059180952000000002 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.751741+0000", 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5579000000000001e-05 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738213+0000", 2024-03-24T12:49:30.961 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.98647265900000003 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738227+0000", 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3541e-05 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.195103+0000", 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45687653900000003 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.962 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.195203+0000", 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9450999999999994e-05 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24989.0:172200 9.0 9:061b9bea:::benchmark_data_smithi029_134815_object60134:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e254)", 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:39:33.785371+0000", 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 597.17297262800002, 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.419709179, 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:30.963 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24989", 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/1732825190", 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 172200 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785371+0000", 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:30.964 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785371+0000", 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785372+0000", 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4789999999999999e-06 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785378+0000", 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2260000000000002e-06 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.965 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785390+0000", 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1697e-05 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785399+0000", 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1880000000000008e-06 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785495+0000", 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6033999999999996e-05 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.966 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:33.785502+0000", 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5849999999999996e-06 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738257+0000", 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.95275513099999998 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:34.738266+0000", 2024-03-24T12:49:30.967 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0219999999999993e-06 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.196663+0000", 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.45839733799999999 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.196781+0000", 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000118023 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,2,6", 2024-03-24T12:49:30.968 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.196907+0000", 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012570600000000001 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202007+0000", 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0050995379999999998 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202150+0000", 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000142955 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.969 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.202201+0000", 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1745e-05 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.204877+0000", 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026757270000000001 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.204947+0000", 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9937000000000005e-05 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.970 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:39:35.205080+0000", 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000132848 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214257 10.14 10:2ceef5e0:::benchmark_data_smithi029_160664_object87283:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.675879+0000", 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.28246479199998, 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8521024619999999, 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:30.971 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214257 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675879+0000", 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:30.972 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675879+0000", 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675880+0000", 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.392e-06 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675886+0000", 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7270000000000004e-06 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.973 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675895+0000", 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2289999999999992e-06 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675905+0000", 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8400000000000007e-06 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.676001+0000", 2024-03-24T12:49:30.974 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6440999999999994e-05 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.676009+0000", 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9889999999999992e-06 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.527872+0000", 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8518627940000001 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:30.975 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.527981+0000", 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010904999999999999 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214269 10.14 10:2d22cebe:::benchmark_data_smithi029_160664_object87295:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.675923+0000", 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.28241989499998, 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.873380117, 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:30.976 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214269 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675923+0000", 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675923+0000", 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.977 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675924+0000", 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6100000000000001e-07 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675927+0000", 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7539999999999998e-06 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:30.978 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675935+0000", 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2390000000000002e-06 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.675942+0000", 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7560000000000002e-06 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.676030+0000", 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8160999999999998e-05 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.979 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.676035+0000", 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.356e-06 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.528231+0000", 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8521956849999999 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for missing object", 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.528328+0000", 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6831999999999997e-05 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.980 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549214+0000", 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.020886693000000001 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549304+0000", 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9179999999999997e-05 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.981 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214277 10.14 10:2bb49b9c:::benchmark_data_smithi029_160664_object87303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.684362+0000", 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.27398171700003, 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.85330631, 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214277 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.982 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684362+0000", 2024-03-24T12:49:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684362+0000", 2024-03-24T12:49:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.983 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684364+0000", 2024-03-24T12:49:30.984 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8920000000000001e-06 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684370+0000", 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0580000000000003e-06 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684382+0000", 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2167999999999999e-05 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.985 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684389+0000", 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7179999999999996e-06 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684452+0000", 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2269999999999998e-05 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.684461+0000", 2024-03-24T12:49:30.986 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4599999999999992e-06 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.528389+0000", 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.843927989 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.528485+0000", 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6043000000000002e-05 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.987 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.528752+0000", 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026698599999999998 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.531958+0000", 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0032062269999999999 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534548+0000", 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002589694 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.988 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537515+0000", 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0029667629999999999 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537568+0000", 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3471999999999999e-05 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537668+0000", 2024-03-24T12:49:30.989 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9569999999999997e-05 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214303 10.14 10:2887a57d:::benchmark_data_smithi029_160664_object87329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.708096+0000", 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.25024750300003, 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8297232299999999, 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:30.990 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214303 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708096+0000", 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708096+0000", 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.991 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708097+0000", 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3069999999999999e-06 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708101+0000", 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.405e-06 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708110+0000", 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0129999999999999e-06 2024-03-24T12:49:30.992 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708116+0000", 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2219999999999999e-06 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708562+0000", 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044582200000000002 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.708572+0000", 2024-03-24T12:49:30.993 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0559e-05 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.530125+0000", 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8215530129999999 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.530242+0000", 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000116858 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.994 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.530479+0000", 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000237149 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534509+0000", 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0040294320000000003 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534589+0000", 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0791000000000006e-05 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.995 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537678+0000", 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0030884240000000002 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537723+0000", 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5048e-05 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537819+0000", 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6187000000000006e-05 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.996 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214310 10.14 10:2f4ebc59:::benchmark_data_smithi029_160664_object87336:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.714353+0000", 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.243990666, 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.82368043, 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214310 2024-03-24T12:49:30.997 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714353+0000", 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714353+0000", 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.998 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714354+0000", 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.483e-06 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714360+0000", 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8889999999999999e-06 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714403+0000", 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3237e-05 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:30.999 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.714413+0000", 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.944e-06 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.718231+0000", 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003818213 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.718244+0000", 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2064e-05 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.000 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.531998+0000", 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8137548749999999 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.532132+0000", 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133435 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.532382+0000", 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025042400000000001 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.001 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534635+0000", 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0022523920000000002 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537828+0000", 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0031931860000000002 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537876+0000", 2024-03-24T12:49:31.002 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8340000000000001e-05 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.537933+0000", 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6700999999999998e-05 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538033+0000", 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000100247 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.003 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214355 10.14 10:2f432cd7:::benchmark_data_smithi029_160664_object87381:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.747009+0000", 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.21133397, 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7913908009999999, 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214355 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.004 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747009+0000", 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747009+0000", 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747011+0000", 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2190000000000001e-06 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.005 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747015+0000", 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7849999999999999e-06 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747061+0000", 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5359999999999999e-05 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747070+0000", 2024-03-24T12:49:31.006 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5079999999999994e-06 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747132+0000", 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1600000000000007e-05 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.747142+0000", 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0634e-05 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.007 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534668+0000", 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.787525668 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.534802+0000", 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000134225 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.535024+0000", 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000221862 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.008 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538084+0000", 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003059562 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538142+0000", 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8168999999999998e-05 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538242+0000", 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010018600000000001 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.009 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538286+0000", 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3784e-05 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538400+0000", 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000114239 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.010 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214453 10.14 10:2d8c7f68:::benchmark_data_smithi029_160664_object87479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.824146+0000", 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.13419771500003, 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7145637759999999, 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214453 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.011 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824146+0000", 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824146+0000", 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824147+0000", 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5e-06 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.012 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824152+0000", 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2009999999999998e-06 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824163+0000", 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0815000000000001e-05 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824171+0000", 2024-03-24T12:49:31.013 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1270000000000003e-06 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824230+0000", 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8696999999999997e-05 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.824239+0000", 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.3039999999999994e-06 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.014 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.536110+0000", 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7118707930000001 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.536215+0000", 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000105368 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.536411+0000", 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000195935 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.015 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538414+0000", 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020029729999999999 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538461+0000", 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6696999999999997e-05 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538513+0000", 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1714e-05 2024-03-24T12:49:31.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538566+0000", 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3233999999999999e-05 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538709+0000", 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000143418 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.017 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214474 10.14 10:29041696:::benchmark_data_smithi029_160664_object87500:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.841093+0000", 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.11725018099997, 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7080451330000002, 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214474 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.018 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841093+0000", 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841093+0000", 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841094+0000", 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7800000000000001e-07 2024-03-24T12:49:31.019 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841099+0000", 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5909999999999998e-06 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841110+0000", 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1664999999999999e-05 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.020 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841118+0000", 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9859999999999994e-06 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841189+0000", 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0298999999999999e-05 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.841197+0000", 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.568e-06 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.021 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538749+0000", 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6975519669999999 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.538876+0000", 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012745700000000001 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.539126+0000", 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024978899999999999 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.022 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.545132+0000", 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0060055610000000004 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.548627+0000", 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0034948169999999999 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.548973+0000", 2024-03-24T12:49:31.023 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00034671000000000002 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549025+0000", 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1919999999999998e-05 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549138+0000", 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113025 2024-03-24T12:49:31.024 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214487 10.14 10:2b7e884d:::benchmark_data_smithi029_160664_object87513:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.850412+0000", 2024-03-24T12:49:31.025 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.10793087299999, 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.69917728, 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214487 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850412+0000", 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.026 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850412+0000", 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850414+0000", 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0109999999999999e-06 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.027 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850417+0000", 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0299999999999998e-06 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850424+0000", 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4569999999999999e-06 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850430+0000", 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8769999999999998e-06 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.028 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850499+0000", 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9182000000000003e-05 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.850508+0000", 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7600000000000008e-06 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.540291+0000", 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6897836399999999 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.029 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.540398+0000", 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106465 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.540593+0000", 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019509900000000001 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549150+0000", 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0085569410000000002 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549360+0000", 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020971799999999999 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549437+0000", 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6941999999999996e-05 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549490+0000", 2024-03-24T12:49:31.031 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3306000000000001e-05 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549590+0000", 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9852000000000006e-05 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214509 10.14 10:2d062f92:::benchmark_data_smithi029_160664_object87535:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.869591+0000", 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.08875280400002, 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6836662840000001, 2024-03-24T12:49:31.032 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214509 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869591+0000", 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.033 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869591+0000", 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869592+0000", 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5850000000000001e-06 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869597+0000", 2024-03-24T12:49:31.034 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2950000000000001e-06 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869611+0000", 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3186e-05 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869620+0000", 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4809999999999995e-06 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.035 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869683+0000", 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.2731999999999995e-05 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.869693+0000", 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0645000000000001e-05 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.541935+0000", 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6722419739999999 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.036 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.542069+0000", 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013380299999999999 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.542284+0000", 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021516600000000001 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549161+0000", 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0068762490000000001 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549650+0000", 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00048899499999999999 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553089+0000", 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0034392509999999999 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553148+0000", 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9278999999999999e-05 2024-03-24T12:49:31.038 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553257+0000", 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000108643 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214518 10.14 10:2c5d038c:::benchmark_data_smithi029_160664_object87544:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.874901+0000", 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.08344201099999, 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.678556089, 2024-03-24T12:49:31.039 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214518 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874901+0000", 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.040 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874901+0000", 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874903+0000", 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2580000000000001e-06 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874907+0000", 2024-03-24T12:49:31.041 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2660000000000003e-06 2024-03-24T12:49:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874929+0000", 2024-03-24T12:49:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.2572999999999999e-05 2024-03-24T12:49:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.042 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874938+0000", 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4570000000000005e-06 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.874999+0000", 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0687999999999999e-05 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.875012+0000", 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3312000000000001e-05 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.043 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.543447+0000", 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6684352900000001 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.543566+0000", 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011909 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.543771+0000", 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020472900000000001 2024-03-24T12:49:31.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549167+0000", 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0053959000000000003 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549709+0000", 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00054198799999999997 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553306+0000", 2024-03-24T12:49:31.045 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0035967490000000002 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553356+0000", 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9925000000000001e-05 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553457+0000", 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000101864 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.046 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214533 10.14 10:2be34e47:::benchmark_data_smithi029_160664_object87559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.888437+0000", 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.06990599, 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.665250957, 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214533 2024-03-24T12:49:31.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888437+0000", 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888437+0000", 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.048 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888439+0000", 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4050000000000001e-06 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888444+0000", 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9309999999999999e-06 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888459+0000", 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4902999999999999e-05 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.049 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888468+0000", 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9749999999999996e-06 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888523+0000", 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5294999999999998e-05 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.888532+0000", 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5680000000000005e-06 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.050 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.545170+0000", 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6566373140000001 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.545291+0000", 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000121144 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.545514+0000", 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000223229 2024-03-24T12:49:31.051 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549173+0000", 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0036587550000000001 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549777+0000", 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00060382399999999998 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.052 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553540+0000", 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0037631269999999998 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553593+0000", 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2966000000000001e-05 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553688+0000", 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5520999999999996e-05 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.053 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214541 10.14 10:2d9371c9:::benchmark_data_smithi029_160664_object87567:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.898613+0000", 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.05973000799997, 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6554179869999999, 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214541 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.054 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898613+0000", 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898613+0000", 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.055 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898615+0000", 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.548e-06 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898620+0000", 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1869999999999996e-06 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898634+0000", 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3512999999999999e-05 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.056 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898641+0000", 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4109999999999999e-06 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898695+0000", 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.3850000000000001e-05 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.898704+0000", 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9099999999999994e-06 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.057 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.547035+0000", 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6483308700000001 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.547162+0000", 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000127198 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.547394+0000", 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023246599999999999 2024-03-24T12:49:31.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549303+0000", 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001908397 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553500+0000", 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0041976440000000004 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553742+0000", 2024-03-24T12:49:31.059 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000241747 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.553795+0000", 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2720999999999999e-05 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.554031+0000", 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023652499999999999 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214561 10.14 10:2d91c4e6:::benchmark_data_smithi029_160664_object87587:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.926538+0000", 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.03180539700003, 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6337680830000001, 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214561 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.061 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926538+0000", 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926538+0000", 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.062 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926540+0000", 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5570000000000001e-06 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926546+0000", 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1879999999999999e-06 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926558+0000", 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2166e-05 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.063 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926566+0000", 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6899999999999992e-06 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926634+0000", 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8573999999999998e-05 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.926645+0000", 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0539e-05 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.064 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549826+0000", 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6231816079999999 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.549951+0000", 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012484400000000001 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.550213+0000", 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026170399999999997 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.065 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.557401+0000", 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0071880670000000002 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.558955+0000", 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0015537400000000001 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.560214+0000", 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0012589719999999999 2024-03-24T12:49:31.066 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.560266+0000", 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2240000000000001e-05 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.560306+0000", 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0194000000000002e-05 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.067 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214562 10.14 10:29c2d725:::benchmark_data_smithi029_160664_object87588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e349)", 2024-03-24T12:49:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:26.927210+0000", 2024-03-24T12:49:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 304.031133152, 2024-03-24T12:49:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.634147952, 2024-03-24T12:49:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.068 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214562 2024-03-24T12:49:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.069 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927210+0000", 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927210+0000", 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927211+0000", 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9699999999999995e-07 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.070 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927215+0000", 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3229999999999999e-06 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927225+0000", 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5820000000000008e-06 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927233+0000", 2024-03-24T12:49:31.071 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6629999999999993e-06 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927288+0000", 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.533e-05 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:26.927298+0000", 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8579999999999995e-06 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.072 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.551528+0000", 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.624230362 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.551640+0000", 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000112281 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 2,4", 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.551868+0000", 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00022806899999999999 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.073 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.558961+0000", 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0070923169999999999 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.559809+0000", 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00084865600000000004 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.561263+0000", 2024-03-24T12:49:31.074 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001453661 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.561318+0000", 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5315999999999997e-05 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.561358+0000", 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9737000000000003e-05 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.075 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:4500 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 93828934254464 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:24.983071+0000", 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 65.975272293000003, 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0252403349999999, 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4500 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.076 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.983071+0000", 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.983071+0000", 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.983073+0000", 2024-03-24T12:49:31.077 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.544e-06 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.983078+0000", 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7590000000000003e-06 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.983090+0000", 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1575e-05 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.078 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.983099+0000", 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5089999999999999e-06 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.999321+0000", 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.016221576000000001 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op must wait for map", 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.999327+0000", 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9900000000000002e-06 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.079 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.039467+0000", 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.040139975000000001 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.039476+0000", 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2380000000000003e-06 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.003321+0000", 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.96384442699999995 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.080 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.003487+0000", 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016681700000000001 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.008311+0000", 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0048239249999999997 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.081 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:162705 11.7 11:e34c2cd9:::benchmark_data_smithi029_192894_object57770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:25.041129+0000", 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 65.917214329999993, 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.98868872799999996, 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 162705 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.082 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.041129+0000", 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.041129+0000", 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.041130+0000", 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1119999999999999e-06 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.083 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.041135+0000", 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1900000000000003e-06 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.041144+0000", 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6640000000000004e-06 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.041151+0000", 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9149999999999999e-06 2024-03-24T12:49:31.084 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.041201+0000", 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9611000000000003e-05 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.041206+0000", 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8390000000000003e-06 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.026871+0000", 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.98566485199999998 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.027031+0000", 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000159877 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4", 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.027181+0000", 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015023700000000001 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.029605+0000", 2024-03-24T12:49:31.086 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002424093 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.029621+0000", 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5709000000000002e-05 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.029723+0000", 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010197 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.029818+0000", 2024-03-24T12:49:31.087 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.4659000000000003e-05 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop(client.25328.0:173917 11.a e419/382 11:52abe032:::benchmark_data_smithi029_192894_object68982:head v 419'11001, mlcod=419'10998)", 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:33.985107+0000", 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 56.973236849000003, 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.22903121900000001, 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.088 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985107+0000", 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985107+0000", 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985107+0000", 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1100000000000005e-07 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.089 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985113+0000", 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9270000000000002e-06 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985123+0000", 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6010000000000001e-06 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985130+0000", 2024-03-24T12:49:31.090 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7009999999999999e-06 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.986258+0000", 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0011282969999999999 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.986277+0000", 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8658e-05 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:34.214086+0000", 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.22780928 2024-03-24T12:49:31.091 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:34.214138+0000", 2024-03-24T12:49:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1944000000000003e-05 2024-03-24T12:49:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:31.092 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:31.092 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:49:31.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:31 smithi190 ceph-mon[33748]: pgmap v1030: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 921 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:31.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:31 smithi029 ceph-mon[29919]: pgmap v1030: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 921 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:31.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:31 smithi029 ceph-mon[41500]: pgmap v1030: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 921 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:33.449 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-24T12:49:33.561 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:49:33.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:49:33.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[41500]: osdmap e429: 8 total, 6 up, 6 in 2024-03-24T12:49:33.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[41500]: 6.1c scrub starts 2024-03-24T12:49:33.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[41500]: 6.1c scrub ok 2024-03-24T12:49:33.562 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[41500]: pgmap v1032: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 577 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:33.562 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:49:33.562 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:49:33.562 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[29919]: osdmap e429: 8 total, 6 up, 6 in 2024-03-24T12:49:33.562 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[29919]: 6.1c scrub starts 2024-03-24T12:49:33.562 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[29919]: 6.1c scrub ok 2024-03-24T12:49:33.563 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-mon[29919]: pgmap v1032: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 577 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:33 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:49:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:33 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:49:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:33 smithi190 ceph-mon[33748]: osdmap e429: 8 total, 6 up, 6 in 2024-03-24T12:49:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:33 smithi190 ceph-mon[33748]: 6.1c scrub starts 2024-03-24T12:49:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:33 smithi190 ceph-mon[33748]: 6.1c scrub ok 2024-03-24T12:49:33.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:33 smithi190 ceph-mon[33748]: pgmap v1032: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 577 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:33.899 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:49:33 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:49:33.669+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:49:34.253 INFO:teuthology.orchestra.run.smithi029.stderr:Pool 8 no longer exists 2024-03-24T12:49:34.296 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:34.527 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:34.527 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:34 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:49:34.528 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:34 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:49:34.528 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:34.528 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:34 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:49:34.528 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:34 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:49:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:34 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:49:34.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:34 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:49:35.321 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is set 2024-03-24T12:49:35.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:35 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/914599442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:49:35.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:35 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:49:35.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:35 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:49:35.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:35 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:49:35.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:35 smithi029 ceph-mon[41500]: pgmap v1033: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:35 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/914599442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:49:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:35 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:49:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:35 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:49:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:35 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:49:35.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:35 smithi029 ceph-mon[29919]: pgmap v1033: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:35 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/914599442' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:49:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:35 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-24T12:49:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:35 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:49:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:35 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:49:35.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:35 smithi190 ceph-mon[33748]: pgmap v1033: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:36.386 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:36 smithi029 ceph-mon[41500]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:49:36.386 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:36 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:49:36.387 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:36 smithi029 ceph-mon[41500]: osdmap e430: 8 total, 6 up, 6 in 2024-03-24T12:49:36.387 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:36 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:49:36.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:36 smithi029 ceph-mon[29919]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:49:36.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:36 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:49:36.388 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:36 smithi029 ceph-mon[29919]: osdmap e430: 8 total, 6 up, 6 in 2024-03-24T12:49:36.388 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:36 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:49:36.411 DEBUG:teuthology.orchestra.run:got remote process result: 10 2024-03-24T12:49:36.411 INFO:tasks.thrashosds.thrasher:Pool went away before processing an import...ignored 2024-03-24T12:49:36.412 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/exp.8.9.5 2024-03-24T12:49:36.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/exp.8.9.5 2024-03-24T12:49:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:36 smithi190 ceph-mon[33748]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-24T12:49:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:36 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-24T12:49:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:36 smithi190 ceph-mon[33748]: osdmap e430: 8 total, 6 up, 6 in 2024-03-24T12:49:36.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:36 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:49:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:49:37.574 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:49:37.574 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-03-24T12:49:37.574 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:49:37.575 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:37 smithi029 ceph-mon[29919]: pgmap v1035: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:37.575 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:37 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:49:37.575 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:37 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:49:37.575 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:37 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:49:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:37 smithi029 ceph-mon[41500]: pgmap v1035: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:37 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:49:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:37 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:49:37.575 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:37 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:49:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:37 smithi190 ceph-mon[33748]: pgmap v1035: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:37 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:49:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:37 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:49:37.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:37 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:49:37.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:49:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:49:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:49:38.280 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-24T12:49:38.533 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:39.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset noscrub 2024-03-24T12:49:39.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:49:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:49:39.915 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:40.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:40.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:39 smithi029 ceph-mon[29919]: pgmap v1036: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:40.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:40.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:39 smithi029 ceph-mon[41500]: pgmap v1036: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:40.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:40.248 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:39 smithi190 ceph-mon[33748]: pgmap v1036: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:41.139 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:49:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:49:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-03-24T12:49:41.139 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-03-24T12:49:41.139 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:49:41.524 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-24T12:49:41.524 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-03-24T12:49:41.524 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-03-24T12:49:41.524 DEBUG:teuthology.orchestra.run.smithi029:osd.0> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T12:49:41.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl start ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.0 2024-03-24T12:49:41.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:41 smithi029 ceph-mon[41500]: pgmap v1037: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:41.551 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:41 smithi029 ceph-mon[29919]: pgmap v1037: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:41.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:41 smithi190 ceph-mon[33748]: pgmap v1037: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:41.879 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:41 smithi029 systemd[1]: Starting Ceph osd.0 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:49:42.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-24T12:49:43.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:42 smithi029 podman[216194]: 2024-03-24 12:49:42.755071837 +0000 UTC m=+0.104740490 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:49:43.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 podman[216194]: 2024-03-24 12:49:43.019775944 +0000 UTC m=+0.369444559 container create 59d79cc1e50546203ea376a875339da043eeae4d92df15bd042ed614f269acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:49:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:43.558 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 podman[216194]: 2024-03-24 12:49:43.276292651 +0000 UTC m=+0.625961282 container init 59d79cc1e50546203ea376a875339da043eeae4d92df15bd042ed614f269acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T12:49:43.558 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 podman[216194]: 2024-03-24 12:49:43.281904158 +0000 UTC m=+0.631572774 container start 59d79cc1e50546203ea376a875339da043eeae4d92df15bd042ed614f269acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:49:43.558 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 podman[216194]: 2024-03-24 12:49:43.401276914 +0000 UTC m=+0.750945529 container attach 59d79cc1e50546203ea376a875339da043eeae4d92df15bd042ed614f269acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:49:43.997 INFO:teuthology.orchestra.run.smithi029.stderr:noscrub is unset 2024-03-24T12:49:44.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-mon[29919]: pgmap v1038: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1830477692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:49:44.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:49:44.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-mon[41500]: pgmap v1038: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:44.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1830477692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:49:44.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:49:44.167 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate[216265]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate[216265]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 bash[216194]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 bash[216194]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 bash[216194]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 bash[216194]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 bash[216194]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 bash[216194]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 bash[216194]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate[216265]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate[216265]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate[216265]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate[216265]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-24T12:49:44.168 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate[216265]: --> ceph-volume raw activate successful for osd ID: 0 2024-03-24T12:49:44.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:43 smithi190 ceph-mon[33748]: pgmap v1038: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:44.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:43 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1830477692' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:49:44.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:43 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-24T12:49:44.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-24T12:49:45.111 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:44 smithi029 ceph-mon[41500]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-24T12:49:45.111 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:44 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:49:45.111 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:44 smithi029 ceph-mon[41500]: osdmap e431: 8 total, 6 up, 6 in 2024-03-24T12:49:45.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:44 smithi029 ceph-mon[29919]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-24T12:49:45.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:44 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:49:45.111 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:44 smithi029 ceph-mon[29919]: osdmap e431: 8 total, 6 up, 6 in 2024-03-24T12:49:45.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:44 smithi190 ceph-mon[33748]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-24T12:49:45.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:44 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-24T12:49:45.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:44 smithi190 ceph-mon[33748]: osdmap e431: 8 total, 6 up, 6 in 2024-03-24T12:49:45.805 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:45 smithi029 podman[216194]: 2024-03-24 12:49:45.493431925 +0000 UTC m=+2.843100540 container died 59d79cc1e50546203ea376a875339da043eeae4d92df15bd042ed614f269acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:49:46.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:45 smithi029 ceph-mon[41500]: pgmap v1040: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:45 smithi029 ceph-mon[29919]: pgmap v1040: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:46.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:45 smithi190 ceph-mon[33748]: pgmap v1040: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:46.423 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:46 smithi029 podman[216465]: 2024-03-24 12:49:46.172699268 +0000 UTC m=+2.304248917 container remove 59d79cc1e50546203ea376a875339da043eeae4d92df15bd042ed614f269acfb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-activate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T12:49:46.704 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214802 10.18 10:1f6ecfac:::benchmark_data_smithi029_160664_object87828:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:28.681877+0000", 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 318.02446185700001, 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4605043090000001, 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.705 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214802 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681877+0000", 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.706 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681877+0000", 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681878+0000", 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.92e-07 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681882+0000", 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1880000000000004e-06 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.707 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681890+0000", 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8099999999999998e-06 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681897+0000", 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3669999999999999e-06 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.695675+0000", 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.013777742000000001 2024-03-24T12:49:46.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.695684+0000", 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0089999999999996e-06 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.137187+0000", 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4415030309999999 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.709 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.137299+0000", 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011141099999999999 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.137575+0000", 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000276493 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.140510+0000", 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0029348019999999998 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.710 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.711 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.711 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142051+0000", 2024-03-24T12:49:46.711 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001541398 2024-03-24T12:49:46.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.711 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.711 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.711 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142239+0000", 2024-03-24T12:49:46.711 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018788999999999999 2024-03-24T12:49:46.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.711 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.712 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.712 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142330+0000", 2024-03-24T12:49:46.712 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1028999999999999e-05 2024-03-24T12:49:46.712 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.712 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.712 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.712 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142381+0000", 2024-03-24T12:49:46.712 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1347000000000002e-05 2024-03-24T12:49:46.712 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214803 10.18 10:1b334f18:::benchmark_data_smithi029_160664_object87829:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e350)", 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:28.681915+0000", 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 318.02442404999999, 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4606544000000001, 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.713 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214803 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681915+0000", 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681915+0000", 2024-03-24T12:49:46.714 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681916+0000", 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9500000000000002e-07 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681918+0000", 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7839999999999999e-06 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.715 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681925+0000", 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.99e-06 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.681932+0000", 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4389999999999997e-06 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.695711+0000", 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.013778874 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.716 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.695717+0000", 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0759999999999999e-06 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.139184+0000", 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.443467133 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.139297+0000", 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00011353600000000001 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.717 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.139522+0000", 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00022444699999999999 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142399+0000", 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028770100000000002 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142452+0000", 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2726000000000003e-05 2024-03-24T12:49:46.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142491+0000", 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9279999999999997e-05 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142535+0000", 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4166999999999999e-05 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.719 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142569+0000", 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4343e-05 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214820 10.18 10:1b29327d:::benchmark_data_smithi029_160664_object87846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:28.704760+0000", 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 318.00157854600002, 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4395860709999999, 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.720 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214820 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.704760+0000", 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.704760+0000", 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.704762+0000", 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.595e-06 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.704768+0000", 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.677e-06 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.722 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.704779+0000", 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1338999999999999e-05 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.704790+0000", 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0594999999999999e-05 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.705114+0000", 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000323851 2024-03-24T12:49:46.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.705122+0000", 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4800000000000001e-06 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.140553+0000", 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.43543135 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.724 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.140668+0000", 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000114789 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.140862+0000", 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00019407699999999999 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142408+0000", 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0015460319999999999 2024-03-24T12:49:46.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.144205+0000", 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001796639 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.144248+0000", 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2880999999999998e-05 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.727 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.727 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.144303+0000", 2024-03-24T12:49:46.727 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5345e-05 2024-03-24T12:49:46.727 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.727 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.727 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.727 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.144347+0000", 2024-03-24T12:49:46.727 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3421000000000003e-05 2024-03-24T12:49:46.727 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214846 10.18 10:1bd603f7:::benchmark_data_smithi029_160664_object87872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:28.722919+0000", 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.98342040099999, 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.425267861, 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.728 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214846 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.722919+0000", 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.729 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.722919+0000", 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.722920+0000", 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3850000000000001e-06 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.722927+0000", 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5560000000000004e-06 2024-03-24T12:49:46.730 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.722942+0000", 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5814000000000001e-05 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.722952+0000", 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6299999999999993e-06 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.731 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.723017+0000", 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4622999999999994e-05 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.723026+0000", 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.9050000000000003e-06 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142622+0000", 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.419596225 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.732 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142743+0000", 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012157 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.142996+0000", 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025284800000000002 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.147908+0000", 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0049113339999999998 2024-03-24T12:49:46.733 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.148011+0000", 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000103892 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.148077+0000", 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5689000000000001e-05 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.734 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.148139+0000", 2024-03-24T12:49:46.735 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1724999999999996e-05 2024-03-24T12:49:46.735 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.735 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.736 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.736 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.148186+0000", 2024-03-24T12:49:46.736 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7664999999999998e-05 2024-03-24T12:49:46.736 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.736 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.736 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.736 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.736 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214940 10.18 10:1e45d1b8:::benchmark_data_smithi029_160664_object87966:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:28.791165+0000", 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.915174439, 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.361628558, 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214940 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.737 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.791165+0000", 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.791165+0000", 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.791166+0000", 2024-03-24T12:49:46.738 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6589999999999999e-06 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.791172+0000", 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0809999999999999e-06 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.791185+0000", 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2462e-05 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.739 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.791195+0000", 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0358e-05 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.791263+0000", 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7949999999999998e-05 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.791273+0000", 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0312999999999999e-05 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.740 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.144403+0000", 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3531296669999999 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.144526+0000", 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000122834 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.144811+0000", 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00028498299999999999 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.741 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.148025+0000", 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0032145120000000001 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.148294+0000", 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026866599999999998 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.152669+0000", 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0043753170000000001 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.742 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.152728+0000", 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8875999999999997e-05 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.152793+0000", 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4880000000000002e-05 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.743 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214943 10.18 10:1bab7a4d:::benchmark_data_smithi029_160664_object87969:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.744 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:28.792110+0000", 2024-03-24T12:49:46.744 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.91422867799997, 2024-03-24T12:49:46.744 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.360848378, 2024-03-24T12:49:46.744 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.744 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.744 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.744 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.744 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.744 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214943 2024-03-24T12:49:46.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792110+0000", 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792110+0000", 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792111+0000", 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6700000000000003e-07 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792115+0000", 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0409999999999999e-06 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.746 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792124+0000", 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.1279999999999997e-06 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792132+0000", 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9049999999999997e-06 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792294+0000", 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016180099999999999 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.747 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792303+0000", 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6559999999999998e-06 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.146346+0000", 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.354042872 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.146462+0000", 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000116469 2024-03-24T12:49:46.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.146753+0000", 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029108399999999999 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.148253+0000", 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001499989 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.749 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.152612+0000", 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0043586620000000001 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.152850+0000", 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000238073 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.152902+0000", 2024-03-24T12:49:46.750 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2417999999999999e-05 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.152959+0000", 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6512999999999999e-05 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214947 10.18 10:1f7ad202:::benchmark_data_smithi029_160664_object87973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:28.792529+0000", 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.91381010700002, 2024-03-24T12:49:46.751 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3649494870000001, 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214947 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792529+0000", 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792529+0000", 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792530+0000", 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0800000000000004e-07 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.753 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792533+0000", 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4649999999999999e-06 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792543+0000", 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0039e-05 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792551+0000", 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7910000000000005e-06 2024-03-24T12:49:46.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792608+0000", 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7229999999999999e-05 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.792616+0000", 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6480000000000003e-06 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.148330+0000", 2024-03-24T12:49:46.755 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.355714402 2024-03-24T12:49:46.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.756 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.756 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.148427+0000", 2024-03-24T12:49:46.756 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7309e-05 2024-03-24T12:49:46.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.756 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.756 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.148674+0000", 2024-03-24T12:49:46.756 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00024689500000000001 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.153031+0000", 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0043568369999999997 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.153109+0000", 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7779000000000002e-05 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.157185+0000", 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0040758340000000004 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.157424+0000", 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00023903300000000001 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.758 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.157478+0000", 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4517e-05 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214953 10.18 10:1b58b093:::benchmark_data_smithi029_160664_object87979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:28.798895+0000", 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.90744420700003, 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3701644230000001, 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.759 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214953 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.798895+0000", 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.798895+0000", 2024-03-24T12:49:46.760 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.798896+0000", 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.249e-06 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.798901+0000", 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9509999999999997e-06 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.761 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.798920+0000", 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8768000000000001e-05 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.798927+0000", 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1570000000000003e-06 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.798978+0000", 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1019000000000001e-05 2024-03-24T12:49:46.762 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.798985+0000", 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1520000000000003e-06 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.153267+0000", 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.354281485 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.153397+0000", 2024-03-24T12:49:46.763 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013034799999999999 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.153749+0000", 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00035198800000000002 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.159454+0000", 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0057051640000000004 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.764 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.161799+0000", 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0023452809999999998 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.168970+0000", 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0071704919999999997 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169017+0000", 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7352999999999999e-05 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.765 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169059+0000", 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2015999999999999e-05 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214985 10.18 10:1f4ce62e:::benchmark_data_smithi029_160664_object88011:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:28.954426+0000", 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.751913264, 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.214771958, 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.766 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214985 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.954426+0000", 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.767 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.954426+0000", 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.954427+0000", 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.421e-06 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.954432+0000", 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0409999999999997e-06 2024-03-24T12:49:46.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.954443+0000", 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.114e-05 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.954451+0000", 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9589999999999995e-06 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.954539+0000", 2024-03-24T12:49:46.769 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7960000000000005e-05 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.954548+0000", 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3270000000000001e-06 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.155279+0000", 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.200731488 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.770 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.155433+0000", 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015395 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.155855+0000", 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000421927 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.159460+0000", 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0036049509999999999 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.771 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.168874+0000", 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0094139459999999994 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169109+0000", 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000235543 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169157+0000", 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7561000000000001e-05 2024-03-24T12:49:46.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169198+0000", 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0744000000000002e-05 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:214991 10.18 10:1a9f1a6c:::benchmark_data_smithi029_160664_object88017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:28.958677+0000", 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.74766190399998, 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2106592709999999, 2024-03-24T12:49:46.773 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 214991 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.958677+0000", 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.958677+0000", 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.958678+0000", 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.139e-06 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.958682+0000", 2024-03-24T12:49:46.775 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.72e-06 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.958691+0000", 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.098e-06 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.958698+0000", 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8390000000000001e-06 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.958742+0000", 2024-03-24T12:49:46.776 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4413999999999999e-05 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:28.958749+0000", 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1840000000000004e-06 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.157515+0000", 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.198766733 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.777 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.157636+0000", 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0001208 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.157902+0000", 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026642199999999999 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.161812+0000", 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0039091459999999996 2024-03-24T12:49:46.778 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.168930+0000", 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0071180619999999997 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169250+0000", 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00031999699999999998 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.779 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169296+0000", 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6415999999999997e-05 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169336+0000", 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0300999999999999e-05 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:215004 10.18 10:1d069521:::benchmark_data_smithi029_160664_object88030:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.780 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:29.011788+0000", 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.694550564, 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1577351280000001, 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 215004 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.781 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.011788+0000", 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.011788+0000", 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.011790+0000", 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.59e-06 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.782 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.011795+0000", 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4330000000000003e-06 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.011808+0000", 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2488e-05 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.011818+0000", 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5510000000000005e-06 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.783 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.011881+0000", 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3715999999999998e-05 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.011893+0000", 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1531e-05 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.159490+0000", 2024-03-24T12:49:46.784 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1475968459999999 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.159749+0000", 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025944599999999998 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.160013+0000", 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026375500000000001 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.785 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.168824+0000", 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0088109220000000005 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169388+0000", 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00056465400000000005 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.786 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169434+0000", 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5277e-05 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169483+0000", 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9200999999999999e-05 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169524+0000", 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0717999999999999e-05 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.787 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:215027 10.18 10:1acb5f77:::benchmark_data_smithi029_160664_object88053:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:29.033397+0000", 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.672942181, 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.141252063, 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 215027 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.788 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.033397+0000", 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.033397+0000", 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.789 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.033398+0000", 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3990000000000001e-06 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.033403+0000", 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1100000000000002e-06 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.033416+0000", 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.288e-05 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.790 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.033426+0000", 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0156000000000001e-05 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.033493+0000", 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6196000000000001e-05 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.033504+0000", 2024-03-24T12:49:46.791 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.181e-05 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.161840+0000", 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.128335758 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.161953+0000", 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000113262 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.792 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.162223+0000", 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026925899999999999 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.168831+0000", 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.006608109 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174378+0000", 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0055470229999999999 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.793 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174540+0000", 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016176 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174604+0000", 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4850000000000004e-05 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174649+0000", 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4490999999999998e-05 2024-03-24T12:49:46.794 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:215030 10.18 10:1c1f4296:::benchmark_data_smithi029_160664_object88056:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:29.036821+0000", 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.66951809900002, 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.13795395, 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.795 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 215030 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.036821+0000", 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.036821+0000", 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.796 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.036822+0000", 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.313e-06 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.036828+0000", 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6339999999999998e-06 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.036843+0000", 2024-03-24T12:49:46.797 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5109000000000001e-05 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.036853+0000", 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5740000000000002e-06 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.037005+0000", 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015251699999999999 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.798 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.037017+0000", 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1487e-05 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.164180+0000", 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.127163669 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.164304+0000", 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012339100000000001 2024-03-24T12:49:46.799 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.164736+0000", 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0004328 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.168886+0000", 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0041497210000000003 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.800 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174418+0000", 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0055316159999999996 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174694+0000", 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000276359 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174738+0000", 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4360000000000002e-05 2024-03-24T12:49:46.801 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174775+0000", 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6399999999999997e-05 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:215061 10.18 10:1f204830:::benchmark_data_smithi029_160664_object88087:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:29.075299+0000", 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.631039638, 2024-03-24T12:49:46.802 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0997132700000001, 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 215061 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.075299+0000", 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.803 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.075299+0000", 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.075301+0000", 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8640000000000001e-06 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.804 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.075307+0000", 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9850000000000003e-06 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.075322+0000", 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4865000000000001e-05 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.075332+0000", 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9690000000000006e-06 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.805 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.075400+0000", 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7929000000000006e-05 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.075411+0000", 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1306e-05 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.167026+0000", 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0916145239999999 2024-03-24T12:49:46.806 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.167134+0000", 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010807299999999999 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.167398+0000", 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00026444000000000001 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169208+0000", 2024-03-24T12:49:46.807 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0018100659999999999 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174452+0000", 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0052437220000000001 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174816+0000", 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036399099999999998 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.808 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.174863+0000", 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7324000000000003e-05 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.175013+0000", 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014921199999999999 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.809 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:215098 10.18 10:1c367dcf:::benchmark_data_smithi029_160664_object88124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:29.139367+0000", 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.56697208899999, 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0454244530000001, 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 215098 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.810 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.139367+0000", 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.139367+0000", 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.139368+0000", 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3379999999999999e-06 2024-03-24T12:49:46.811 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.139373+0000", 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8810000000000004e-06 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.139383+0000", 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0233e-05 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.812 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.139390+0000", 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.046e-06 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.139442+0000", 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1100999999999997e-05 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.139449+0000", 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5669999999999997e-06 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.813 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169558+0000", 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.030109188 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.169692+0000", 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000133941 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.814 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.170042+0000", 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00034990899999999997 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.175882+0000", 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0058398579999999999 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.177801+0000", 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001919023 2024-03-24T12:49:46.815 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.816 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.816 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.184695+0000", 2024-03-24T12:49:46.817 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0068935259999999996 2024-03-24T12:49:46.817 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.817 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.817 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.817 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.184745+0000", 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0858e-05 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.184791+0000", 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5983999999999999e-05 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:215099 10.18 10:1805897c:::benchmark_data_smithi029_160664_object88125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e351)", 2024-03-24T12:49:46.818 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:44:29.141852+0000", 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 317.56448738699999, 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.043116267, 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 215099 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.819 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.141852+0000", 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.141852+0000", 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.141853+0000", 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8999999999999995e-07 2024-03-24T12:49:46.820 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.141857+0000", 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1049999999999997e-06 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.141868+0000", 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1849e-05 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.821 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.141875+0000", 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7179999999999999e-06 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.141945+0000", 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0005000000000005e-05 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:29.141967+0000", 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1552999999999998e-05 2024-03-24T12:49:46.822 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.171711+0000", 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.029743759 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.171828+0000", 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000117798 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,3", 2024-03-24T12:49:46.823 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.172117+0000", 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000288767 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.175890+0000", 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.003772484 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.184832+0000", 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0089427699999999992 2024-03-24T12:49:46.824 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.825 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.825 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.825 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.184870+0000", 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7219999999999999e-05 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.184927+0000", 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7148999999999998e-05 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:30.184968+0000", 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1199999999999999e-05 2024-03-24T12:49:46.826 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:162692 11.6 11:64f519d8:::benchmark_data_smithi029_192894_object57757:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:25.007008+0000", 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 81.699331164, 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.03529917, 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:49:46.827 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:49:46.828 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 162692 2024-03-24T12:49:46.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.828 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.828 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.828 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007008+0000", 2024-03-24T12:49:46.828 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.828 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.828 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007008+0000", 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007008+0000", 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.8200000000000001e-07 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007010+0000", 2024-03-24T12:49:46.829 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1950000000000002e-06 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007016+0000", 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6149999999999996e-06 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007021+0000", 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2390000000000002e-06 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.011383+0000", 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0043618600000000004 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.011390+0000", 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2540000000000003e-06 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.038142+0000", 2024-03-24T12:49:46.831 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0267516130000001 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for missing object", 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.038240+0000", 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.7689000000000006e-05 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.042251+0000", 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0040109309999999997 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.042307+0000", 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6391999999999998e-05 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:162698 11.8 11:14546c11:::benchmark_data_smithi029_192894_object57763:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:25.007028+0000", 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 81.699311123000001, 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0362576219999999, 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.833 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 162698 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007028+0000", 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.834 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007028+0000", 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007028+0000", 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.84e-07 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007030+0000", 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.9230000000000001e-06 2024-03-24T12:49:46.835 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007035+0000", 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1669999999999998e-06 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007041+0000", 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2209999999999996e-06 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.836 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.009715+0000", 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00267432 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.009721+0000", 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4060000000000002e-06 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.034120+0000", 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.024398723 2024-03-24T12:49:46.837 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.034259+0000", 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013865399999999999 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4", 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.034381+0000", 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012199800000000001 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.838 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.043008+0000", 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0086274609999999995 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.043069+0000", 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0545999999999999e-05 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.043190+0000", 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000121358 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.839 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.043286+0000", 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5461e-05 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:162699 11.8 11:12734890:::benchmark_data_smithi029_192894_object57764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:25.007047+0000", 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 81.699292448999998, 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0378051669999999, 2024-03-24T12:49:46.840 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 162699 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007047+0000", 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.841 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007047+0000", 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007047+0000", 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.1e-07 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007049+0000", 2024-03-24T12:49:46.842 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1119999999999999e-06 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007054+0000", 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9439999999999996e-06 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.007059+0000", 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8400000000000002e-06 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.843 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.009748+0000", 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026890989999999999 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.009754+0000", 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6159999999999998e-06 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.036356+0000", 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0266014000000001 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.844 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.036460+0000", 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010382699999999999 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4", 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.036564+0000", 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000104318 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.043294+0000", 2024-03-24T12:49:46.845 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0067295150000000001 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.044713+0000", 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0014197960000000001 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.044765+0000", 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1672999999999998e-05 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.846 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.044852+0000", 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6717000000000003e-05 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:162701 11.1 11:81601aae:::benchmark_data_smithi029_192894_object57766:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:25.010312+0000", 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 81.696026801000002, 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0313084260000001, 2024-03-24T12:49:46.847 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 162701 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.010312+0000", 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.848 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.010312+0000", 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.010314+0000", 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3289999999999999e-06 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.010318+0000", 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1880000000000004e-06 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.849 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.010328+0000", 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9089999999999995e-06 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.010337+0000", 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0410000000000003e-06 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.014686+0000", 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0043489269999999998 2024-03-24T12:49:46.850 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.014692+0000", 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4699999999999999e-06 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.034361+0000", 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.019668918 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.034469+0000", 2024-03-24T12:49:46.851 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000107879 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 4", 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.034578+0000", 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000109226 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.039034+0000", 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0044558530000000001 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:49:46.852 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.041464+0000", 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0024300089999999999 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.041530+0000", 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5827999999999994e-05 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.041621+0000", 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0848999999999997e-05 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.853 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:49:46.854 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:49:46.854 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:49:47.359 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:47 smithi029 ceph-mon[41500]: 3.1e deep-scrub starts 2024-03-24T12:49:47.360 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:47 smithi029 ceph-mon[41500]: 3.1e deep-scrub ok 2024-03-24T12:49:47.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:47 smithi029 ceph-mon[29919]: 3.1e deep-scrub starts 2024-03-24T12:49:47.360 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:47 smithi029 ceph-mon[29919]: 3.1e deep-scrub ok 2024-03-24T12:49:47.360 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:47 smithi029 podman[216662]: 2024-03-24 12:49:46.989820945 +0000 UTC m=+0.248343306 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:49:47.361 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:47 smithi029 podman[216662]: 2024-03-24 12:49:47.164380888 +0000 UTC m=+0.422903248 container create 79eb38f7cef0645ed2598e616098f61b77daff9d97771f634d5d682679d22b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:49:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:47 smithi190 ceph-mon[33748]: 3.1e deep-scrub starts 2024-03-24T12:49:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:47 smithi190 ceph-mon[33748]: 3.1e deep-scrub ok 2024-03-24T12:49:47.766 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:49:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:49:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:49:47.766 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:47 smithi029 podman[216662]: 2024-03-24 12:49:47.521809292 +0000 UTC m=+0.780331667 container init 79eb38f7cef0645ed2598e616098f61b77daff9d97771f634d5d682679d22b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T12:49:47.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:47 smithi029 podman[216662]: 2024-03-24 12:49:47.527632272 +0000 UTC m=+0.786154632 container start 79eb38f7cef0645ed2598e616098f61b77daff9d97771f634d5d682679d22b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-24T12:49:47.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:47 smithi029 ceph-osd[216740]: -- 172.21.15.29:0/2975972520 <== mon.0 v2:172.21.15.29:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55baf73d71e0 con 0x55baf73a8c00 2024-03-24T12:49:47.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:47 smithi029 bash[216662]: 79eb38f7cef0645ed2598e616098f61b77daff9d97771f634d5d682679d22b08 2024-03-24T12:49:47.767 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:47 smithi029 systemd[1]: Started Ceph osd.0 for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:49:47.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:49:47.769 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:48.075 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:48 smithi029 ceph-mon[41500]: pgmap v1041: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:48.076 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:48 smithi029 ceph-mon[41500]: 5.12 scrub starts 2024-03-24T12:49:48.076 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:48 smithi029 ceph-mon[41500]: 5.12 scrub ok 2024-03-24T12:49:48.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:48 smithi029 ceph-mon[29919]: pgmap v1041: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:48.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:48 smithi029 ceph-mon[29919]: 5.12 scrub starts 2024-03-24T12:49:48.076 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:48 smithi029 ceph-mon[29919]: 5.12 scrub ok 2024-03-24T12:49:48.379 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:48 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[216717]: 2024-03-24T12:49:48.162+0000 7f498e35e7c0 -1 Falling back to public interface 2024-03-24T12:49:48.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:48 smithi190 ceph-mon[33748]: pgmap v1041: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:48 smithi190 ceph-mon[33748]: 5.12 scrub starts 2024-03-24T12:49:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:48 smithi190 ceph-mon[33748]: 5.12 scrub ok 2024-03-24T12:49:49.003 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:49:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:49:49.004 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:49:48 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:49:48.680+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:49:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:49 smithi190 ceph-mon[33748]: 1.7 scrub starts 2024-03-24T12:49:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:49 smithi190 ceph-mon[33748]: 1.7 scrub ok 2024-03-24T12:49:49.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:49.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:49 smithi029 ceph-mon[41500]: 1.7 scrub starts 2024-03-24T12:49:49.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:49 smithi029 ceph-mon[41500]: 1.7 scrub ok 2024-03-24T12:49:49.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:49.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:49 smithi029 ceph-mon[29919]: 1.7 scrub starts 2024-03-24T12:49:49.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:49 smithi029 ceph-mon[29919]: 1.7 scrub ok 2024-03-24T12:49:49.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:49:49.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-24T12:49:49.605 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:50.221 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:50 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:50.221 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:50 smithi029 ceph-mon[41500]: 4.10 scrub starts 2024-03-24T12:49:50.221 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:50 smithi029 ceph-mon[41500]: 4.10 scrub ok 2024-03-24T12:49:50.221 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:50 smithi029 ceph-mon[41500]: pgmap v1042: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:50.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:50 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:50.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:50 smithi029 ceph-mon[29919]: 4.10 scrub starts 2024-03-24T12:49:50.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:50 smithi029 ceph-mon[29919]: 4.10 scrub ok 2024-03-24T12:49:50.221 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:50 smithi029 ceph-mon[29919]: pgmap v1042: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:50.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:50 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:50.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:50 smithi190 ceph-mon[33748]: 4.10 scrub starts 2024-03-24T12:49:50.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:50 smithi190 ceph-mon[33748]: 4.10 scrub ok 2024-03-24T12:49:50.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:50 smithi190 ceph-mon[33748]: pgmap v1042: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:51.550 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:51 smithi029 ceph-mon[29919]: pgmap v1043: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:51.550 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:51 smithi029 ceph-mon[41500]: pgmap v1043: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:51.551 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[216717]: 2024-03-24T12:49:51.447+0000 7f498e35e7c0 -1 osd.0 417 log_to_monitors true 2024-03-24T12:49:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:51 smithi190 ceph-mon[33748]: pgmap v1043: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:51.768 INFO:teuthology.orchestra.run.smithi029.stderr:no valid command found; 10 closest matches: 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:0 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:1 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:2 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:abort 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:assert 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:bluefs debug_inject_read_zeros 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:bluefs files list 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:bluefs stats 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:bluestore allocator dump block 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:bluestore allocator fragmentation block 2024-03-24T12:49:51.769 INFO:teuthology.orchestra.run.smithi029.stderr:admin_socket: invalid command 2024-03-24T12:49:52.230 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:52.305 INFO:teuthology.orchestra.run.smithi029.stderr:nodeep-scrub is unset 2024-03-24T12:49:52.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-mon[29919]: from='osd.0 v2:172.21.15.29:6801/1184012970' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T12:49:52.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-mon[29919]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T12:49:52.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/93410084' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:49:52.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-mon[29919]: 6.12 deep-scrub starts 2024-03-24T12:49:52.500 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-mon[29919]: 6.12 deep-scrub ok 2024-03-24T12:49:52.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-mon[41500]: from='osd.0 v2:172.21.15.29:6801/1184012970' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T12:49:52.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-mon[41500]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T12:49:52.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/93410084' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:49:52.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-mon[41500]: 6.12 deep-scrub starts 2024-03-24T12:49:52.501 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-mon[41500]: 6.12 deep-scrub ok 2024-03-24T12:49:52.501 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:49:52 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[216717]: 2024-03-24T12:49:52.324+0000 7f497b7ae700 -1 osd.0 417 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T12:49:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:52 smithi190 ceph-mon[33748]: from='osd.0 v2:172.21.15.29:6801/1184012970' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T12:49:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:52 smithi190 ceph-mon[33748]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T12:49:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:52 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/93410084' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-24T12:49:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:52 smithi190 ceph-mon[33748]: 6.12 deep-scrub starts 2024-03-24T12:49:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:52 smithi190 ceph-mon[33748]: 6.12 deep-scrub ok 2024-03-24T12:49:53.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:53 smithi190 ceph-mon[33748]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T12:49:53.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:53 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/93410084' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:49:53.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:53 smithi190 ceph-mon[33748]: osdmap e432: 8 total, 6 up, 6 in 2024-03-24T12:49:53.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:53 smithi190 ceph-mon[33748]: from='osd.0 v2:172.21.15.29:6801/1184012970' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:49:53.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:53 smithi190 ceph-mon[33748]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:49:53.634 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:53 smithi190 ceph-mon[33748]: pgmap v1045: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:53.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[41500]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/93410084' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[41500]: osdmap e432: 8 total, 6 up, 6 in 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[41500]: from='osd.0 v2:172.21.15.29:6801/1184012970' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[41500]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[41500]: pgmap v1045: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[29919]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/93410084' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[29919]: osdmap e432: 8 total, 6 up, 6 in 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[29919]: from='osd.0 v2:172.21.15.29:6801/1184012970' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[29919]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi029", "root=default"]}]: dispatch 2024-03-24T12:49:53.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:53 smithi029 ceph-mon[29919]: pgmap v1045: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:54 smithi190 ceph-mon[33748]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T12:49:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:54 smithi190 ceph-mon[33748]: osd.0 v2:172.21.15.29:6801/1184012970 boot 2024-03-24T12:49:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:54 smithi190 ceph-mon[33748]: osdmap e433: 8 total, 7 up, 6 in 2024-03-24T12:49:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:54 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:49:54.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:54 smithi190 ceph-mon[33748]: osdmap e434: 8 total, 7 up, 6 in 2024-03-24T12:49:54.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:54 smithi029 ceph-mon[41500]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T12:49:54.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:54 smithi029 ceph-mon[41500]: osd.0 v2:172.21.15.29:6801/1184012970 boot 2024-03-24T12:49:54.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:54 smithi029 ceph-mon[41500]: osdmap e433: 8 total, 7 up, 6 in 2024-03-24T12:49:54.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:54 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:49:54.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:54 smithi029 ceph-mon[41500]: osdmap e434: 8 total, 7 up, 6 in 2024-03-24T12:49:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:54 smithi029 ceph-mon[29919]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-24T12:49:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:54 smithi029 ceph-mon[29919]: osd.0 v2:172.21.15.29:6801/1184012970 boot 2024-03-24T12:49:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:54 smithi029 ceph-mon[29919]: osdmap e433: 8 total, 7 up, 6 in 2024-03-24T12:49:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:54 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T12:49:54.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:54 smithi029 ceph-mon[29919]: osdmap e434: 8 total, 7 up, 6 in 2024-03-24T12:49:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:55 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:49:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:55 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:49:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:55 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:49:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:55 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:49:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:55 smithi190 ceph-mon[33748]: pgmap v1048: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:55.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:55 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:49:55.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:55 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:49:55.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:55 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:49:55.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:55 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:49:55.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:55 smithi029 ceph-mon[41500]: pgmap v1048: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:55.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:55 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:49:55.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:55 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:49:55.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:55 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:49:55.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:55 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:49:55.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:55 smithi029 ceph-mon[29919]: pgmap v1048: 142 pgs: 51 active+undersized, 34 active+undersized+degraded, 57 active+clean; 740 KiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 128/600 objects degraded (21.333%) 2024-03-24T12:49:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:49:55.805 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-03-24T12:49:56.037 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:49:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:49:56.037 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-03-24T12:49:56.038 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:49:56.613 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[41500]: 1.2 scrub starts 2024-03-24T12:49:56.613 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[41500]: 1.2 scrub ok 2024-03-24T12:49:56.614 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[41500]: 5.17 scrub starts 2024-03-24T12:49:56.614 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[41500]: 5.17 scrub ok 2024-03-24T12:49:56.614 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[41500]: 3.0 deep-scrub starts 2024-03-24T12:49:56.614 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[41500]: 3.0 deep-scrub ok 2024-03-24T12:49:56.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[29919]: 1.2 scrub starts 2024-03-24T12:49:56.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[29919]: 1.2 scrub ok 2024-03-24T12:49:56.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[29919]: 5.17 scrub starts 2024-03-24T12:49:56.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[29919]: 5.17 scrub ok 2024-03-24T12:49:56.614 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[29919]: 3.0 deep-scrub starts 2024-03-24T12:49:56.615 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:56 smithi029 ceph-mon[29919]: 3.0 deep-scrub ok 2024-03-24T12:49:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:56 smithi190 ceph-mon[33748]: 1.2 scrub starts 2024-03-24T12:49:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:56 smithi190 ceph-mon[33748]: 1.2 scrub ok 2024-03-24T12:49:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:56 smithi190 ceph-mon[33748]: 5.17 scrub starts 2024-03-24T12:49:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:56 smithi190 ceph-mon[33748]: 5.17 scrub ok 2024-03-24T12:49:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:56 smithi190 ceph-mon[33748]: 3.0 deep-scrub starts 2024-03-24T12:49:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:56 smithi190 ceph-mon[33748]: 3.0 deep-scrub ok 2024-03-24T12:49:56.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-24T12:49:57.216 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:49:57.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:57 smithi029 ceph-mon[41500]: 1.6 scrub starts 2024-03-24T12:49:57.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:57 smithi029 ceph-mon[41500]: 1.6 scrub ok 2024-03-24T12:49:57.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:57 smithi029 ceph-mon[41500]: pgmap v1049: 142 pgs: 38 active+undersized, 21 active+undersized+degraded, 83 active+clean; 740 KiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 78/600 objects degraded (13.000%) 2024-03-24T12:49:57.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:57 smithi029 ceph-mon[29919]: 1.6 scrub starts 2024-03-24T12:49:57.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:57 smithi029 ceph-mon[29919]: 1.6 scrub ok 2024-03-24T12:49:57.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:57 smithi029 ceph-mon[29919]: pgmap v1049: 142 pgs: 38 active+undersized, 21 active+undersized+degraded, 83 active+clean; 740 KiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 78/600 objects degraded (13.000%) 2024-03-24T12:49:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:57 smithi190 ceph-mon[33748]: 1.6 scrub starts 2024-03-24T12:49:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:57 smithi190 ceph-mon[33748]: 1.6 scrub ok 2024-03-24T12:49:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:57 smithi190 ceph-mon[33748]: pgmap v1049: 142 pgs: 38 active+undersized, 21 active+undersized+degraded, 83 active+clean; 740 KiB data, 5.7 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 78/600 objects degraded (13.000%) 2024-03-24T12:49:57.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:49:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:49:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:49:58.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:58 smithi029 ceph-mon[41500]: 1.1 scrub starts 2024-03-24T12:49:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:58 smithi029 ceph-mon[41500]: 1.1 scrub ok 2024-03-24T12:49:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:58 smithi029 ceph-mon[41500]: Health check update: Degraded data redundancy: 78/600 objects degraded (13.000%), 21 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-24T12:49:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:58 smithi029 ceph-mon[41500]: 1.a scrub starts 2024-03-24T12:49:58.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:58 smithi029 ceph-mon[41500]: 1.a scrub ok 2024-03-24T12:49:58.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:58 smithi029 ceph-mon[29919]: 1.1 scrub starts 2024-03-24T12:49:58.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:58 smithi029 ceph-mon[29919]: 1.1 scrub ok 2024-03-24T12:49:58.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:58 smithi029 ceph-mon[29919]: Health check update: Degraded data redundancy: 78/600 objects degraded (13.000%), 21 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-24T12:49:58.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:58 smithi029 ceph-mon[29919]: 1.a scrub starts 2024-03-24T12:49:58.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:58 smithi029 ceph-mon[29919]: 1.a scrub ok 2024-03-24T12:49:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:58 smithi190 ceph-mon[33748]: 1.1 scrub starts 2024-03-24T12:49:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:58 smithi190 ceph-mon[33748]: 1.1 scrub ok 2024-03-24T12:49:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:58 smithi190 ceph-mon[33748]: Health check update: Degraded data redundancy: 78/600 objects degraded (13.000%), 21 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-03-24T12:49:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:58 smithi190 ceph-mon[33748]: 1.a scrub starts 2024-03-24T12:49:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:58 smithi190 ceph-mon[33748]: 1.a scrub ok 2024-03-24T12:49:59.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:49:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:49:59.460 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:49:59.460 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:49:59.460 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-03-24T12:49:59.460 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-03-24T12:49:59.461 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:49:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:59 smithi190 ceph-mon[33748]: pgmap v1050: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:49:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:59 smithi190 ceph-mon[33748]: 3.11 deep-scrub starts 2024-03-24T12:49:59.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:49:59 smithi190 ceph-mon[33748]: 3.11 deep-scrub ok 2024-03-24T12:49:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:59 smithi029 ceph-mon[41500]: pgmap v1050: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:49:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:59 smithi029 ceph-mon[41500]: 3.11 deep-scrub starts 2024-03-24T12:49:59.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:49:59 smithi029 ceph-mon[41500]: 3.11 deep-scrub ok 2024-03-24T12:49:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:49:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:59 smithi029 ceph-mon[29919]: pgmap v1050: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:49:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:59 smithi029 ceph-mon[29919]: 3.11 deep-scrub starts 2024-03-24T12:49:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:49:59 smithi029 ceph-mon[29919]: 3.11 deep-scrub ok 2024-03-24T12:50:00.152 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-24T12:50:00.406 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:00.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[29919]: 6.13 scrub starts 2024-03-24T12:50:00.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[29919]: 6.13 scrub ok 2024-03-24T12:50:00.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[29919]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/600 objects degraded (13.000%), 21 pgs degraded, 59 pgs undersized) 2024-03-24T12:50:00.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[29919]: Cluster is now healthy 2024-03-24T12:50:00.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[29919]: 5.1 deep-scrub starts 2024-03-24T12:50:00.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[29919]: 5.1 deep-scrub ok 2024-03-24T12:50:00.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[29919]: overall HEALTH_OK 2024-03-24T12:50:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[41500]: 6.13 scrub starts 2024-03-24T12:50:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[41500]: 6.13 scrub ok 2024-03-24T12:50:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[41500]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/600 objects degraded (13.000%), 21 pgs degraded, 59 pgs undersized) 2024-03-24T12:50:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[41500]: Cluster is now healthy 2024-03-24T12:50:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[41500]: 5.1 deep-scrub starts 2024-03-24T12:50:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[41500]: 5.1 deep-scrub ok 2024-03-24T12:50:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:00 smithi029 ceph-mon[41500]: overall HEALTH_OK 2024-03-24T12:50:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:00 smithi190 ceph-mon[33748]: 6.13 scrub starts 2024-03-24T12:50:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:00 smithi190 ceph-mon[33748]: 6.13 scrub ok 2024-03-24T12:50:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:00 smithi190 ceph-mon[33748]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 78/600 objects degraded (13.000%), 21 pgs degraded, 59 pgs undersized) 2024-03-24T12:50:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:00 smithi190 ceph-mon[33748]: Cluster is now healthy 2024-03-24T12:50:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:00 smithi190 ceph-mon[33748]: 5.1 deep-scrub starts 2024-03-24T12:50:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:00 smithi190 ceph-mon[33748]: 5.1 deep-scrub ok 2024-03-24T12:50:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:00 smithi190 ceph-mon[33748]: overall HEALTH_OK 2024-03-24T12:50:00.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:50:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:01 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:50:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:01 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:50:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:01 smithi190 ceph-mon[33748]: 4.12 scrub starts 2024-03-24T12:50:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:01 smithi190 ceph-mon[33748]: 4.12 scrub ok 2024-03-24T12:50:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:01 smithi190 ceph-mon[33748]: 1.5 scrub starts 2024-03-24T12:50:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:01 smithi190 ceph-mon[33748]: 1.5 scrub ok 2024-03-24T12:50:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:01 smithi190 ceph-mon[33748]: pgmap v1051: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:01 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:50:01.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:01 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:50:01.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:50:01.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:50:01.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[29919]: 4.12 scrub starts 2024-03-24T12:50:01.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[29919]: 4.12 scrub ok 2024-03-24T12:50:01.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[29919]: 1.5 scrub starts 2024-03-24T12:50:01.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[29919]: 1.5 scrub ok 2024-03-24T12:50:01.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[29919]: pgmap v1051: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:01.655 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:50:01.656 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:50:01.656 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:50:01.656 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:50:01.656 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[41500]: 4.12 scrub starts 2024-03-24T12:50:01.656 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[41500]: 4.12 scrub ok 2024-03-24T12:50:01.656 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[41500]: 1.5 scrub starts 2024-03-24T12:50:01.656 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[41500]: 1.5 scrub ok 2024-03-24T12:50:01.656 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[41500]: pgmap v1051: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:01.656 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:50:01.656 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:01 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:50:01.719 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:02.518 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:02 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:50:02.518 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:02 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:50:02.518 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:02 smithi029 ceph-mon[29919]: 5.a scrub starts 2024-03-24T12:50:02.518 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:02 smithi029 ceph-mon[29919]: 5.a scrub ok 2024-03-24T12:50:02.518 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:02 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:50:02.518 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:02 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:50:02.519 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:02 smithi029 ceph-mon[41500]: 5.a scrub starts 2024-03-24T12:50:02.519 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:02 smithi029 ceph-mon[41500]: 5.a scrub ok 2024-03-24T12:50:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:02 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:50:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:02 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:50:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:02 smithi190 ceph-mon[33748]: 5.a scrub starts 2024-03-24T12:50:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:02 smithi190 ceph-mon[33748]: 5.a scrub ok 2024-03-24T12:50:03.314 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:50:03.314 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:50:03.314 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:50:03.314 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:50:03.314 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.314 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:187077 10.15 10:aad622d8:::benchmark_data_smithi029_160664_object60103:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:43:59.367831+0000", 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 363.15373591700001, 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.84238948199999997, 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 187077 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.315 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.367831+0000", 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.367831+0000", 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.316 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.367833+0000", 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.886e-06 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.367840+0000", 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5030000000000002e-06 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.367877+0000", 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7808000000000001e-05 2024-03-24T12:50:03.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.369806+0000", 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00192837 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.371343+0000", 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0015372420000000001 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.318 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.371643+0000", 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00030009600000000002 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,1,2", 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.372040+0000", 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000396752 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.376194+0000", 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0041540550000000002 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.319 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.376208+0000", 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3754000000000001e-05 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.390269+0000", 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.014061830000000001 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.390283+0000", 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3375e-05 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.320 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.210138+0000", 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.81985514800000003 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.210221+0000", 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2663000000000005e-05 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:187239 10.15 10:ada4a2f2:::benchmark_data_smithi029_160664_object60265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:43:59.489999+0000", 2024-03-24T12:50:03.321 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 363.03156843400001, 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.73166062499999995, 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 187239 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.322 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.489999+0000", 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.489999+0000", 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490000+0000", 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.658e-06 2024-03-24T12:50:03.323 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490006+0000", 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0229999999999998e-06 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490023+0000", 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6283999999999999e-05 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490033+0000", 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0560999999999999e-05 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490102+0000", 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8380000000000006e-05 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490112+0000", 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0268000000000001e-05 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.210929+0000", 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.72081673300000004 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.325 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.211054+0000", 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012556899999999999 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.211449+0000", 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00039495300000000002 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.218947+0000", 2024-03-24T12:50:03.326 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0074974100000000004 2024-03-24T12:50:03.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.327 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.327 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.327 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.220806+0000", 2024-03-24T12:50:03.327 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0018592439999999999 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.220880+0000", 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4443999999999994e-05 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.221056+0000", 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00017564100000000001 2024-03-24T12:50:03.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.221659+0000", 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00060345699999999995 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:187240 10.15 10:ac3578f4:::benchmark_data_smithi029_160664_object60266:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:43:59.490423+0000", 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 363.03114413999998, 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.73162572599999998, 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.329 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 187240 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490423+0000", 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.330 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490423+0000", 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490424+0000", 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.6899999999999996e-07 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490428+0000", 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3510000000000004e-06 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.331 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490442+0000", 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3363999999999999e-05 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490450+0000", 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9929999999999995e-06 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490504+0000", 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4620999999999998e-05 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.332 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.490513+0000", 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5199999999999997e-06 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.213406+0000", 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.72289349700000005 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.213538+0000", 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013178899999999999 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.213884+0000", 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00034592200000000003 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.221709+0000", 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0078245880000000004 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.221764+0000", 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4994999999999998e-05 2024-03-24T12:50:03.334 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.221782+0000", 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8836000000000001e-05 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.221849+0000", 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6118999999999995e-05 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.222049+0000", 2024-03-24T12:50:03.335 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00020016200000000001 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.15279.0:187254 10.15 10:ae6a9e6e:::benchmark_data_smithi029_160664_object60280:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:43:59.499455+0000", 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 363.02211194099999, 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.72288140599999995, 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.336 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.15279", 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 187254 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.499455+0000", 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.499455+0000", 2024-03-24T12:50:03.337 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.499457+0000", 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8250000000000001e-06 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.499463+0000", 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5979999999999996e-06 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.338 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.499473+0000", 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0706e-05 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.499481+0000", 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1880000000000002e-06 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.499555+0000", 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3211000000000001e-05 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:50:03.339 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:43:59.499563+0000", 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5960000000000004e-06 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.215451+0000", 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.71588755999999998 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.215533+0000", 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2103999999999999e-05 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 0,1", 2024-03-24T12:50:03.340 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.215789+0000", 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00025584400000000003 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.222116+0000", 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0063269839999999999 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.222161+0000", 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5352000000000002e-05 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.341 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.222188+0000", 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.7081999999999999e-05 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.222237+0000", 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9135999999999998e-05 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:44:00.222337+0000", 2024-03-24T12:50:03.342 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9220000000000002e-05 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:118811 11.a 11:5e22fc87:::benchmark_data_smithi029_192894_object13876:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e411)", 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:47:39.462758+0000", 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 143.05880898800001, 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.252176552, 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:50:03.343 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 118811 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.462758+0000", 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.462758+0000", 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.344 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.462759+0000", 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1689999999999999e-06 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.462764+0000", 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.4619999999999998e-06 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.345 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.462774+0000", 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0107e-05 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.462782+0000", 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9470000000000003e-06 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.462834+0000", 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1771000000000003e-05 2024-03-24T12:50:03.346 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.462995+0000", 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000161087 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.463077+0000", 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1928000000000006e-05 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.347 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.466351+0000", 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0032743870000000001 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.466854+0000", 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00050328199999999997 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.714664+0000", 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.247810218 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.348 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.714871+0000", 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000206868 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:39.714935+0000", 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3325999999999996e-05 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:124499 11.a 11:531d25c5:::benchmark_data_smithi029_192894_object19564:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-24T12:50:03.349 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:47:45.822039+0000", 2024-03-24T12:50:03.350 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.69952813500001, 2024-03-24T12:50:03.350 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.27164059600000001, 2024-03-24T12:50:03.350 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.350 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.350 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.350 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:50:03.350 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:03.350 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 124499 2024-03-24T12:50:03.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.351 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.351 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.351 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.351 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.822039+0000", 2024-03-24T12:50:03.351 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.351 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.351 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.352 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.352 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.822039+0000", 2024-03-24T12:50:03.352 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.352 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.352 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.352 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.352 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.822041+0000", 2024-03-24T12:50:03.352 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8300000000000001e-06 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.822047+0000", 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9270000000000002e-06 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.822063+0000", 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6562000000000002e-05 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.353 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.822072+0000", 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0140000000000004e-06 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.822124+0000", 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1694999999999999e-05 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.822243+0000", 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000119178 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.354 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.822348+0000", 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010477200000000001 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.830202+0000", 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0078542739999999996 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.837113+0000", 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0069110869999999998 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.355 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.093559+0000", 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.25644524899999999 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.093630+0000", 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1786999999999996e-05 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.093680+0000", 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9221000000000002e-05 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.356 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:124504 11.a 11:50e9ad5a:::benchmark_data_smithi029_192894_object19569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:47:45.823546+0000", 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.69802160899999, 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.27032788600000002, 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.357 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:50:03.358 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:03.358 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 124504 2024-03-24T12:50:03.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.358 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.358 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.358 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.358 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.823546+0000", 2024-03-24T12:50:03.358 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.358 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.358 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.823546+0000", 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.823547+0000", 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.172e-06 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.823552+0000", 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9570000000000001e-06 2024-03-24T12:50:03.359 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.823563+0000", 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0885e-05 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.823570+0000", 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.8790000000000003e-06 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.826257+0000", 2024-03-24T12:50:03.360 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026869160000000001 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.826387+0000", 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012978100000000001 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.826488+0000", 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000101035 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.830357+0000", 2024-03-24T12:50:03.361 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0038690859999999999 2024-03-24T12:50:03.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.362 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.362 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.362 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:45.837272+0000", 2024-03-24T12:50:03.362 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0069146040000000004 2024-03-24T12:50:03.362 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.362 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.362 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.362 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.093731+0000", 2024-03-24T12:50:03.363 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.25645935399999997 2024-03-24T12:50:03.363 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.363 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.363 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.363 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.093827+0000", 2024-03-24T12:50:03.363 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5784000000000003e-05 2024-03-24T12:50:03.364 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.364 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.364 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.364 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.093873+0000", 2024-03-24T12:50:03.364 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.6433000000000001e-05 2024-03-24T12:50:03.364 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.365 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.365 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.365 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.365 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.365 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:124938 11.a 11:55b4636c:::benchmark_data_smithi029_192894_object20003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-24T12:50:03.365 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:47:46.432049+0000", 2024-03-24T12:50:03.366 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 136.089518575, 2024-03-24T12:50:03.366 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.21977424700000001, 2024-03-24T12:50:03.366 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.366 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.366 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.366 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:50:03.367 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:03.367 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 124938 2024-03-24T12:50:03.367 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.367 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.367 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.368 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.368 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.432049+0000", 2024-03-24T12:50:03.368 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.368 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.368 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.368 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.369 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.432049+0000", 2024-03-24T12:50:03.369 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.369 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.369 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.369 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.369 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.432050+0000", 2024-03-24T12:50:03.370 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.747e-06 2024-03-24T12:50:03.370 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.370 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.370 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.370 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.432058+0000", 2024-03-24T12:50:03.371 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.4950000000000002e-06 2024-03-24T12:50:03.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.371 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.371 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.371 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.432078+0000", 2024-03-24T12:50:03.371 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0418e-05 2024-03-24T12:50:03.371 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.372 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.372 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.372 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.432089+0000", 2024-03-24T12:50:03.372 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.064e-05 2024-03-24T12:50:03.372 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.372 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.373 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.373 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.432152+0000", 2024-03-24T12:50:03.373 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.3391000000000003e-05 2024-03-24T12:50:03.373 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.373 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.373 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.374 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.432276+0000", 2024-03-24T12:50:03.374 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012369400000000001 2024-03-24T12:50:03.374 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.374 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.374 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:03.374 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.432381+0000", 2024-03-24T12:50:03.375 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00010516799999999999 2024-03-24T12:50:03.375 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.375 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.375 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.375 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.437907+0000", 2024-03-24T12:50:03.375 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0055254529999999996 2024-03-24T12:50:03.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.376 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.376 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.376 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.651439+0000", 2024-03-24T12:50:03.376 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.21353229300000001 2024-03-24T12:50:03.376 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.377 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.377 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.651482+0000", 2024-03-24T12:50:03.377 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.2972999999999997e-05 2024-03-24T12:50:03.377 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.377 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.378 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.378 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.651771+0000", 2024-03-24T12:50:03.378 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000289301 2024-03-24T12:50:03.378 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.378 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.379 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.379 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:47:46.651823+0000", 2024-03-24T12:50:03.379 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1674e-05 2024-03-24T12:50:03.379 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.379 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.379 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.380 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.380 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.380 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:4498 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 93828934255616 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:50:03.380 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:24.982805+0000", 2024-03-24T12:50:03.380 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 97.538762547999994, 2024-03-24T12:50:03.380 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0249253869999999, 2024-03-24T12:50:03.381 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.381 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:03.381 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.381 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:50:03.381 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:03.381 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4498 2024-03-24T12:50:03.382 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.382 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.382 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.382 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.382 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982805+0000", 2024-03-24T12:50:03.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.383 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.383 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.383 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.383 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982805+0000", 2024-03-24T12:50:03.383 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982807+0000", 2024-03-24T12:50:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0499999999999999e-06 2024-03-24T12:50:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.384 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.385 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982814+0000", 2024-03-24T12:50:03.385 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7030000000000006e-06 2024-03-24T12:50:03.385 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.385 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.385 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982839+0000", 2024-03-24T12:50:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4606000000000001e-05 2024-03-24T12:50:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.386 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982851+0000", 2024-03-24T12:50:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1959e-05 2024-03-24T12:50:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.387 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982966+0000", 2024-03-24T12:50:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000115578 2024-03-24T12:50:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op must wait for map", 2024-03-24T12:50:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982974+0000", 2024-03-24T12:50:03.388 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7889999999999995e-06 2024-03-24T12:50:03.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.389 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.389 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.389 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.992014+0000", 2024-03-24T12:50:03.389 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0090395170000000004 2024-03-24T12:50:03.389 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.390 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.390 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:50:03.390 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.992022+0000", 2024-03-24T12:50:03.390 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4859999999999997e-06 2024-03-24T12:50:03.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.391 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.001369+0000", 2024-03-24T12:50:03.391 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0093466289999999 2024-03-24T12:50:03.391 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.391 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.001527+0000", 2024-03-24T12:50:03.392 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00015763999999999999 2024-03-24T12:50:03.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.392 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.392 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.392 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.007730+0000", 2024-03-24T12:50:03.393 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0062034300000000002 2024-03-24T12:50:03.393 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.393 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.393 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.393 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.394 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:4499 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 93828934249856 gen 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:50:03.394 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:24.982880+0000", 2024-03-24T12:50:03.394 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 97.538686802000001, 2024-03-24T12:50:03.394 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0214835950000001, 2024-03-24T12:50:03.394 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.394 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:03.395 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.395 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:50:03.395 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:03.395 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4499 2024-03-24T12:50:03.395 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.395 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.396 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982880+0000", 2024-03-24T12:50:03.396 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.396 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.396 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982880+0000", 2024-03-24T12:50:03.397 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.397 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.397 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.397 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982881+0000", 2024-03-24T12:50:03.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8699999999999995e-07 2024-03-24T12:50:03.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.398 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.398 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982884+0000", 2024-03-24T12:50:03.398 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4400000000000001e-06 2024-03-24T12:50:03.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.399 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.399 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982893+0000", 2024-03-24T12:50:03.399 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1659999999999994e-06 2024-03-24T12:50:03.399 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.400 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.400 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.982899+0000", 2024-03-24T12:50:03.400 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8050000000000001e-06 2024-03-24T12:50:03.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.983039+0000", 2024-03-24T12:50:03.401 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000140161 2024-03-24T12:50:03.401 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.401 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.401 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op must wait for map", 2024-03-24T12:50:03.401 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.983046+0000", 2024-03-24T12:50:03.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0580000000000003e-06 2024-03-24T12:50:03.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.402 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.402 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.990904+0000", 2024-03-24T12:50:03.402 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0078580869999999997 2024-03-24T12:50:03.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:50:03.403 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:24.990919+0000", 2024-03-24T12:50:03.403 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5648e-05 2024-03-24T12:50:03.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.403 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.403 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:25.999965+0000", 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.009045489 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.000147+0000", 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00018222299999999999 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:26.004364+0000", 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0042169310000000002 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.404 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:173917 11.a 11:52abe032:::benchmark_data_smithi029_192894_object68982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:33.983994+0000", 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 88.537573561000002, 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.23047083199999999, 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 173917 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.983994+0000", 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.983994+0000", 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.983995+0000", 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.561e-06 2024-03-24T12:50:03.406 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.984002+0000", 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4459999999999998e-06 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.984018+0000", 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6509000000000001e-05 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.984026+0000", 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2870000000000004e-06 2024-03-24T12:50:03.407 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.984081+0000", 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4296000000000003e-05 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.984220+0000", 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013911199999999999 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985016+0000", 2024-03-24T12:50:03.408 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00079582999999999997 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.996538+0000", 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.011522622 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:34.000402+0000", 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0038635539999999999 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.409 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:34.214321+0000", 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.213919208 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:34.214416+0000", 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.5198999999999994e-05 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:34.214464+0000", 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8208000000000003e-05 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.410 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:173918 11.a 11:5340a55e:::benchmark_data_smithi029_192894_object68983:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:33.985830+0000", 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 88.535737492999999, 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.22890075600000001, 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 173918 2024-03-24T12:50:03.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985830+0000", 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985830+0000", 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985831+0000", 2024-03-24T12:50:03.412 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.37e-06 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985836+0000", 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0159999999999999e-06 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985847+0000", 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1263e-05 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.413 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.985855+0000", 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1759999999999993e-06 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.990700+0000", 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0048450430000000003 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.990831+0000", 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013087099999999999 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.414 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.990961+0000", 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013000500000000001 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:33.998664+0000", 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0077024820000000001 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:34.001170+0000", 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0025057830000000001 2024-03-24T12:50:03.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:34.214507+0000", 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.21333703400000001 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:34.214564+0000", 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7370000000000001e-05 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:34.214730+0000", 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000166343 2024-03-24T12:50:03.416 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:177084 11.a 11:5df6765e:::benchmark_data_smithi029_192894_object72149:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:36.779790+0000", 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 85.741777600000006, 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.21982268299999999, 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:03.417 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 177084 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.779790+0000", 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.779790+0000", 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.418 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.779791+0000", 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7150000000000001e-06 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.779798+0000", 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0049999999999998e-06 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.779819+0000", 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.0931e-05 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.419 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.779829+0000", 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0159999999999999e-05 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.780881+0000", 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001051427 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.781041+0000", 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000160617 2024-03-24T12:50:03.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.781179+0000", 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000137885 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.792097+0000", 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.01091778 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.800437+0000", 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0083400790000000002 2024-03-24T12:50:03.421 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.999507+0000", 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.19906939900000001 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.999565+0000", 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8613000000000003e-05 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.422 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.999612+0000", 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7071999999999999e-05 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:177090 11.a 11:52ac6f7e:::benchmark_data_smithi029_192894_object72155:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:36.782677+0000", 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 85.738890268000006, 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.22857925100000001, 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:50:03.423 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 177090 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.782677+0000", 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.782677+0000", 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.424 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.782678+0000", 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.407e-06 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.782684+0000", 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2830000000000001e-06 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.782699+0000", 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5804999999999999e-05 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.425 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.782710+0000", 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0545e-05 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.787021+0000", 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0043111010000000003 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.787150+0000", 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012875199999999999 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.426 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.787261+0000", 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000110809 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.803580+0000", 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0163191 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.803619+0000", 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9415999999999997e-05 2024-03-24T12:50:03.427 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:37.011145+0000", 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.207526353 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:37.011207+0000", 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1310999999999996e-05 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:37.011256+0000", 2024-03-24T12:50:03.428 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.9369000000000001e-05 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:177102 11.a 11:581cf13f:::benchmark_data_smithi029_192894_object72167:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:36.791567+0000", 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 85.730000537999999, 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.22023541999999999, 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:50:03.429 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 177102 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.791567+0000", 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.791567+0000", 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.430 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.791577+0000", 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0626e-05 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.791584+0000", 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7120000000000003e-06 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.791602+0000", 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.844e-05 2024-03-24T12:50:03.431 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.791613+0000", 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.059e-05 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.792133+0000", 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00052045699999999999 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.792268+0000", 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00013486300000000001 2024-03-24T12:50:03.432 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.792597+0000", 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00032871199999999999 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.803584+0000", 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.010987046 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.803646+0000", 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1920000000000003e-05 2024-03-24T12:50:03.433 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:37.011716+0000", 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.208070167 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:37.011765+0000", 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8741e-05 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.434 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:37.011802+0000", 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.7146e-05 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.25328.0:177104 11.a 11:5aa35ed0:::benchmark_data_smithi029_192894_object72169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:48:36.792621+0000", 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 85.728946524999998, 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.21982159100000001, 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.25328", 2024-03-24T12:50:03.435 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 177104 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.792621+0000", 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.792621+0000", 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.436 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.792621+0000", 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5899999999999996e-07 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.792625+0000", 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.3680000000000002e-06 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.792633+0000", 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1939999999999998e-06 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.437 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.792640+0000", 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.9280000000000004e-06 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.800458+0000", 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0078181029999999999 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.800533+0000", 2024-03-24T12:50:03.438 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5728000000000004e-05 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.800600+0000", 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.6697000000000002e-05 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op_commit", 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.803676+0000", 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0030760760000000001 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.439 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:36.806241+0000", 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0025647550000000002 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:37.012152+0000", 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.205910654 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "commit_sent", 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:37.012314+0000", 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00016245699999999999 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.440 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:48:37.012442+0000", 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012807199999999999 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:4649 5.15 5:a93a5511:::notify.2:head [watch ping cookie 93828934255616 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:38.488158+0000", 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 24.033408761, 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00046664500000000002, 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.441 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4649 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488158+0000", 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488158+0000", 2024-03-24T12:50:03.442 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488161+0000", 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9160000000000001e-06 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488174+0000", 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2938e-05 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.443 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488204+0000", 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.004e-05 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488221+0000", 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6725999999999999e-05 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488368+0000", 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00014674800000000001 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488454+0000", 2024-03-24T12:50:03.444 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6283e-05 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488625+0000", 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000170994 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:4650 5.3 5:c609908c:::notify.5:head [watch ping cookie 93828934249856 gen 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:38.488262+0000", 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 24.033304810000001, 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00069204300000000004, 2024-03-24T12:50:03.445 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4650 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488262+0000", 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.446 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488262+0000", 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488263+0000", 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0699999999999997e-07 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488266+0000", 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.9059999999999998e-06 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.447 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488276+0000", 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0356e-05 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488283+0000", 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5889999999999999e-06 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488666+0000", 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000382956 2024-03-24T12:50:03.448 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488742+0000", 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.6521999999999997e-05 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:38.488954+0000", 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00021220700000000001 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.449 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:4659 5.15 5:a93a5511:::notify.2:head [watch ping cookie 93828934255616 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e430)", 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:43.489170+0000", 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 19.032396824999999, 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00046383699999999998, 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4659 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.450 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:43.489170+0000", 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:43.489170+0000", 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:43.489171+0000", 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0999999999999997e-07 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.451 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:43.489175+0000", 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.3000000000000003e-06 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:43.489184+0000", 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1119999999999996e-06 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:43.489193+0000", 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9080000000000007e-06 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.452 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:43.489265+0000", 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.1172000000000002e-05 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:43.489364+0000", 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9457999999999995e-05 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:43.489634+0000", 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00027007700000000002 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.453 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "pg_update_log_missing(5.15 epoch 434/433 rep_tid 0 entries 434'13 (0'0) error 5:a93a5511:::notify.2:head by client.24344.0:4688 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:54.345155+0000", 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 8.1764123180000006, 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00056696199999999996, 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.345155+0000", 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.454 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.345155+0000", 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.345155+0000", 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9700000000000002e-07 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.345162+0000", 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.0350000000000004e-06 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.455 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.345177+0000", 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.4504e-05 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.345193+0000", 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6201e-05 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.345285+0000", 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2520000000000002e-05 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:03.456 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.345722+0000", 2024-03-24T12:50:03.457 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000436305 2024-03-24T12:50:03.457 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.457 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.457 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.457 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:03.457 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:03.457 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:50:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:03 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:50:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:03 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:50:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:03 smithi190 ceph-mon[33748]: pgmap v1052: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:03 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:50:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:03 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:50:03.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:50:03.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:50:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-mon[29919]: pgmap v1052: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:50:03.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:50:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:50:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:50:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-mon[41500]: pgmap v1052: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:50:03.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:50:04.167 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:50:03 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:50:03.697+0000 7f3fa5b1e700 -1 mgr get_metadata_python Requested missing service osd.5 2024-03-24T12:50:04.292 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-24T12:50:04.532 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[41500]: 4.6 scrub starts 2024-03-24T12:50:04.532 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[41500]: 4.6 scrub ok 2024-03-24T12:50:04.532 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:50:04.532 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:04.532 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:50:04.532 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[41500]: 4.11 scrub ok 2024-03-24T12:50:04.532 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[41500]: 6.15 scrub starts 2024-03-24T12:50:04.532 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[29919]: 4.6 scrub starts 2024-03-24T12:50:04.532 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[29919]: 4.6 scrub ok 2024-03-24T12:50:04.533 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:50:04.533 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:04.533 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:50:04.533 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[29919]: 4.11 scrub ok 2024-03-24T12:50:04.533 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:04 smithi029 ceph-mon[29919]: 6.15 scrub starts 2024-03-24T12:50:04.552 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:04 smithi190 ceph-mon[33748]: 4.6 scrub starts 2024-03-24T12:50:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:04 smithi190 ceph-mon[33748]: 4.6 scrub ok 2024-03-24T12:50:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:50:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:04 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:50:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:04 smithi190 ceph-mon[33748]: 4.11 scrub ok 2024-03-24T12:50:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:04 smithi190 ceph-mon[33748]: 6.15 scrub starts 2024-03-24T12:50:05.457 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[41500]: 6.18 scrub starts 2024-03-24T12:50:05.457 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[41500]: 6.18 scrub ok 2024-03-24T12:50:05.457 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[41500]: 6.15 scrub ok 2024-03-24T12:50:05.457 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:50:05.457 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:50:05.458 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:50:05.458 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:50:05.458 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[41500]: osdmap e435: 8 total, 7 up, 6 in 2024-03-24T12:50:05.458 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[41500]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:05.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[29919]: 6.18 scrub starts 2024-03-24T12:50:05.458 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[29919]: 6.18 scrub ok 2024-03-24T12:50:05.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[29919]: 6.15 scrub ok 2024-03-24T12:50:05.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:50:05.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:50:05.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:50:05.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:50:05.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[29919]: osdmap e435: 8 total, 7 up, 6 in 2024-03-24T12:50:05.459 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:05 smithi029 ceph-mon[29919]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:05.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:05 smithi190 ceph-mon[33748]: 6.18 scrub starts 2024-03-24T12:50:05.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:05 smithi190 ceph-mon[33748]: 6.18 scrub ok 2024-03-24T12:50:05.517 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:05 smithi190 ceph-mon[33748]: 6.15 scrub ok 2024-03-24T12:50:05.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:05 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:50:05.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:05 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:50:05.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:05 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:50:05.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:05 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:50:05.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:05 smithi190 ceph-mon[33748]: osdmap e435: 8 total, 7 up, 6 in 2024-03-24T12:50:05.518 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:05 smithi190 ceph-mon[33748]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:05.593 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:50:05.593 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:50:05.593 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-03-24T12:50:05.593 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:50:06.320 INFO:tasks.thrashosds.thrasher:reviving osd 2024-03-24T12:50:06.320 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2024-03-24T12:50:06.321 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2024-03-24T12:50:06.321 DEBUG:teuthology.orchestra.run.smithi190:osd.5> sudo journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T12:50:06.324 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl start ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.5 2024-03-24T12:50:06.350 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:06.350 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:50:06.351 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-03-24T12:50:06.351 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:06.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:06 smithi190 systemd[1]: Starting Ceph osd.5 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:50:06.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:06 smithi190 ceph-mon[33748]: 4.19 scrub starts 2024-03-24T12:50:06.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:06 smithi190 ceph-mon[33748]: 4.19 scrub ok 2024-03-24T12:50:06.976 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-24T12:50:07.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:06 smithi029 ceph-mon[29919]: 4.19 scrub starts 2024-03-24T12:50:07.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:06 smithi029 ceph-mon[29919]: 4.19 scrub ok 2024-03-24T12:50:07.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:06 smithi029 ceph-mon[41500]: 4.19 scrub starts 2024-03-24T12:50:07.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:06 smithi029 ceph-mon[41500]: 4.19 scrub ok 2024-03-24T12:50:07.454 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 podman[156162]: 2024-03-24 12:50:07.108603893 +0000 UTC m=+0.018157156 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:50:07.454 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 podman[156162]: 2024-03-24 12:50:07.247402814 +0000 UTC m=+0.156956118 container create 7f861f98965707aaa913e69b913f3a3845108808231e8332eed6c4fc7ef332ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:50:07.454 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 podman[156162]: 2024-03-24 12:50:07.459238288 +0000 UTC m=+0.368791558 container init 7f861f98965707aaa913e69b913f3a3845108808231e8332eed6c4fc7ef332ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:50:07.598 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:07.721 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-mon[33748]: 3.10 scrub starts 2024-03-24T12:50:07.721 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-mon[33748]: 3.10 scrub ok 2024-03-24T12:50:07.721 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-mon[33748]: pgmap v1055: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:07.721 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-mon[33748]: 4.17 scrub starts 2024-03-24T12:50:07.721 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-mon[33748]: 4.17 scrub ok 2024-03-24T12:50:07.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 podman[156162]: 2024-03-24 12:50:07.466046538 +0000 UTC m=+0.375599797 container start 7f861f98965707aaa913e69b913f3a3845108808231e8332eed6c4fc7ef332ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T12:50:07.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 podman[156162]: 2024-03-24 12:50:07.566786744 +0000 UTC m=+0.476340018 container attach 7f861f98965707aaa913e69b913f3a3845108808231e8332eed6c4fc7ef332ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T12:50:07.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:50:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:50:07.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-mon[29919]: 3.10 scrub starts 2024-03-24T12:50:07.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-mon[29919]: 3.10 scrub ok 2024-03-24T12:50:07.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-mon[29919]: pgmap v1055: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:07.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-mon[29919]: 4.17 scrub starts 2024-03-24T12:50:07.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-mon[29919]: 4.17 scrub ok 2024-03-24T12:50:07.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-mon[41500]: 3.10 scrub starts 2024-03-24T12:50:07.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-mon[41500]: 3.10 scrub ok 2024-03-24T12:50:07.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-mon[41500]: pgmap v1055: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-24T12:50:07.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-mon[41500]: 4.17 scrub starts 2024-03-24T12:50:07.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:07 smithi029 ceph-mon[41500]: 4.17 scrub ok 2024-03-24T12:50:08.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate[156190]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T12:50:08.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate[156190]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T12:50:08.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate[156190]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate[156190]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate[156190]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate[156190]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate[156190]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 bash[156162]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 bash[156162]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 bash[156162]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 bash[156162]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 bash[156162]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 bash[156162]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-24T12:50:08.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:07 smithi190 bash[156162]: --> ceph-volume raw activate successful for osd ID: 5 2024-03-24T12:50:09.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:08 smithi190 ceph-mon[33748]: 6.a deep-scrub starts 2024-03-24T12:50:09.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:08 smithi190 ceph-mon[33748]: 6.a deep-scrub ok 2024-03-24T12:50:09.030 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:50:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:50:09.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:08 smithi190 podman[156162]: 2024-03-24 12:50:08.748491742 +0000 UTC m=+1.658045011 container died 7f861f98965707aaa913e69b913f3a3845108808231e8332eed6c4fc7ef332ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T12:50:09.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:08 smithi029 ceph-mon[41500]: 6.a deep-scrub starts 2024-03-24T12:50:09.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:08 smithi029 ceph-mon[41500]: 6.a deep-scrub ok 2024-03-24T12:50:09.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:08 smithi029 ceph-mon[29919]: 6.a deep-scrub starts 2024-03-24T12:50:09.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:08 smithi029 ceph-mon[29919]: 6.a deep-scrub ok 2024-03-24T12:50:09.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:09 smithi190 podman[156395]: 2024-03-24 12:50:09.26817717 +0000 UTC m=+1.244126953 container remove 7f861f98965707aaa913e69b913f3a3845108808231e8332eed6c4fc7ef332ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-db0330b, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:50:09.837 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:09.838 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:50:09.838 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-03-24T12:50:09.838 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-03-24T12:50:09.838 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:10.027 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:09 smithi190 ceph-mon[33748]: 5.7 deep-scrub starts 2024-03-24T12:50:10.028 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:09 smithi190 ceph-mon[33748]: 5.7 deep-scrub ok 2024-03-24T12:50:10.028 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:10.028 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:09 smithi190 ceph-mon[33748]: 6.1e scrub starts 2024-03-24T12:50:10.028 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:09 smithi190 ceph-mon[33748]: 6.1e scrub ok 2024-03-24T12:50:10.028 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:09 smithi190 ceph-mon[33748]: pgmap v1056: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:10.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:09 smithi190 podman[156539]: 2024-03-24 12:50:09.837658106 +0000 UTC m=+0.018827460 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:50:10.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:09 smithi190 podman[156539]: 2024-03-24 12:50:09.947993622 +0000 UTC m=+0.129162970 container create a6cac83b1f7c6cd2622fa0de1b641595ef77cb5bbed34b5a43387e92342b4608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T12:50:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[41500]: 5.7 deep-scrub starts 2024-03-24T12:50:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[41500]: 5.7 deep-scrub ok 2024-03-24T12:50:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[41500]: 6.1e scrub starts 2024-03-24T12:50:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[41500]: 6.1e scrub ok 2024-03-24T12:50:10.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[41500]: pgmap v1056: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:10.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[29919]: 5.7 deep-scrub starts 2024-03-24T12:50:10.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[29919]: 5.7 deep-scrub ok 2024-03-24T12:50:10.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:10.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[29919]: 6.1e scrub starts 2024-03-24T12:50:10.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[29919]: 6.1e scrub ok 2024-03-24T12:50:10.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:09 smithi029 ceph-mon[29919]: pgmap v1056: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:10.435 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:50:10.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:10 smithi190 podman[156539]: 2024-03-24 12:50:10.326943227 +0000 UTC m=+0.508112582 container init a6cac83b1f7c6cd2622fa0de1b641595ef77cb5bbed34b5a43387e92342b4608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:50:10.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:10 smithi190 podman[156539]: 2024-03-24 12:50:10.333552349 +0000 UTC m=+0.514721700 container start a6cac83b1f7c6cd2622fa0de1b641595ef77cb5bbed34b5a43387e92342b4608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:50:10.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:10 smithi190 ceph-osd[156589]: -- 172.21.15.190:0/4004405405 <== mon.1 v2:172.21.15.190:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bc7a4f51e0 con 0x55bc7a4c7000 2024-03-24T12:50:10.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:10 smithi190 bash[156539]: a6cac83b1f7c6cd2622fa0de1b641595ef77cb5bbed34b5a43387e92342b4608 2024-03-24T12:50:10.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:10 smithi190 systemd[1]: Started Ceph osd.5 for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:50:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:10 smithi190 ceph-mon[33748]: 4.7 scrub starts 2024-03-24T12:50:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:10 smithi190 ceph-mon[33748]: 4.7 scrub ok 2024-03-24T12:50:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:10 smithi190 ceph-mon[33748]: 4.b scrub starts 2024-03-24T12:50:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:10 smithi190 ceph-mon[33748]: 4.b scrub ok 2024-03-24T12:50:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:10 smithi190 ceph-mon[33748]: 4.1a scrub starts 2024-03-24T12:50:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:10 smithi190 ceph-mon[33748]: 4.1a scrub ok 2024-03-24T12:50:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:10 smithi190 ceph-mon[33748]: 5.d scrub starts 2024-03-24T12:50:10.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:10 smithi190 ceph-mon[33748]: 5.d scrub ok 2024-03-24T12:50:10.944 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-24T12:50:11.018 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:11.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:10 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:10.893+0000 7f192275d7c0 -1 Falling back to public interface 2024-03-24T12:50:11.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[41500]: 4.7 scrub starts 2024-03-24T12:50:11.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[41500]: 4.7 scrub ok 2024-03-24T12:50:11.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[41500]: 4.b scrub starts 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[41500]: 4.b scrub ok 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[41500]: 4.1a scrub starts 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[41500]: 4.1a scrub ok 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[41500]: 5.d scrub starts 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[41500]: 5.d scrub ok 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[29919]: 4.7 scrub starts 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[29919]: 4.7 scrub ok 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[29919]: 4.b scrub starts 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[29919]: 4.b scrub ok 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[29919]: 4.1a scrub starts 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[29919]: 4.1a scrub ok 2024-03-24T12:50:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[29919]: 5.d scrub starts 2024-03-24T12:50:11.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:10 smithi029 ceph-mon[29919]: 5.d scrub ok 2024-03-24T12:50:11.761 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:12.016 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:11 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:50:12.017 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:11 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:50:12.017 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:11 smithi190 ceph-mon[33748]: pgmap v1057: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:12.017 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:11 smithi190 ceph-mon[33748]: 3.4 scrub starts 2024-03-24T12:50:12.017 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:11 smithi190 ceph-mon[33748]: 3.4 scrub ok 2024-03-24T12:50:12.017 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:11 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:50:12.017 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:11 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:50:12.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:50:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:50:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[41500]: pgmap v1057: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[41500]: 3.4 scrub starts 2024-03-24T12:50:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[41500]: 3.4 scrub ok 2024-03-24T12:50:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:50:12.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:50:12.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:50:12.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:50:12.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[29919]: pgmap v1057: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:12.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[29919]: 3.4 scrub starts 2024-03-24T12:50:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[29919]: 3.4 scrub ok 2024-03-24T12:50:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:50:12.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:11 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:50:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:12 smithi190 ceph-mon[33748]: 5.b scrub starts 2024-03-24T12:50:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:12 smithi190 ceph-mon[33748]: 5.b scrub ok 2024-03-24T12:50:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:12 smithi190 ceph-mon[33748]: 3.d scrub starts 2024-03-24T12:50:13.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:12 smithi190 ceph-mon[33748]: 3.d scrub ok 2024-03-24T12:50:13.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:12 smithi029 ceph-mon[41500]: 5.b scrub starts 2024-03-24T12:50:13.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:12 smithi029 ceph-mon[41500]: 5.b scrub ok 2024-03-24T12:50:13.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:12 smithi029 ceph-mon[41500]: 3.d scrub starts 2024-03-24T12:50:13.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:12 smithi029 ceph-mon[41500]: 3.d scrub ok 2024-03-24T12:50:13.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:12 smithi029 ceph-mon[29919]: 5.b scrub starts 2024-03-24T12:50:13.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:12 smithi029 ceph-mon[29919]: 5.b scrub ok 2024-03-24T12:50:13.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:12 smithi029 ceph-mon[29919]: 3.d scrub starts 2024-03-24T12:50:13.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:12 smithi029 ceph-mon[29919]: 3.d scrub ok 2024-03-24T12:50:13.255 INFO:teuthology.orchestra.run.smithi190.stderr:no valid command found; 10 closest matches: 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:0 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:1 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:2 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:abort 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:assert 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:config diff 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:config diff get 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:config get 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:config help [] 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:config set ... 2024-03-24T12:50:13.256 INFO:teuthology.orchestra.run.smithi190.stderr:admin_socket: invalid command 2024-03-24T12:50:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:13 smithi190 ceph-mon[33748]: 4.1 scrub starts 2024-03-24T12:50:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:13 smithi190 ceph-mon[33748]: 4.1 scrub ok 2024-03-24T12:50:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:13 smithi190 ceph-mon[33748]: pgmap v1058: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:13 smithi190 ceph-mon[33748]: 4.15 deep-scrub starts 2024-03-24T12:50:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:13 smithi190 ceph-mon[33748]: 4.15 deep-scrub ok 2024-03-24T12:50:14.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:13 smithi029 ceph-mon[29919]: 4.1 scrub starts 2024-03-24T12:50:14.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:13 smithi029 ceph-mon[29919]: 4.1 scrub ok 2024-03-24T12:50:14.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:13 smithi029 ceph-mon[29919]: pgmap v1058: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:14.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:13 smithi029 ceph-mon[29919]: 4.15 deep-scrub starts 2024-03-24T12:50:14.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:13 smithi029 ceph-mon[29919]: 4.15 deep-scrub ok 2024-03-24T12:50:14.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:13 smithi029 ceph-mon[41500]: 4.1 scrub starts 2024-03-24T12:50:14.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:13 smithi029 ceph-mon[41500]: 4.1 scrub ok 2024-03-24T12:50:14.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:13 smithi029 ceph-mon[41500]: pgmap v1058: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:14.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:13 smithi029 ceph-mon[41500]: 4.15 deep-scrub starts 2024-03-24T12:50:14.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:13 smithi029 ceph-mon[41500]: 4.15 deep-scrub ok 2024-03-24T12:50:14.370 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T12:50:14.371 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2024-03-24T12:50:14.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.364+0000 7f192275d7c0 -1 osd.5 203 log_to_monitors true 2024-03-24T12:50:14.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-mon[33748]: 3.7 scrub starts 2024-03-24T12:50:14.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-mon[33748]: 3.7 scrub ok 2024-03-24T12:50:14.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-mon[33748]: from='osd.5 v2:172.21.15.190:6804/477033924' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:50:14.853 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-mon[33748]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.731+0000 7f190f548700 -1 osd.5 203 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.854 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.855 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.855 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.855 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.855 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.855 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.855 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.855 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.855 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.855 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.859 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.860 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.861 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.862 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.863 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:14.864 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:14.865 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.739+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:14.866 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:14.867 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:14.868 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:14.869 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:14.869 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:14.871 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:14.872 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:14.873 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:14.874 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.875 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.876 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.877 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.740+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.878 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:14.879 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:14.880 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:14.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:14.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:14.883 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:14.884 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:14.885 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:14.886 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.887 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.888 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.741+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.889 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.890 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.891 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.891 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.891 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.891 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.891 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.891 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.891 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.891 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:14.896 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:14.897 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:14.898 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:14.899 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:14.900 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:14.901 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.742+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:14.902 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:14.903 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:14.903 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:14.903 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:14.903 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:14.903 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:14.903 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:14.903 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:14.903 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:14.903 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:14.904 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.905 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.906 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.907 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.908 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.909 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.743+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.910 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.911 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:14.912 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:14.913 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:14.914 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:14.914 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:14.914 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:14.914 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:14.914 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:14.914 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:14.914 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:14.914 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:14.914 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:14.915 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:14.916 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:14.917 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:14.917 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:14.917 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:14.917 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:14.917 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:14.917 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:14.917 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:14.917 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:14.917 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:14.917 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:14.918 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.744+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:14.919 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:14.920 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.921 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.922 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.923 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.923 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.923 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.923 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.923 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.923 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.924 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.924 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.924 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.924 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.924 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.924 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.924 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.924 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.925 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.926 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.926 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.926 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.926 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.926 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.926 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.926 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.926 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.926 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.926 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.927 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.927 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.927 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.927 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.927 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.927 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.927 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.927 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.928 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.929 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.929 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.929 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.929 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.929 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.929 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.929 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.929 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.745+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:14.929 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.930 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.931 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:14.932 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:14.933 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:14.934 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:14.935 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:14.935 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:14.935 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:14.935 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:14.935 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:14.935 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:14.935 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:14.935 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:14.936 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.746+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:14.937 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:14.938 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:14.939 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:14.940 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.941 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.942 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.943 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.944 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.945 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.945 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.945 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.945 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.945 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.945 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.945 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.945 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.946 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.946 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.946 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.946 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.946 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.946 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.947 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:14.947 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.747+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:14.947 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:14.947 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:14.947 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:14.947 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:14.947 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:14.947 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:14.951 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:14.952 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:14.952 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:14.952 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:14.952 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:14.952 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:14.952 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:14.952 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:14.952 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:14.952 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:14.952 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:14.953 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:14.954 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:14.954 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:14.954 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:14.954 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:14.954 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:14.954 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:14.954 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:14.954 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:14.954 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:14.955 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:14.955 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:14.955 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:14.955 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:14.955 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:14.955 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:14.955 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:14.955 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:14.956 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:14.957 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:14.957 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:14.957 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:14.957 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:14.957 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:14.957 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:14.957 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:14.957 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:14.957 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:14.958 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:14.959 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.960 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.961 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.961 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.961 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.961 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.961 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.961 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.961 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.961 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.962 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.963 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.748+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.964 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.965 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.966 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.967 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.967 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.967 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.967 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.967 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.967 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.967 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.967 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.967 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.967 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:14.968 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:14.969 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:14.970 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:14.971 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:14.972 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:14.973 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:14.973 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:14.973 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:14.973 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:14.973 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:14.973 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:14.973 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:14.973 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:14.973 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:14.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:14.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:14.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:14.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:14.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:14.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:14.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:14.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:14.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:14.974 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:14.975 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:14.976 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.749+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.977 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.978 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.978 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.978 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.978 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.978 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.978 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.978 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.978 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.978 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.979 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.980 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.981 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.982 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.983 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.983 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.983 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.983 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.983 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.983 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:14.984 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:14.985 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:14.985 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:14.985 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:14.985 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:14.985 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:14.985 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:14.985 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:14.985 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:14.985 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:14.985 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:14.986 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:14.987 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:14.988 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:14.989 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.750+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:14.990 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:14.991 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:14.992 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:14.993 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:14.993 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:14.993 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:14.993 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:14.993 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:14.993 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:14.993 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:14.993 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:14.993 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:14.994 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:14.994 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:14.994 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:14.994 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:14.994 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:14.994 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:14.994 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:14.995 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:14.996 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:14.997 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:14.998 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:14.998 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:14.998 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:14.998 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:14.998 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:14.998 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:14.998 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:14.998 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:14.999 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-03-24T12:50:14.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-03-24T12:50:14.999 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:50:14.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:14.999 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:55104 10.1 10:8eda7192:::benchmark_data_smithi029_160664_object55103:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:50:14.999 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:41:32.693019+0000", 2024-03-24T12:50:14.999 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 522.21241190399996, 2024-03-24T12:50:14.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.372459186, 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 55104 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.693019+0000", 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.693019+0000", 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.693020+0000", 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9700000000000005e-07 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.693159+0000", 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013939900000000001 2024-03-24T12:50:15.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.693169+0000", 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2310000000000002e-06 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.693176+0000", 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0110000000000003e-06 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.693219+0000", 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2975000000000001e-05 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.693342+0000", 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000123041 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.003 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.693464+0000", 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012239400000000001 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.695610+0000", 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021462270000000001 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.697925+0000", 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023150279999999998 2024-03-24T12:50:15.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:33.065379+0000", 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36745318599999999 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:33.065426+0000", 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7466000000000003e-05 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:33.065478+0000", 2024-03-24T12:50:15.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2330999999999998e-05 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:184577 10.e 10:7c517942:::benchmark_data_smithi029_160664_object57603:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:55.578136+0000", 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 379.32729524899997, 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25765790100000002, 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.006 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184577 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578136+0000", 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578136+0000", 2024-03-24T12:50:15.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578136+0000", 2024-03-24T12:50:15.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4799999999999998e-07 2024-03-24T12:50:15.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578140+0000", 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1990000000000002e-06 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578148+0000", 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.242e-06 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578153+0000", 2024-03-24T12:50:15.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0889999999999999e-06 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578198+0000", 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5086000000000002e-05 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578287+0000", 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9458999999999998e-05 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578402+0000", 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011423500000000001 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.583575+0000", 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0051728349999999998 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593589+0000", 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010014099 2024-03-24T12:50:15.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.835728+0000", 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24213921399999999 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.835763+0000", 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.5018999999999998e-05 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.835794+0000", 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0876000000000001e-05 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:184580 10.e 10:7710b3ed:::benchmark_data_smithi029_160664_object57606:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:55.578308+0000", 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 379.32712318400002, 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25762811699999999, 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.013 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.014 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.014 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184580 2024-03-24T12:50:15.017 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.017 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.017 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.017 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.017 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.017 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.018 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.018 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.018 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.018 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.018 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.018 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.018 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.018 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.018 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.018 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.019 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.020 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.021 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.022 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.022 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.022 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.022 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.022 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.022 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.022 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.022 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.022 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.023 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.023 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.023 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.023 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.023 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.023 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.023 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.023 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.023 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.023 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.024 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.751+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.025 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.026 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.027 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.027 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.027 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.027 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.027 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.027 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.028 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.029 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.030 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.031 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.032 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.752+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.033 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.034 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.035 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.036 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.036 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.036 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.036 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.036 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.036 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.036 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.036 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.037 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.037 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.037 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.037 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.037 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.037 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.037 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.037 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.038 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.038 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.038 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.038 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.038 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.038 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.038 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.038 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.039 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.039 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.039 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.039 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.039 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.039 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.039 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.039 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.040 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.040 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.040 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.040 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.040 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.040 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.040 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.040 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.040 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.041 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.041 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.041 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.041 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.041 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.041 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.041 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.041 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.042 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.042 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.042 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.042 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.042 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.042 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.042 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.042 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.043 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.043 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.043 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.043 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.043 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.043 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.043 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.043 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.044 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.044 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.044 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.044 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.044 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.044 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.044 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.044 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.045 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.045 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.045 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.045 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.045 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.045 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.045 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.045 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.045 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.046 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.046 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.046 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.753+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.046 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.046 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.046 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.046 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.046 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.047 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.048 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.049 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.050 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.051 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.052 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.053 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.054 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.055 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.754+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.056 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.057 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.058 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.059 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.061 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578308+0000", 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578308+0000", 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578309+0000", 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0699999999999996e-07 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578313+0000", 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4109999999999998e-06 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578323+0000", 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7780000000000002e-06 2024-03-24T12:50:15.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.578331+0000", 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8539999999999997e-06 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.580481+0000", 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021499729999999999 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.580594+0000", 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112796 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.580709+0000", 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115458 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.583582+0000", 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0028728569999999999 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593637+0000", 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010054883000000001 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.835842+0000", 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24220524800000001 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.835893+0000", 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1022e-05 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.835936+0000", 2024-03-24T12:50:15.066 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2929999999999997e-05 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:184587 10.e 10:7375221f:::benchmark_data_smithi029_160664_object57613:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:55.584428+0000", 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 379.32100314600001, 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26203288000000002, 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.067 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184587 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.584428+0000", 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.584428+0000", 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.584429+0000", 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9999999999999997e-07 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.584433+0000", 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0169999999999998e-06 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.584441+0000", 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0220000000000004e-06 2024-03-24T12:50:15.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.584448+0000", 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2010000000000002e-06 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.584594+0000", 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014663400000000001 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.584703+0000", 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010867 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.584815+0000", 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000111594 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.587729+0000", 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029142349999999998 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593698+0000", 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0059685659999999998 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846350+0000", 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.252652349 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846406+0000", 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5738000000000001e-05 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846461+0000", 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5154000000000001e-05 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:184592 10.e 10:79f5c262:::benchmark_data_smithi029_160664_object57618:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:55.588410+0000", 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 379.31702117899999, 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.258187314, 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.073 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184592 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.588410+0000", 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.074 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.588410+0000", 2024-03-24T12:50:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.588411+0000", 2024-03-24T12:50:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6899999999999996e-07 2024-03-24T12:50:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.078 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.078 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.078 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.078 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.755+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.079 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.080 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.081 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.082 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.760+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.083 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.084 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.085 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.086 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.087 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.088 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.090 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.761+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.091 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.092 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.093 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.094 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.095 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.096 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.097 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.098 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.099 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.100 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.762+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.101 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.102 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.103 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.104 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.105 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.106 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.107 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.108 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.770+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.109 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.110 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.111 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.112 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.113 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.588414+0000", 2024-03-24T12:50:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6519999999999999e-06 2024-03-24T12:50:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.588422+0000", 2024-03-24T12:50:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4460000000000004e-06 2024-03-24T12:50:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.588428+0000", 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3180000000000003e-06 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.589763+0000", 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013347980000000001 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.589870+0000", 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106947 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.589974+0000", 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000104406 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593710+0000", 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003735726 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.595424+0000", 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0017141249999999999 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846509+0000", 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25108476400000002 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846552+0000", 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3390999999999998e-05 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846597+0000", 2024-03-24T12:50:15.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4872e-05 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:184596 10.e 10:7682318a:::benchmark_data_smithi029_160664_object57622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:55.591970+0000", 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 379.31346083199998, 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25478352100000001, 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.118 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184596 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.591970+0000", 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.591970+0000", 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.591971+0000", 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4099999999999997e-07 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.591975+0000", 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.63e-06 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.591984+0000", 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5240000000000006e-06 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.591992+0000", 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5560000000000002e-06 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.592054+0000", 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2651999999999996e-05 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.592133+0000", 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9052000000000001e-05 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.592219+0000", 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5697000000000003e-05 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.595388+0000", 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031686280000000002 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.596663+0000", 2024-03-24T12:50:15.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012750249999999999 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846656+0000", 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.24999278 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846711+0000", 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5685e-05 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846754+0000", 2024-03-24T12:50:15.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2450999999999997e-05 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:184598 10.e 10:774e9ea7:::benchmark_data_smithi029_160664_object57624:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e340)", 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:55.593419+0000", 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 379.312011654, 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25349273500000002, 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.124 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 184598 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593419+0000", 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593419+0000", 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593420+0000", 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7899999999999997e-07 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593425+0000", 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.374e-06 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593435+0000", 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9469999999999998e-06 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593443+0000", 2024-03-24T12:50:15.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9189999999999999e-06 2024-03-24T12:50:15.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593733+0000", 2024-03-24T12:50:15.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029049700000000002 2024-03-24T12:50:15.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.130 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.130 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.130 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.130 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.130 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.130 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.130 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.130 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.130 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.131 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.132 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.133 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.134 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.135 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.771+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.136 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.137 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.138 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.772+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.139 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.140 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.141 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.142 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.143 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.144 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.145 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.146 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.147 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.779+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.148 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.149 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.150 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.151 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.152 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.153 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.154 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.155 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.780+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.156 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.157 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.158 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.159 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.160 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.781+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.161 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.162 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.163 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.163 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.163 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.163 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.163 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.163 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.163 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593861+0000", 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012780499999999999 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.593985+0000", 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012450699999999999 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.595514+0000", 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015283510000000001 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.596768+0000", 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012539389999999999 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846808+0000", 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.250040072 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846870+0000", 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2348999999999995e-05 2024-03-24T12:50:15.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:55.846912+0000", 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2095999999999998e-05 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:186466 10.1 10:86a6a74d:::benchmark_data_smithi029_160664_object59492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e342)", 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:58.332044+0000", 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 376.573386626, 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.26041075499999999, 2024-03-24T12:50:15.166 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 186466 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.332044+0000", 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.332044+0000", 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.332045+0000", 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8100000000000002e-07 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.332049+0000", 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8689999999999997e-06 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.332074+0000", 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5111000000000001e-05 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.332085+0000", 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1194e-05 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.332320+0000", 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00023507899999999999 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.332419+0000", 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8621000000000003e-05 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.332727+0000", 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030792800000000001 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.334861+0000", 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021342470000000001 2024-03-24T12:50:15.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.335163+0000", 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030178399999999998 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.592310+0000", 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.25714732800000001 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.592402+0000", 2024-03-24T12:50:15.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1811000000000006e-05 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:58.592455+0000", 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3102000000000001e-05 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:188184 10.1 10:831c03fe:::benchmark_data_smithi029_160664_object61210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:00.794518+0000", 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 374.110912869, 2024-03-24T12:50:15.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29675615700000002, 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 188184 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.794518+0000", 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.794518+0000", 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.794520+0000", 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3519999999999999e-06 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.794525+0000", 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3730000000000001e-06 2024-03-24T12:50:15.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.794541+0000", 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5874000000000001e-05 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.794549+0000", 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1130000000000001e-06 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.794652+0000", 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000102994 2024-03-24T12:50:15.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.794763+0000", 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011143 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.795180+0000", 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00041632699999999997 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.797043+0000", 2024-03-24T12:50:15.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0018635940000000001 2024-03-24T12:50:15.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:00.797485+0000", 2024-03-24T12:50:15.177 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:14 smithi029 ceph-mon[41500]: 3.7 scrub starts 2024-03-24T12:50:15.178 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:14 smithi029 ceph-mon[41500]: 3.7 scrub ok 2024-03-24T12:50:15.178 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:14 smithi029 ceph-mon[41500]: from='osd.5 v2:172.21.15.190:6804/477033924' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:50:15.178 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:14 smithi029 ceph-mon[41500]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:50:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:14 smithi029 ceph-mon[29919]: 3.7 scrub starts 2024-03-24T12:50:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:14 smithi029 ceph-mon[29919]: 3.7 scrub ok 2024-03-24T12:50:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:14 smithi029 ceph-mon[29919]: from='osd.5 v2:172.21.15.190:6804/477033924' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:50:15.178 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:14 smithi029 ceph-mon[29919]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T12:50:15.181 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.181 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.181 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.181 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.181 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.182 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.788+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.183 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.184 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.185 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.186 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.187 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.188 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.189 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.190 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.789+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.191 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.192 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.193 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.194 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.195 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.196 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.197 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.198 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.790+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.199 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.200 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.201 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.791+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.202 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.203 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.797+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.204 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.205 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.206 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.207 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.208 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.209 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.210 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.798+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.211 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.212 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.213 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.214 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.215 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.216 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.216 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044214200000000002 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:01.091170+0000", 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.29368470899999999 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:01.091225+0000", 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5095e-05 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.217 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:01.091274+0000", 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9153999999999997e-05 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:189438 10.e 10:72e56c19:::benchmark_data_smithi029_160664_object62464:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:03.063700+0000", 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 371.84173081, 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32436774899999998, 2024-03-24T12:50:15.218 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 189438 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.063700+0000", 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.219 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.063700+0000", 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.063701+0000", 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2219999999999999e-06 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.063706+0000", 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3320000000000002e-06 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.220 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.063718+0000", 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2383e-05 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.063726+0000", 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5329999999999997e-06 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.063784+0000", 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8199999999999998e-05 2024-03-24T12:50:15.221 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.063908+0000", 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012400700000000001 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 2,6", 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.064016+0000", 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107872 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.222 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066316+0000", 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023006459999999999 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066416+0000", 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9784000000000006e-05 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.387942+0000", 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.32152615899999998 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.223 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.388020+0000", 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7809e-05 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.388068+0000", 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7802000000000001e-05 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.224 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:189439 10.1 10:834e662e:::benchmark_data_smithi029_160664_object62465:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:03.066010+0000", 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 371.83942089200002, 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.279979115, 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 189439 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066010+0000", 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.225 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066010+0000", 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066011+0000", 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2049999999999999e-06 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066016+0000", 2024-03-24T12:50:15.226 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9989999999999999e-06 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066032+0000", 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5624000000000002e-05 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066040+0000", 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8199999999999997e-06 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.227 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066091+0000", 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1078999999999997e-05 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066251+0000", 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016018 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.066588+0000", 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00033734699999999999 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.228 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.068000+0000", 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0014117929999999999 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.068384+0000", 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00038383900000000002 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.345852+0000", 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27746762400000002 2024-03-24T12:50:15.229 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.345936+0000", 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4091000000000005e-05 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:03.345989+0000", 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3514000000000003e-05 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.230 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.233 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.233 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.233 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.233 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.233 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.234 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.235 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.236 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.799+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.237 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.238 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.239 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.240 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.800+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.241 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.242 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.243 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.244 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.245 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.807+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.246 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.247 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.248 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.249 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.250 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18fb384700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.251 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.252 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.253 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.808+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.254 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.255 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.256 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.257 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.258 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.259 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.260 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.809+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.816+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.816+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.816+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.816+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.816+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.816+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.816+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.261 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.816+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.262 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.263 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.264 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.266 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.267 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.268 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.269 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.269 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:192881 10.1 10:814c8d6e:::benchmark_data_smithi029_160664_object65907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e344)", 2024-03-24T12:50:15.269 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.354535+0000", 2024-03-24T12:50:15.269 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 368.55089621000002, 2024-03-24T12:50:15.269 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.303914982, 2024-03-24T12:50:15.269 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.269 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.269 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.269 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:15.269 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 192881 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.354535+0000", 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.354535+0000", 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.270 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.354536+0000", 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6099999999999999e-07 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.354540+0000", 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2849999999999996e-06 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.354552+0000", 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1472e-05 2024-03-24T12:50:15.271 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.354560+0000", 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2409999999999995e-06 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.354603+0000", 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2698999999999999e-05 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.272 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.354682+0000", 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9549999999999996e-05 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,7", 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.354965+0000", 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00028269999999999999 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.356339+0000", 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013742310000000001 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.273 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.357366+0000", 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010273470000000001 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.658231+0000", 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.30086508200000001 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.658291+0000", 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9886000000000002e-05 2024-03-24T12:50:15.274 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.658450+0000", 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015852800000000001 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25328.0:124511 11.e 11:7fe391ca:::benchmark_data_smithi029_192894_object19576:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:47:45.831718+0000", 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 149.07371303100001, 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.27706878899999998, 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.275 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25328", 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 124511 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.831718+0000", 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.831718+0000", 2024-03-24T12:50:15.276 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.831719+0000", 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.018e-06 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.831723+0000", 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8299999999999998e-06 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.277 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.831731+0000", 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6299999999999998e-06 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.831736+0000", 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2970000000000003e-06 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.831788+0000", 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1700000000000003e-05 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.278 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.831900+0000", 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011275 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,7", 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.831985+0000", 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4569999999999998e-05 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.833488+0000", 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001503269 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.279 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:45.833793+0000", 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030453000000000001 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:46.108722+0000", 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.274929541 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:46.108756+0000", 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4026000000000003e-05 2024-03-24T12:50:15.280 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:47:46.108787+0000", 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0627999999999999e-05 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24344.0:4502 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 93828934256768 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:48:24.983377+0000", 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 109.922054067, 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0563708009999999, 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.281 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:15.282 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.282 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24344", 2024-03-24T12:50:15.282 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:15.282 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 4502 2024-03-24T12:50:15.282 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.282 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.282 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.282 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.286 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.286 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.286 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.286 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.286 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.286 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.286 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.286 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.286 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.817+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.287 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.288 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.289 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.290 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.291 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.292 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.293 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.294 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.295 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.818+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.296 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.297 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.298 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.299 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.300 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.819+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.301 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.302 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.823+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.303 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.304 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.305 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.306 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.307 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.308 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.309 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.310 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.311 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.312 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.824+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.313 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.314 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.315 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.316 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.317 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.318 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.318 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f18ff38c700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.318 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.318 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.318 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.318 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.318 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.318 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.318 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.318 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.319 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.825+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.320 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.321 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.826+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.322 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.323 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.324 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.325 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.832+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.326 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.327 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983377+0000", 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983377+0000", 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983379+0000", 2024-03-24T12:50:15.328 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.795e-06 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983385+0000", 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8540000000000002e-06 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983396+0000", 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1666e-05 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.329 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983405+0000", 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8100000000000004e-06 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983704+0000", 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00029840699999999999 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983709+0000", 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0900000000000004e-06 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.330 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.994631+0000", 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.010922197999999999 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.994639+0000", 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.926e-06 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.001461+0000", 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0068221939999999 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.331 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.001934+0000", 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047324699999999999 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.039748+0000", 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.037813614000000002 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.332 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24344.0:4503 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 93828938916992 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:48:24.983426+0000", 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 109.922004734, 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.020021267, 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24344", 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 4503 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.333 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983426+0000", 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983426+0000", 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983427+0000", 2024-03-24T12:50:15.334 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0200000000000002e-07 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983429+0000", 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4399999999999999e-06 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983435+0000", 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6949999999999997e-06 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.335 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983441+0000", 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6119999999999998e-06 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983552+0000", 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011144700000000001 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op must wait for map", 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.983558+0000", 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.331e-06 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.336 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.990196+0000", 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0066375770000000004 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.990204+0000", 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5569999999999996e-06 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.999269+0000", 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0090648090000001 2024-03-24T12:50:15.337 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.999617+0000", 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00034778800000000003 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.003448+0000", 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0038305090000000002 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.338 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25328.0:162704 11.b 11:df620fcd:::benchmark_data_smithi029_192894_object57769:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:48:25.027706+0000", 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 109.877725246, 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.002494421, 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25328", 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 162704 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.339 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.027706+0000", 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.027706+0000", 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.027707+0000", 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.268e-06 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.340 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.027713+0000", 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2609999999999999e-06 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.027728+0000", 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4758e-05 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.027742+0000", 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4165000000000001e-05 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.341 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.342 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.027806+0000", 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.345 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.346 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.348 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.349 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.350 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.351 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.833+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.352 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.353 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.354 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.355 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.356 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.357 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.358 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.359 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.360 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.834+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.361 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.362 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.363 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.364 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.365 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.366 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.367 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.842+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.368 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.369 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.370 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.371 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.372 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.373 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.374 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.375 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.376 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.843+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.377 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.378 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.379 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.380 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.381 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.383 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3578999999999996e-05 2024-03-24T12:50:15.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:15.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.027818+0000", 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1857e-05 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.026089+0000", 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.99827120700000005 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.027034+0000", 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00094461600000000001 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1", 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.027106+0000", 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2069000000000004e-05 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.029074+0000", 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019683309999999998 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.030037+0000", 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00096326500000000004 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.030074+0000", 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6494e-05 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.030200+0000", 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000126551 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24344.0:4682 5.e 5:7759931f:::notify.3:head [watch ping cookie 93828934256768 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:49:53.491303+0000", 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.414127984, 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.84640064699999995, 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24344", 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 4682 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491303+0000", 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491303+0000", 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491303+0000", 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9999999999999998e-07 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491305+0000", 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9870000000000002e-06 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491310+0000", 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7400000000000004e-06 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491315+0000", 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3610000000000003e-06 2024-03-24T12:50:15.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491394+0000", 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9142999999999998e-05 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491410+0000", 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6745e-05 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.337466+0000", 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.846055483 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.337613+0000", 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014673400000000001 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.337704+0000", 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1053999999999999e-05 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24344.0:4683 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch ping cookie 93828938916992 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:49:53.491325+0000", 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.414106059000002, 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.84529331100000005, 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24344", 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 4683 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491325+0000", 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491325+0000", 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491325+0000", 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6800000000000001e-07 2024-03-24T12:50:15.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491328+0000", 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.2299999999999998e-06 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491333+0000", 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3349999999999997e-06 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491339+0000", 2024-03-24T12:50:15.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6799999999999998e-06 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491401+0000", 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2497999999999997e-05 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491413+0000", 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1915e-05 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.335571+0000", 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.84415803899999997 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.336527+0000", 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00095590200000000001 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.336618+0000", 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1343999999999998e-05 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.397 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24344.0:4690 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 93828934256768 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-03-24T12:50:15.397 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:49:53.491736+0000", 2024-03-24T12:50:15.399 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.399 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.399 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.399 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.399 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.399 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.844+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.400 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.401 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.849+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.402 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.403 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.404 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.405 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.406 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.407 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.408 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.409 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.410 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.850+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.411 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.413 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.414 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.415 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.416 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.417 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.418 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.851+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.419 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.420 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.421 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.852+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.422 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.423 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.424 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.425 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.426 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.426 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.426 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.426 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.426 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.426 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.426 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.426 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.426 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.426 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.427 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.413695116, 2024-03-24T12:50:15.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.86233004499999999, 2024-03-24T12:50:15.427 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.427 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:15.427 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.427 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24344", 2024-03-24T12:50:15.427 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:15.427 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 4690 2024-03-24T12:50:15.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491736+0000", 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491736+0000", 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491737+0000", 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4600000000000003e-07 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491741+0000", 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4309999999999996e-06 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491752+0000", 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0943e-05 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491759+0000", 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6490000000000002e-06 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491815+0000", 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6403000000000002e-05 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491833+0000", 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7829000000000002e-05 2024-03-24T12:50:15.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.337772+0000", 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.84593875299999999 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.337857+0000", 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.462e-05 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.354066+0000", 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016209471 2024-03-24T12:50:15.431 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24344.0:4691 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 93828938916992 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:49:53.491773+0000", 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 21.413657778000001, 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.85808331199999999, 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24344", 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:15.432 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 4691 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491773+0000", 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491773+0000", 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491774+0000", 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.08e-07 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491776+0000", 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9379999999999999e-06 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491781+0000", 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0549999999999999e-06 2024-03-24T12:50:15.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491784+0000", 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7740000000000002e-06 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491833+0000", 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8238999999999997e-05 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:53.491851+0000", 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.857e-05 2024-03-24T12:50:15.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.336647+0000", 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.844795933 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.336726+0000", 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8910000000000002e-05 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:49:54.349857+0000", 2024-03-24T12:50:15.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013130485000000001 2024-03-24T12:50:15.437 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.437 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.437 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.437 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:15.437 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:15.437 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:15.460 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.460 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.460 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.460 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.460 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.460 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.460 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.461 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.462 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.463 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.859+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.464 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.465 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.466 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.467 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.468 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.469 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.470 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.470 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.470 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.470 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.470 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.470 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.470 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.470 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.470 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.470 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.471 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.472 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.860+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.473 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.474 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.475 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.476 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.477 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.478 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.479 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.861+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.480 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.481 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.482 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.483 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.484 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.874+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.485 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.486 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.487 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.488 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.489 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.490 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.491 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.492 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.493 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.875+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.494 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.495 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.496 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.497 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.498 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.499 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.876+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.500 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.501 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.502 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.503 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.504 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.505 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.506 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.507 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.889+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.508 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.509 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.510 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.511 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.512 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.513 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.514 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.515 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.516 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.890+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.517 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.518 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.519 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.520 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.521 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.891+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.522 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.524 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.525 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.526 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.527 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.904+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.528 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.529 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.530 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.531 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:14 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.532 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.533 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.534 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.905+0000 7f18fdb89700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.535 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.536 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.537 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.538 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.539 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.540 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.541 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.542 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.543 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.906+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.544 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.907+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.545 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.546 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.547 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.931+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.548 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.549 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.550 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.551 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.552 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.553 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.554 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.554 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.554 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.554 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.554 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.554 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.554 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.554 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.554 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.554 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.555 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.932+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.556 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.557 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.558 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.559 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.560 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.561 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.562 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.563 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.933+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.564 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.565 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.566 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.934+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.567 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.962+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.568 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.569 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.570 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.571 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.572 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.573 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.574 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.575 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.576 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.963+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.577 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.578 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.579 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.580 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.581 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.582 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.583 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.584 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.964+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.585 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.586 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.587 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.588 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.965+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.589 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.590 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.591 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.592 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.593 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.594 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.595 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.596 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.985+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.597 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.598 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.600 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.601 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.602 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.603 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.604 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.605 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1902392700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.606 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.607 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.986+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f1900b8f700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.608 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.609 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:14.987+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.610 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.003+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.611 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.612 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.613 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.614 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.615 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.004+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.616 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.617 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.618 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.619 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.620 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.005+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.621 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.622 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.006+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.023+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.023+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.023+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.023+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.023+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.023+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.023+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.023+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.023+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.623 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.624 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.625 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.626 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.627 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.628 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.629 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.630 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.024+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.631 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.634 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.635 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.636 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.637 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.638 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.639 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.025+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.640 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.641 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18feb8b700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.642 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.643 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.026+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.644 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f1901b91700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.645 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.646 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.647 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.034+0000 7f18fe38a700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.648 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.649 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.650 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.651 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.652 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.653 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.654 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.035+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.655 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.656 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.657 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.036+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.047+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.658 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.047+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.047+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.047+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.047+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.047+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.659 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.660 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.661 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.662 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.663 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.664 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.665 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.666 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.048+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.667 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.668 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.669 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.670 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.671 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.672 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.672 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.672 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.672 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.672 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.672 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.673 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.673 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.673 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.674 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.675 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.049+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.676 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.677 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.678 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.679 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.050+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.680 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.681 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.682 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.683 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.054+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.684 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.685 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.686 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.687 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.688 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.689 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.690 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.691 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.055+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.692 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.693 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.694 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.695 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.696 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.697 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.698 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.056+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.699 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.700 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.701 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.057+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.702 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.703 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.704 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.705 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.706 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.707 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.065+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.708 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fbb85700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.709 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.710 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.711 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.712 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.713 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.714 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.715 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.716 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.717 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.066+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.718 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.719 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.720 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 204, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.721 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.722 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.723 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.724 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.067+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.725 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.726 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.727 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.075+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.728 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.729 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.730 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.731 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.732 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.733 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.734 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.735 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fc386700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 205, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.076+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 206, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 207, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 208, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 209, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 210, got 0 bytes 2024-03-24T12:50:15.736 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 211, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 212, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 213, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 214, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 215, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 216, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 217, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 218, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 219, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 220, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 221, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 222, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 223, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 224, got 0 bytes 2024-03-24T12:50:15.737 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 225, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 226, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 227, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 228, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 229, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 230, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.738 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.739 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.740 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.741 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.742 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.743 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.744 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.077+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.745 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.746 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.078+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.086+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.086+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.747 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.086+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.086+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.086+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.086+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.086+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.086+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.748 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18ffb8d700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.749 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.750 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.751 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.752 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f18fcb87700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.753 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.754 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.755 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.087+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 231, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 232, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 233, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 234, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.756 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.757 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.758 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.759 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.760 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.088+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.761 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.762 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f190038e700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.763 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.089+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 235, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 236, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 237, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 238, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 239, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 240, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 241, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 242, got 0 bytes 2024-03-24T12:50:15.764 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 243, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 244, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 245, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 246, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 247, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 248, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 249, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 250, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 251, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 252, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 253, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 254, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 255, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 256, got 0 bytes 2024-03-24T12:50:15.765 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 257, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 258, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 259, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 260, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 261, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 262, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 263, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 264, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 265, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 266, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 267, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 268, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 269, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 270, got 0 bytes 2024-03-24T12:50:15.766 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 271, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 272, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 273, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 274, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 275, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 276, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 277, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 278, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 279, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 280, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 281, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 282, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 283, got 0 bytes 2024-03-24T12:50:15.767 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 284, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 285, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 286, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 287, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 288, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 289, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 290, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 291, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 292, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 293, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 294, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 295, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 296, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 297, got 0 bytes 2024-03-24T12:50:15.768 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 298, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 299, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 300, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 301, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 302, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 303, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 304, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 305, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 306, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 307, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 308, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 309, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 310, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 311, got 0 bytes 2024-03-24T12:50:15.769 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 312, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 313, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 314, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 315, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 316, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 317, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 318, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 319, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 320, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 321, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 322, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 323, got 0 bytes 2024-03-24T12:50:15.770 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 324, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 325, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 326, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 327, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 328, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 329, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.771 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.772 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.096+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.773 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f1901390700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.774 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fd388700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 330, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 331, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 332, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 333, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 334, got 0 bytes 2024-03-24T12:50:15.775 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 335, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 336, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 337, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 338, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 339, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 340, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 341, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 342, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 343, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 344, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 345, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 346, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 347, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 348, got 0 bytes 2024-03-24T12:50:15.776 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 349, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 350, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 351, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 352, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 353, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 354, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 355, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 356, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 357, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 358, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 359, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 360, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 361, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 362, got 0 bytes 2024-03-24T12:50:15.777 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 363, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 364, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 365, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 366, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 367, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 368, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 369, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 370, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 371, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 372, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 373, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 374, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 375, got 0 bytes 2024-03-24T12:50:15.778 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 376, got 0 bytes 2024-03-24T12:50:15.779 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.097+0000 7f18fab83700 -1 osd.5 378 failed to load OSD map for epoch 377, got 0 bytes 2024-03-24T12:50:15.779 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:50:15.161+0000 7f1913bb5700 -1 osd.5 436 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T12:50:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-mon[33748]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T12:50:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-mon[33748]: from='osd.5 v2:172.21.15.190:6804/477033924' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:50:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-mon[33748]: osdmap e436: 8 total, 7 up, 6 in 2024-03-24T12:50:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-mon[33748]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:50:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:50:16.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:15 smithi190 ceph-mon[33748]: pgmap v1060: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:16.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[41500]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T12:50:16.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[41500]: from='osd.5 v2:172.21.15.190:6804/477033924' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:50:16.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[41500]: osdmap e436: 8 total, 7 up, 6 in 2024-03-24T12:50:16.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[41500]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:50:16.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:50:16.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[41500]: pgmap v1060: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:16.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[29919]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T12:50:16.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[29919]: from='osd.5 v2:172.21.15.190:6804/477033924' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:50:16.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[29919]: osdmap e436: 8 total, 7 up, 6 in 2024-03-24T12:50:16.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[29919]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi190", "root=default"]}]: dispatch 2024-03-24T12:50:16.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:50:16.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:15 smithi029 ceph-mon[29919]: pgmap v1060: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:16.500 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-24T12:50:16.758 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:17.123 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:16 smithi190 ceph-mon[33748]: osd.5 v2:172.21.15.190:6804/477033924 boot 2024-03-24T12:50:17.123 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:16 smithi190 ceph-mon[33748]: osdmap e437: 8 total, 8 up, 6 in 2024-03-24T12:50:17.123 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:16 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:50:17.123 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:16 smithi190 ceph-mon[33748]: 5.16 scrub starts 2024-03-24T12:50:17.124 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:16 smithi190 ceph-mon[33748]: 5.16 scrub ok 2024-03-24T12:50:17.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:16 smithi029 ceph-mon[41500]: osd.5 v2:172.21.15.190:6804/477033924 boot 2024-03-24T12:50:17.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:16 smithi029 ceph-mon[41500]: osdmap e437: 8 total, 8 up, 6 in 2024-03-24T12:50:17.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:16 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:50:17.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:16 smithi029 ceph-mon[41500]: 5.16 scrub starts 2024-03-24T12:50:17.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:16 smithi029 ceph-mon[41500]: 5.16 scrub ok 2024-03-24T12:50:17.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:16 smithi029 ceph-mon[29919]: osd.5 v2:172.21.15.190:6804/477033924 boot 2024-03-24T12:50:17.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:16 smithi029 ceph-mon[29919]: osdmap e437: 8 total, 8 up, 6 in 2024-03-24T12:50:17.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:16 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T12:50:17.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:16 smithi029 ceph-mon[29919]: 5.16 scrub starts 2024-03-24T12:50:17.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:16 smithi029 ceph-mon[29919]: 5.16 scrub ok 2024-03-24T12:50:17.741 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:17 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:50:17.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[41500]: osdmap e438: 8 total, 8 up, 6 in 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[41500]: pgmap v1063: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[41500]: 5.4 deep-scrub starts 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[41500]: 5.4 deep-scrub ok 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[29919]: osdmap e438: 8 total, 8 up, 6 in 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[29919]: pgmap v1063: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[29919]: 5.4 deep-scrub starts 2024-03-24T12:50:17.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-mon[29919]: 5.4 deep-scrub ok 2024-03-24T12:50:17.918 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:50:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:50:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:50:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:17 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:50:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:17 smithi190 ceph-mon[33748]: osdmap e438: 8 total, 8 up, 6 in 2024-03-24T12:50:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:17 smithi190 ceph-mon[33748]: pgmap v1063: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:50:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:17 smithi190 ceph-mon[33748]: 5.4 deep-scrub starts 2024-03-24T12:50:18.071 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:17 smithi190 ceph-mon[33748]: 5.4 deep-scrub ok 2024-03-24T12:50:18.676 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:18.676 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:50:18.676 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-03-24T12:50:18.676 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:18.970 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:18 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:50:18.970 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:18 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:50:18.970 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:18 smithi190 ceph-mon[33748]: 5.14 scrub starts 2024-03-24T12:50:18.970 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:18 smithi190 ceph-mon[33748]: 5.14 scrub ok 2024-03-24T12:50:18.970 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:18.970 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:50:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:50:19.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:18 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:50:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:18 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:50:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:18 smithi029 ceph-mon[29919]: 5.14 scrub starts 2024-03-24T12:50:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:18 smithi029 ceph-mon[29919]: 5.14 scrub ok 2024-03-24T12:50:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:18 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:50:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:18 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:50:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:18 smithi029 ceph-mon[41500]: 5.14 scrub starts 2024-03-24T12:50:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:18 smithi029 ceph-mon[41500]: 5.14 scrub ok 2024-03-24T12:50:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:19.288 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-24T12:50:19.372 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:50:19.536 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:19.931 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:19 smithi190 ceph-mon[33748]: 3.1 scrub starts 2024-03-24T12:50:19.931 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:19 smithi190 ceph-mon[33748]: 3.1 scrub ok 2024-03-24T12:50:19.931 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:19.931 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:19 smithi190 ceph-mon[33748]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:50:20.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:19 smithi029 ceph-mon[41500]: 3.1 scrub starts 2024-03-24T12:50:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:19 smithi029 ceph-mon[41500]: 3.1 scrub ok 2024-03-24T12:50:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:20.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:19 smithi029 ceph-mon[41500]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:50:20.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:19 smithi029 ceph-mon[29919]: 3.1 scrub starts 2024-03-24T12:50:20.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:19 smithi029 ceph-mon[29919]: 3.1 scrub ok 2024-03-24T12:50:20.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:20.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:19 smithi029 ceph-mon[29919]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:50:20.241 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:21.028 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:20 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:50:21.028 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:20 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:50:21.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:20 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:50:21.029 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:20 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:50:21.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:20 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:50:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:20 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:50:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:20 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:50:21.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:20 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:50:21.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:20 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:50:21.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:20 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:50:21.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:20 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:50:21.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:20 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:50:21.881 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:21 smithi190 ceph-mon[33748]: pgmap v1065: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 821 B/s rd, 0 op/s 2024-03-24T12:50:22.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:21 smithi029 ceph-mon[41500]: pgmap v1065: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 821 B/s rd, 0 op/s 2024-03-24T12:50:22.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:21 smithi029 ceph-mon[29919]: pgmap v1065: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 821 B/s rd, 0 op/s 2024-03-24T12:50:22.683 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:50:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-03-24T12:50:22.684 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-03-24T12:50:22.684 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:22 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:50:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:22 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:50:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:22 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:50:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:22 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:50:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:22 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:50:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:22 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:50:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:22 smithi190 ceph-mon[33748]: 1.4 scrub starts 2024-03-24T12:50:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:22 smithi190 ceph-mon[33748]: 1.4 scrub ok 2024-03-24T12:50:22.916 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:22.916 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:50:22.916 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-03-24T12:50:22.917 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:23.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:50:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:50:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:50:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:50:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:50:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:50:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[41500]: 1.4 scrub starts 2024-03-24T12:50:23.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[41500]: 1.4 scrub ok 2024-03-24T12:50:23.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:50:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:50:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:50:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:50:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:50:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:50:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[29919]: 1.4 scrub starts 2024-03-24T12:50:23.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:22 smithi029 ceph-mon[29919]: 1.4 scrub ok 2024-03-24T12:50:23.555 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-24T12:50:23.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:23 smithi190 ceph-mon[33748]: 3.c scrub starts 2024-03-24T12:50:23.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:23 smithi190 ceph-mon[33748]: 3.c scrub ok 2024-03-24T12:50:23.873 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:23 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:50:23.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:23 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:50:23.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:23 smithi190 ceph-mon[33748]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:50:23.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:23 smithi190 ceph-mon[33748]: 5.1a scrub starts 2024-03-24T12:50:23.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:23 smithi190 ceph-mon[33748]: 5.1a scrub ok 2024-03-24T12:50:24.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[41500]: 3.c scrub starts 2024-03-24T12:50:24.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[41500]: 3.c scrub ok 2024-03-24T12:50:24.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:50:24.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:50:24.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[41500]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:50:24.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[41500]: 5.1a scrub starts 2024-03-24T12:50:24.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[41500]: 5.1a scrub ok 2024-03-24T12:50:24.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[29919]: 3.c scrub starts 2024-03-24T12:50:24.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[29919]: 3.c scrub ok 2024-03-24T12:50:24.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:50:24.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:50:24.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[29919]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:50:24.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[29919]: 5.1a scrub starts 2024-03-24T12:50:24.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:23 smithi029 ceph-mon[29919]: 5.1a scrub ok 2024-03-24T12:50:24.191 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-24T12:50:24.192 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-24T12:50:24.192 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd in 5 2024-03-24T12:50:24.260 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:24.451 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:24.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:24 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:50:24.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:24 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:50:24.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:24 smithi029 ceph-mon[41500]: 1.8 scrub starts 2024-03-24T12:50:24.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:24 smithi029 ceph-mon[41500]: 1.8 scrub ok 2024-03-24T12:50:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:24 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:50:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:24 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:50:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:24 smithi029 ceph-mon[29919]: 1.8 scrub starts 2024-03-24T12:50:24.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:24 smithi029 ceph-mon[29919]: 1.8 scrub ok 2024-03-24T12:50:25.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:24 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:50:25.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:24 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:50:25.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:24 smithi190 ceph-mon[33748]: 1.8 scrub starts 2024-03-24T12:50:25.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:24 smithi190 ceph-mon[33748]: 1.8 scrub ok 2024-03-24T12:50:25.994 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:25 smithi190 ceph-mon[33748]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:50:25.995 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:25.995 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-03-24T12:50:25.995 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-03-24T12:50:25.995 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:50:25.995 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:25.995 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:185019 10.12 10:4f9c5757:::benchmark_data_smithi029_160664_object58045:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:50:25.995 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:56.368465+0000", 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 389.62971706000002, 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1435284029999999, 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185019 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:25.996 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368465+0000", 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368465+0000", 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368466+0000", 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8899999999999998e-07 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:25.997 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:25.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368470+0000", 2024-03-24T12:50:25.998 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.318e-06 2024-03-24T12:50:25.998 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:25.998 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:25.998 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:25.998 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368481+0000", 2024-03-24T12:50:25.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1601e-05 2024-03-24T12:50:25.999 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:25.999 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:25.999 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:25.999 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368490+0000", 2024-03-24T12:50:25.999 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2439999999999993e-06 2024-03-24T12:50:26.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368546+0000", 2024-03-24T12:50:26.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6743000000000002e-05 2024-03-24T12:50:26.000 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.000 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.000 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.000 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368556+0000", 2024-03-24T12:50:26.000 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.431e-06 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.493465+0000", 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1249090230000001 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for missing object", 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.493561+0000", 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6572999999999998e-05 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.001 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.511925+0000", 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.018363593000000001 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.511993+0000", 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7887999999999998e-05 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.002 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:185024 10.12 10:4b6f71a6:::benchmark_data_smithi029_160664_object58050:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:56.368507+0000", 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 389.62967451999998, 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1547719889999999, 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185024 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.003 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368507+0000", 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368507+0000", 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368508+0000", 2024-03-24T12:50:26.004 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9200000000000001e-07 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368511+0000", 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9299999999999999e-06 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368518+0000", 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9859999999999997e-06 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.005 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368524+0000", 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7490000000000001e-06 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368599+0000", 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4171999999999994e-05 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.368606+0000", 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3749999999999997e-06 2024-03-24T12:50:26.006 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.498894+0000", 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1302883260000001 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for missing object", 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.498975+0000", 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0455000000000001e-05 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.007 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.523206+0000", 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.024231058 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.523279+0000", 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3345999999999999e-05 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:185064 10.12 10:496f5ede:::benchmark_data_smithi029_160664_object58090:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:56.386289+0000", 2024-03-24T12:50:26.008 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 389.611892813, 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1193564659999999, 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185064 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.009 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386289+0000", 2024-03-24T12:50:26.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.010 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.010 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.010 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386289+0000", 2024-03-24T12:50:26.010 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.010 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.011 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.011 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386289+0000", 2024-03-24T12:50:26.011 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4300000000000003e-07 2024-03-24T12:50:26.011 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.011 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.012 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386292+0000", 2024-03-24T12:50:26.012 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0060000000000001e-06 2024-03-24T12:50:26.012 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.012 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.012 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.013 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386300+0000", 2024-03-24T12:50:26.013 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3470000000000001e-06 2024-03-24T12:50:26.013 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.013 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.013 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.386308+0000", 2024-03-24T12:50:26.014 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1499999999999999e-06 2024-03-24T12:50:26.014 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.014 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.014 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.014 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.389925+0000", 2024-03-24T12:50:26.015 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036172639999999998 2024-03-24T12:50:26.015 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.015 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.015 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.015 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.389945+0000", 2024-03-24T12:50:26.016 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9607999999999999e-05 2024-03-24T12:50:26.016 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.016 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.016 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.016 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.499022+0000", 2024-03-24T12:50:26.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.109077343 2024-03-24T12:50:26.017 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.017 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.017 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.017 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.499107+0000", 2024-03-24T12:50:26.017 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4967000000000002e-05 2024-03-24T12:50:26.018 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.018 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.018 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.018 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.499315+0000", 2024-03-24T12:50:26.018 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020838200000000001 2024-03-24T12:50:26.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.019 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.019 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505173+0000", 2024-03-24T12:50:26.019 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.005857831 2024-03-24T12:50:26.019 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.019 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.020 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505280+0000", 2024-03-24T12:50:26.020 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107166 2024-03-24T12:50:26.020 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.020 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.020 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505451+0000", 2024-03-24T12:50:26.021 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017029499999999999 2024-03-24T12:50:26.021 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.021 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.021 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.021 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505513+0000", 2024-03-24T12:50:26.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2137000000000005e-05 2024-03-24T12:50:26.022 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.022 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.022 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.022 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505645+0000", 2024-03-24T12:50:26.022 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013242700000000001 2024-03-24T12:50:26.023 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.023 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.023 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.023 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.023 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.023 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:185065 10.12 10:4e7c706a:::benchmark_data_smithi029_160664_object58091:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e341)", 2024-03-24T12:50:26.023 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:43:56.387863+0000", 2024-03-24T12:50:26.024 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 389.61031860600002, 2024-03-24T12:50:26.024 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1179670509999999, 2024-03-24T12:50:26.024 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.024 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.024 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.024 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.025 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.025 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 185065 2024-03-24T12:50:26.025 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.025 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.025 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.025 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.025 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387863+0000", 2024-03-24T12:50:26.025 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.026 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.026 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387863+0000", 2024-03-24T12:50:26.026 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.026 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.026 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387864+0000", 2024-03-24T12:50:26.027 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4000000000000001e-07 2024-03-24T12:50:26.027 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.027 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.027 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.027 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387867+0000", 2024-03-24T12:50:26.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3299999999999999e-06 2024-03-24T12:50:26.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.028 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.028 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.028 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387875+0000", 2024-03-24T12:50:26.028 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2539999999999992e-06 2024-03-24T12:50:26.028 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.029 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.029 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.387880+0000", 2024-03-24T12:50:26.029 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2240000000000003e-06 2024-03-24T12:50:26.029 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.029 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.030 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.390385+0000", 2024-03-24T12:50:26.030 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0025046170000000002 2024-03-24T12:50:26.030 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.030 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.030 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:56.390393+0000", 2024-03-24T12:50:26.031 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4590000000000001e-06 2024-03-24T12:50:26.031 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.031 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.031 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.031 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.500335+0000", 2024-03-24T12:50:26.032 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.109942121 2024-03-24T12:50:26.032 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.032 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.032 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.032 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.500422+0000", 2024-03-24T12:50:26.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7647999999999998e-05 2024-03-24T12:50:26.033 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.033 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.033 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.033 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.500567+0000", 2024-03-24T12:50:26.033 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014450800000000001 2024-03-24T12:50:26.034 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.034 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.034 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.034 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505179+0000", 2024-03-24T12:50:26.034 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046126179999999998 2024-03-24T12:50:26.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.035 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.035 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.035 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505398+0000", 2024-03-24T12:50:26.035 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021868199999999999 2024-03-24T12:50:26.035 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.036 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.036 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505719+0000", 2024-03-24T12:50:26.036 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00032113699999999998 2024-03-24T12:50:26.036 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.036 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.037 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505759+0000", 2024-03-24T12:50:26.037 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9745e-05 2024-03-24T12:50:26.037 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.037 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.037 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.038 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:43:57.505830+0000", 2024-03-24T12:50:26.038 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.1068000000000005e-05 2024-03-24T12:50:26.038 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.038 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.038 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.038 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.038 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.038 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242879 10.a 10:59050bb8:::benchmark_data_smithi029_160664_object115905:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-24T12:50:26.039 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.187072+0000", 2024-03-24T12:50:26.039 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.81111004500002, 2024-03-24T12:50:26.039 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.470001313, 2024-03-24T12:50:26.039 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.039 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:50:26.039 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.039 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.039 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.040 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242879 2024-03-24T12:50:26.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.040 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.040 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.040 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.040 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187072+0000", 2024-03-24T12:50:26.040 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.040 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187072+0000", 2024-03-24T12:50:26.041 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.041 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.041 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.041 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.041 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187072+0000", 2024-03-24T12:50:26.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4700000000000002e-07 2024-03-24T12:50:26.042 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.042 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.042 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.042 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187074+0000", 2024-03-24T12:50:26.042 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0209999999999998e-06 2024-03-24T12:50:26.044 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:25 smithi029 ceph-mon[41500]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:50:26.045 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:25 smithi029 ceph-mon[29919]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187079+0000", 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3249999999999998e-06 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.187084+0000", 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8119999999999998e-06 2024-03-24T12:50:26.046 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.189749+0000", 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026646040000000001 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.189867+0000", 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011839400000000001 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.189987+0000", 2024-03-24T12:50:26.047 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000119677 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.193807+0000", 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003819977 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.206807+0000", 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012999892000000001 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.048 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208165+0000", 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013583060000000001 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208176+0000", 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0784e-05 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657012+0000", 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4488366720000001 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.049 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657073+0000", 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0501999999999998e-05 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242894 10.a 10:56dffcb3:::benchmark_data_smithi029_160664_object115920:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.198057+0000", 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.80012435800001, 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4592828680000001, 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.050 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:50:26.051 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.051 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.051 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.051 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242894 2024-03-24T12:50:26.051 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.051 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.051 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.052 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198057+0000", 2024-03-24T12:50:26.052 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.052 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.052 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.052 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198057+0000", 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198058+0000", 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2109999999999999e-06 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198063+0000", 2024-03-24T12:50:26.053 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4710000000000001e-06 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198074+0000", 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0937999999999999e-05 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198082+0000", 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9200000000000004e-06 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.054 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198127+0000", 2024-03-24T12:50:26.055 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5082e-05 2024-03-24T12:50:26.055 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.055 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.055 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.055 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198213+0000", 2024-03-24T12:50:26.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5724999999999997e-05 2024-03-24T12:50:26.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.056 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.056 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198291+0000", 2024-03-24T12:50:26.056 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8238999999999994e-05 2024-03-24T12:50:26.056 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.056 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.206673+0000", 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083826409999999997 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208220+0000", 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0015461610000000001 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208230+0000", 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0702e-05 2024-03-24T12:50:26.057 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657299+0000", 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4490690509999999 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657340+0000", 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0726999999999998e-05 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.058 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242898 10.a 10:57906632:::benchmark_data_smithi029_160664_object115924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.198875+0000", 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.79930629, 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4585599380000001, 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242898 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.059 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198875+0000", 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198875+0000", 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198876+0000", 2024-03-24T12:50:26.060 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4099999999999999e-07 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198879+0000", 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7219999999999999e-06 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198885+0000", 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0959999999999997e-06 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.061 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198890+0000", 2024-03-24T12:50:26.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.972e-06 2024-03-24T12:50:26.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.062 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.200493+0000", 2024-03-24T12:50:26.062 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0016029149999999999 2024-03-24T12:50:26.062 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.062 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.062 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.200568+0000", 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5512000000000005e-05 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.200671+0000", 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010329500000000001 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.207043+0000", 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063716459999999999 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.063 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208325+0000", 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0012816150000000001 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208330+0000", 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6489999999999996e-06 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657383+0000", 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4490531529999999 2024-03-24T12:50:26.064 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657435+0000", 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1822e-05 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242899 10.a 10:5fc4cbc7:::benchmark_data_smithi029_160664_object115925:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e359)", 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.198897+0000", 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.79928442300002, 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4668848320000001, 2024-03-24T12:50:26.065 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:50:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242899 2024-03-24T12:50:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.066 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.067 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.067 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198897+0000", 2024-03-24T12:50:26.067 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.067 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.067 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.067 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198897+0000", 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198897+0000", 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4600000000000001e-07 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.068 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198899+0000", 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4500000000000001e-06 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198903+0000", 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3449999999999999e-06 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.198908+0000", 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5059999999999997e-06 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.069 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.202670+0000", 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003762095 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.202763+0000", 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3602000000000003e-05 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.070 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.202917+0000", 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015330399999999999 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.207114+0000", 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041972679999999997 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208358+0000", 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00124391 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.071 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.208363+0000", 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.366e-06 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657465+0000", 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4491021100000001 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for missing object", 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657581+0000", 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115147 2024-03-24T12:50:26.072 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665689+0000", 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0081088710000000001 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665782+0000", 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2612e-05 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.073 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242909 10.a 10:5a5b0db0:::benchmark_data_smithi029_160664_object115935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.219926+0000", 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.77825557099999, 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.445532576, 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.074 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242909 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219926+0000", 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219926+0000", 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.075 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219927+0000", 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9100000000000002e-07 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219930+0000", 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3969999999999998e-06 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219940+0000", 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9219999999999992e-06 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.076 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.219947+0000", 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3199999999999996e-06 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.220010+0000", 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3509000000000005e-05 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.220024+0000", 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4392e-05 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.077 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657678+0000", 2024-03-24T12:50:26.078 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.437653662 2024-03-24T12:50:26.078 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.078 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.078 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.078 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657789+0000", 2024-03-24T12:50:26.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000110549 2024-03-24T12:50:26.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.079 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.079 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.657898+0000", 2024-03-24T12:50:26.079 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010885100000000001 2024-03-24T12:50:26.079 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.079 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665298+0000", 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074007999999999999 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665309+0000", 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0481e-05 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665358+0000", 2024-03-24T12:50:26.080 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9308999999999998e-05 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665393+0000", 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4406999999999998e-05 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665459+0000", 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6086000000000004e-05 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.081 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242925 10.a 10:5108928b:::benchmark_data_smithi029_160664_object115951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.242462+0000", 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.75571931000002, 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.427779428, 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242925 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.082 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242462+0000", 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242462+0000", 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242463+0000", 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1500000000000003e-07 2024-03-24T12:50:26.083 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242467+0000", 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8829999999999999e-06 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242491+0000", 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3615999999999999e-05 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242499+0000", 2024-03-24T12:50:26.084 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3639999999999999e-06 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242562+0000", 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2490999999999995e-05 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.242571+0000", 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3330000000000003e-06 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.085 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.660542+0000", 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4179714620000001 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.660650+0000", 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010791800000000001 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.660767+0000", 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000116977 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.086 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665315+0000", 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0045479730000000003 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670080+0000", 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0047643410000000001 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670130+0000", 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0512e-05 2024-03-24T12:50:26.087 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670183+0000", 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2911000000000002e-05 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670242+0000", 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8731999999999999e-05 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.088 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242978 10.a 10:5cfa1afd:::benchmark_data_smithi029_160664_object116004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.279976+0000", 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.71820541300002, 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3931952830000001, 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242978 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.089 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.279976+0000", 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.279976+0000", 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.279977+0000", 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.207e-06 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.090 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.279982+0000", 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1070000000000004e-06 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.279995+0000", 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.218e-05 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280002+0000", 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5109999999999997e-06 2024-03-24T12:50:26.091 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280070+0000", 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7609999999999998e-05 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280078+0000", 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.456e-06 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.092 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.663048+0000", 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.382970129 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.663178+0000", 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000129457 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.663314+0000", 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013637999999999999 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.093 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665652+0000", 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0023377350000000001 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670287+0000", 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0046346520000000004 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673035+0000", 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0027482729999999999 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.094 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673086+0000", 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0701999999999997e-05 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673171+0000", 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5884000000000006e-05 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.095 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:242979 10.a 10:5130e7c7:::benchmark_data_smithi029_160664_object116005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.280020+0000", 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.71816202399998, 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.399110842, 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 242979 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.096 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280020+0000", 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280020+0000", 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280020+0000", 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2500000000000001e-07 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.097 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280022+0000", 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4040000000000002e-06 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280037+0000", 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4293e-05 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280042+0000", 2024-03-24T12:50:26.098 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4659999999999998e-06 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280186+0000", 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014369299999999999 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.280194+0000", 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6139999999999992e-06 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.099 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665801+0000", 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3856060989999999 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665887+0000", 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6231999999999997e-05 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.665965+0000", 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8607000000000001e-05 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.100 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670250+0000", 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0042849450000000001 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673222+0000", 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0029716479999999999 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.678966+0000", 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0057435560000000004 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.101 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.679019+0000", 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3699999999999997e-05 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.679130+0000", 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011116 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.102 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243008 10.a 10:5682a121:::benchmark_data_smithi029_160664_object116034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.298145+0000", 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.70003701600001, 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3832145630000001, 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243008 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.103 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298145+0000", 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298145+0000", 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298145+0000", 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6100000000000001e-07 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.104 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298148+0000", 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0939999999999999e-06 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298155+0000", 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8700000000000003e-06 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298161+0000", 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2389999999999999e-06 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.105 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298221+0000", 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.0004999999999999e-05 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.298229+0000", 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9680000000000006e-06 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.667328+0000", 2024-03-24T12:50:26.106 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3690987400000001 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.667444+0000", 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115541 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.667578+0000", 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013406099999999999 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.107 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670255+0000", 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026769789999999999 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673280+0000", 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0030254119999999999 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681242+0000", 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.007962406 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.108 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681281+0000", 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8198999999999998e-05 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681359+0000", 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8487999999999998e-05 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243059 10.a 10:530f856a:::benchmark_data_smithi029_160664_object116085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.109 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.343440+0000", 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.65474147499998, 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3380952930000001, 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243059 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343440+0000", 2024-03-24T12:50:26.110 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343440+0000", 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343441+0000", 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9400000000000005e-07 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.111 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343444+0000", 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2770000000000001e-06 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343453+0000", 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9900000000000003e-06 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343458+0000", 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.322e-06 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.112 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343515+0000", 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.643e-05 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.343522+0000", 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7440000000000007e-06 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.668816+0000", 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.325293437 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.668926+0000", 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011024 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.669004+0000", 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7571999999999995e-05 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670366+0000", 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001362458 2024-03-24T12:50:26.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681400+0000", 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011033612 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681444+0000", 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4146000000000001e-05 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681474+0000", 2024-03-24T12:50:26.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0300000000000001e-05 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681535+0000", 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1171000000000001e-05 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243073 10.a 10:5fcea2f5:::benchmark_data_smithi029_160664_object116099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.355624+0000", 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.6425577, 2024-03-24T12:50:26.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.339988797, 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243073 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355624+0000", 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355624+0000", 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355624+0000", 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.1099999999999996e-07 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355628+0000", 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.591e-06 2024-03-24T12:50:26.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355635+0000", 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2030000000000003e-06 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.355642+0000", 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4239999999999998e-06 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.358531+0000", 2024-03-24T12:50:26.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0028895750000000001 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.358539+0000", 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9300000000000003e-06 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670391+0000", 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.311852249 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670477+0000", 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5298999999999998e-05 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.670546+0000", 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9014e-05 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673233+0000", 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0026870029999999999 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681583+0000", 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0083506770000000008 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.695514+0000", 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.013930471 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.695562+0000", 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8649000000000001e-05 2024-03-24T12:50:26.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.695613+0000", 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0201000000000003e-05 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243083 10.a 10:59ed33c5:::benchmark_data_smithi029_160664_object116109:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.365353+0000", 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.63282868900001, 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.333598788, 2024-03-24T12:50:26.123 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243083 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365353+0000", 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365353+0000", 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365354+0000", 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.09e-07 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365357+0000", 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6600000000000001e-06 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365367+0000", 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6840000000000008e-06 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365373+0000", 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9000000000000003e-06 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365420+0000", 2024-03-24T12:50:26.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7219999999999999e-05 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.365427+0000", 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3060000000000003e-06 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.671684+0000", 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3062570259999999 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.671769+0000", 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5117000000000005e-05 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.671915+0000", 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000146324 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.674484+0000", 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00256939 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681628+0000", 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0071437640000000004 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.698875+0000", 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.017246678000000001 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.698916+0000", 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0667999999999998e-05 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.698952+0000", 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6142000000000001e-05 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243125 10.a 10:5a44a46d:::benchmark_data_smithi029_160664_object116151:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.404780+0000", 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.59340155799998, 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.294317451, 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.130 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243125 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404780+0000", 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404780+0000", 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404781+0000", 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2500000000000005e-07 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404785+0000", 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7790000000000002e-06 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404794+0000", 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0289999999999994e-06 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404799+0000", 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3639999999999998e-06 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404860+0000", 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1155999999999995e-05 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.404872+0000", 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1440000000000001e-05 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673313+0000", 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.268441698 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673391+0000", 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7636e-05 2024-03-24T12:50:26.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.673466+0000", 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4825000000000003e-05 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681202+0000", 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0077358399999999999 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.692661+0000", 2024-03-24T12:50:26.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011459517000000001 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.698999+0000", 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063383349999999996 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699056+0000", 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.677e-05 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699097+0000", 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1337000000000001e-05 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243130 10.a 10:5eea474b:::benchmark_data_smithi029_160664_object116156:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.408457+0000", 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.58972500900001, 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.290807756, 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.137 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243130 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408457+0000", 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408457+0000", 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408457+0000", 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6300000000000005e-07 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408460+0000", 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1099999999999999e-06 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408469+0000", 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2439999999999993e-06 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408474+0000", 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.2970000000000003e-06 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408515+0000", 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0802e-05 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.408521+0000", 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3929999999999996e-06 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.674521+0000", 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2659998699999999 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.674637+0000", 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000115696 2024-03-24T12:50:26.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.674721+0000", 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4814000000000005e-05 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681211+0000", 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064895100000000004 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.692899+0000", 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011688298999999999 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699162+0000", 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.006262821 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699218+0000", 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5619999999999999e-05 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699264+0000", 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6717e-05 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243131 10.a 10:50c3ecbe:::benchmark_data_smithi029_160664_object116157:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.411006+0000", 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.58717599200003, 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2883988180000001, 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.144 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243131 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411006+0000", 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411006+0000", 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411006+0000", 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4499999999999997e-07 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411009+0000", 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0879999999999999e-06 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411016+0000", 2024-03-24T12:50:26.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.2429999999999999e-06 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411021+0000", 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8969999999999999e-06 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411059+0000", 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7589000000000002e-05 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.411065+0000", 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.993e-06 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.675805+0000", 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.264740341 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.675885+0000", 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9600999999999999e-05 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.675962+0000", 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6913e-05 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681410+0000", 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0054483989999999996 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.692933+0000", 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011522897000000001 2024-03-24T12:50:26.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699321+0000", 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0063874530000000004 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699368+0000", 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7788999999999999e-05 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699404+0000", 2024-03-24T12:50:26.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6069999999999999e-05 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:243147 10.a 10:500de219:::benchmark_data_smithi029_160664_object116173:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e360)", 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:55.453385+0000", 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 330.54479649799998, 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.246141194, 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:26.151 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 243147 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453385+0000", 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453385+0000", 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453386+0000", 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9800000000000003e-07 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453388+0000", 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6029999999999999e-06 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453396+0000", 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4959999999999999e-06 2024-03-24T12:50:26.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453401+0000", 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.3229999999999997e-06 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453448+0000", 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7191000000000003e-05 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:55.453458+0000", 2024-03-24T12:50:26.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4760000000000004e-06 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.677016+0000", 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.223558675 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.677140+0000", 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000123387 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,2", 2024-03-24T12:50:26.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.677257+0000", 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000116666 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.681418+0000", 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0041616070000000003 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.692964+0000", 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.011546201000000001 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699446+0000", 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0064813780000000003 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699492+0000", 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6221999999999999e-05 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:56.699526+0000", 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4371e-05 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.157 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.158 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:26.158 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:26.158 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:26.777 INFO:teuthology.orchestra.run.smithi029.stderr:marked in osd.5. 2024-03-24T12:50:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:26 smithi190 ceph-mon[33748]: 5.1e deep-scrub starts 2024-03-24T12:50:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:26 smithi190 ceph-mon[33748]: 5.1e deep-scrub ok 2024-03-24T12:50:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:26 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2982081198' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-24T12:50:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:26 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-24T12:50:26.959 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-24T12:50:27.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:26 smithi029 ceph-mon[41500]: 5.1e deep-scrub starts 2024-03-24T12:50:27.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:26 smithi029 ceph-mon[41500]: 5.1e deep-scrub ok 2024-03-24T12:50:27.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:26 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2982081198' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-24T12:50:27.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:26 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-24T12:50:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:26 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a[29896]: 2024-03-24T12:50:26.684+0000 7f0eeb338700 -1 mon.a@0(leader).osd e438 definitely_dead 0 2024-03-24T12:50:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:26 smithi029 ceph-mon[29919]: 5.1e deep-scrub starts 2024-03-24T12:50:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:26 smithi029 ceph-mon[29919]: 5.1e deep-scrub ok 2024-03-24T12:50:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:26 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2982081198' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-24T12:50:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:26 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-24T12:50:27.199 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:27.642 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-24T12:50:27.642 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-24T12:50:27.642 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-03-24T12:50:27.642 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd in 6 2024-03-24T12:50:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:27 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-24T12:50:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:27 smithi190 ceph-mon[33748]: osdmap e439: 8 total, 8 up, 7 in 2024-03-24T12:50:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:27 smithi190 ceph-mon[33748]: 5.0 scrub starts 2024-03-24T12:50:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:27 smithi190 ceph-mon[33748]: 5.0 scrub ok 2024-03-24T12:50:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:27 smithi190 ceph-mon[33748]: 5.13 scrub starts 2024-03-24T12:50:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:27 smithi190 ceph-mon[33748]: 5.13 scrub ok 2024-03-24T12:50:27.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:27 smithi190 ceph-mon[33748]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:27.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:27 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:50:27.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:27 smithi190 ceph-mon[33748]: 4.4 scrub ok 2024-03-24T12:50:27.883 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:27 smithi190 ceph-mon[33748]: osdmap e440: 8 total, 8 up, 7 in 2024-03-24T12:50:27.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:50:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:50:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-24T12:50:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[29919]: osdmap e439: 8 total, 8 up, 7 in 2024-03-24T12:50:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[29919]: 5.0 scrub starts 2024-03-24T12:50:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[29919]: 5.0 scrub ok 2024-03-24T12:50:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[29919]: 5.13 scrub starts 2024-03-24T12:50:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[29919]: 5.13 scrub ok 2024-03-24T12:50:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[29919]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:50:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[29919]: 4.4 scrub ok 2024-03-24T12:50:27.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[29919]: osdmap e440: 8 total, 8 up, 7 in 2024-03-24T12:50:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-24T12:50:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[41500]: osdmap e439: 8 total, 8 up, 7 in 2024-03-24T12:50:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[41500]: 5.0 scrub starts 2024-03-24T12:50:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[41500]: 5.0 scrub ok 2024-03-24T12:50:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[41500]: 5.13 scrub starts 2024-03-24T12:50:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[41500]: 5.13 scrub ok 2024-03-24T12:50:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[41500]: pgmap v1069: 142 pgs: 142 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:50:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[41500]: 4.4 scrub ok 2024-03-24T12:50:27.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:27 smithi029 ceph-mon[41500]: osdmap e440: 8 total, 8 up, 7 in 2024-03-24T12:50:27.963 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:28.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:28 smithi190 ceph-mon[33748]: 6.b deep-scrub starts 2024-03-24T12:50:28.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:28 smithi190 ceph-mon[33748]: 6.b deep-scrub ok 2024-03-24T12:50:28.919 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:28 smithi190 ceph-mon[33748]: osdmap e441: 8 total, 8 up, 7 in 2024-03-24T12:50:28.919 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:50:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:50:28.937 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:50:28.937 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-03-24T12:50:28.937 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:29.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:28 smithi029 ceph-mon[41500]: 6.b deep-scrub starts 2024-03-24T12:50:29.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:28 smithi029 ceph-mon[41500]: 6.b deep-scrub ok 2024-03-24T12:50:29.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:28 smithi029 ceph-mon[41500]: osdmap e441: 8 total, 8 up, 7 in 2024-03-24T12:50:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:28 smithi029 ceph-mon[29919]: 6.b deep-scrub starts 2024-03-24T12:50:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:28 smithi029 ceph-mon[29919]: 6.b deep-scrub ok 2024-03-24T12:50:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:28 smithi029 ceph-mon[29919]: osdmap e441: 8 total, 8 up, 7 in 2024-03-24T12:50:29.716 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-24T12:50:29.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:50:29.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:50:29.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:29.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[29919]: 5.18 scrub starts 2024-03-24T12:50:29.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[29919]: 5.18 scrub ok 2024-03-24T12:50:29.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[29919]: pgmap v1072: 142 pgs: 1 remapped+peering, 25 peering, 116 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 79 B/s, 0 objects/s recovering 2024-03-24T12:50:29.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:50:29.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[29919]: 6.1b scrub starts 2024-03-24T12:50:29.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[29919]: osdmap e442: 8 total, 8 up, 7 in 2024-03-24T12:50:29.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:50:29.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:50:29.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:29.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[41500]: 5.18 scrub starts 2024-03-24T12:50:29.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[41500]: 5.18 scrub ok 2024-03-24T12:50:29.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[41500]: pgmap v1072: 142 pgs: 1 remapped+peering, 25 peering, 116 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 79 B/s, 0 objects/s recovering 2024-03-24T12:50:29.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:50:29.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[41500]: 6.1b scrub starts 2024-03-24T12:50:29.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:29 smithi029 ceph-mon[41500]: osdmap e442: 8 total, 8 up, 7 in 2024-03-24T12:50:29.970 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:30.093 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:29 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:50:30.093 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:29 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:50:30.093 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:29 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:30.093 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:29 smithi190 ceph-mon[33748]: 5.18 scrub starts 2024-03-24T12:50:30.093 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:29 smithi190 ceph-mon[33748]: 5.18 scrub ok 2024-03-24T12:50:30.093 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:29 smithi190 ceph-mon[33748]: pgmap v1072: 142 pgs: 1 remapped+peering, 25 peering, 116 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s; 79 B/s, 0 objects/s recovering 2024-03-24T12:50:30.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:29 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:50:30.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:29 smithi190 ceph-mon[33748]: 6.1b scrub starts 2024-03-24T12:50:30.094 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:29 smithi190 ceph-mon[33748]: osdmap e442: 8 total, 8 up, 7 in 2024-03-24T12:50:30.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a[29896]: 2024-03-24T12:50:30.085+0000 7f0eeb338700 -1 mon.a@0(leader).osd e442 definitely_dead 0 2024-03-24T12:50:30.781 INFO:teuthology.orchestra.run.smithi029.stderr:marked in osd.6. 2024-03-24T12:50:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:30 smithi190 ceph-mon[33748]: 6.1b scrub ok 2024-03-24T12:50:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:30 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:50:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:30 smithi190 ceph-mon[33748]: 6.11 deep-scrub starts 2024-03-24T12:50:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:30 smithi190 ceph-mon[33748]: 6.11 deep-scrub ok 2024-03-24T12:50:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:30 smithi190 ceph-mon[33748]: 4.14 scrub starts 2024-03-24T12:50:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:30 smithi190 ceph-mon[33748]: 4.14 scrub ok 2024-03-24T12:50:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:30 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3462979588' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-24T12:50:31.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:30 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3462979588' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-24T12:50:31.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:30 smithi190 ceph-mon[33748]: osdmap e443: 8 total, 8 up, 8 in 2024-03-24T12:50:31.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[41500]: 6.1b scrub ok 2024-03-24T12:50:31.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:50:31.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[41500]: 6.11 deep-scrub starts 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[41500]: 6.11 deep-scrub ok 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[41500]: 4.14 scrub starts 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[41500]: 4.14 scrub ok 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3462979588' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3462979588' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[41500]: osdmap e443: 8 total, 8 up, 8 in 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[29919]: 6.1b scrub ok 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[29919]: 6.11 deep-scrub starts 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[29919]: 6.11 deep-scrub ok 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[29919]: 4.14 scrub starts 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[29919]: 4.14 scrub ok 2024-03-24T12:50:31.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3462979588' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-03-24T12:50:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3462979588' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-03-24T12:50:31.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:30 smithi029 ceph-mon[29919]: osdmap e443: 8 total, 8 up, 8 in 2024-03-24T12:50:31.856 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-03-24T12:50:31.856 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-24T12:50:31.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd pool set .mgr pgp_num 11 2024-03-24T12:50:32.091 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:31 smithi190 ceph-mon[33748]: 5.1c scrub starts 2024-03-24T12:50:32.091 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:31 smithi190 ceph-mon[33748]: 5.1c scrub ok 2024-03-24T12:50:32.091 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:31 smithi190 ceph-mon[33748]: pgmap v1075: 142 pgs: 1 remapped+peering, 29 peering, 112 active+clean; 315 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 146 KiB/s, 14 objects/s recovering 2024-03-24T12:50:32.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:31 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:50:32.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:31 smithi190 ceph-mon[33748]: 3.13 scrub starts 2024-03-24T12:50:32.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:31 smithi190 ceph-mon[33748]: 3.13 scrub ok 2024-03-24T12:50:32.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:31 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:50:32.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:31 smithi190 ceph-mon[33748]: 5.1b scrub starts 2024-03-24T12:50:32.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:31 smithi190 ceph-mon[33748]: 5.1b scrub ok 2024-03-24T12:50:32.092 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:31 smithi190 ceph-mon[33748]: osdmap e444: 8 total, 8 up, 8 in 2024-03-24T12:50:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:32.128 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:50:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-03-24T12:50:32.128 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-03-24T12:50:32.128 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:32.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[41500]: 5.1c scrub starts 2024-03-24T12:50:32.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[41500]: 5.1c scrub ok 2024-03-24T12:50:32.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[41500]: pgmap v1075: 142 pgs: 1 remapped+peering, 29 peering, 112 active+clean; 315 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 146 KiB/s, 14 objects/s recovering 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[41500]: 3.13 scrub starts 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[41500]: 3.13 scrub ok 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[41500]: 5.1b scrub starts 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[41500]: 5.1b scrub ok 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[41500]: osdmap e444: 8 total, 8 up, 8 in 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[29919]: 5.1c scrub starts 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[29919]: 5.1c scrub ok 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[29919]: pgmap v1075: 142 pgs: 1 remapped+peering, 29 peering, 112 active+clean; 315 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 146 KiB/s, 14 objects/s recovering 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:50:32.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[29919]: 3.13 scrub starts 2024-03-24T12:50:32.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[29919]: 3.13 scrub ok 2024-03-24T12:50:32.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:50:32.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[29919]: 5.1b scrub starts 2024-03-24T12:50:32.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[29919]: 5.1b scrub ok 2024-03-24T12:50:32.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:31 smithi029 ceph-mon[29919]: osdmap e444: 8 total, 8 up, 8 in 2024-03-24T12:50:33.063 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:32 smithi190 ceph-mon[33748]: 6.9 deep-scrub starts 2024-03-24T12:50:33.063 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:32 smithi190 ceph-mon[33748]: 6.9 deep-scrub ok 2024-03-24T12:50:33.063 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:32 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:50:33.063 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:32 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:50:33.063 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:32 smithi190 ceph-mon[33748]: osdmap e445: 8 total, 8 up, 8 in 2024-03-24T12:50:33.094 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-24T12:50:33.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:32 smithi029 ceph-mon[29919]: 6.9 deep-scrub starts 2024-03-24T12:50:33.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:32 smithi029 ceph-mon[29919]: 6.9 deep-scrub ok 2024-03-24T12:50:33.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:32 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:50:33.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:32 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:50:33.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:32 smithi029 ceph-mon[29919]: osdmap e445: 8 total, 8 up, 8 in 2024-03-24T12:50:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:32 smithi029 ceph-mon[41500]: 6.9 deep-scrub starts 2024-03-24T12:50:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:32 smithi029 ceph-mon[41500]: 6.9 deep-scrub ok 2024-03-24T12:50:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:32 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:50:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:32 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:50:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:32 smithi029 ceph-mon[41500]: osdmap e445: 8 total, 8 up, 8 in 2024-03-24T12:50:33.341 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:34.042 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:50:34.042 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:50:34.042 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:50:34.042 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:50:34.042 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[41500]: 6.1f scrub starts 2024-03-24T12:50:34.042 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[41500]: pgmap v1078: 142 pgs: 1 active+undersized+remapped, 3 remapped+peering, 8 peering, 130 active+clean; 315 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 146 KiB/s, 32 objects/s recovering 2024-03-24T12:50:34.042 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[41500]: 6.1f scrub ok 2024-03-24T12:50:34.042 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:34.042 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:50:34.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:50:34.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:50:34.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:50:34.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[29919]: 6.1f scrub starts 2024-03-24T12:50:34.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[29919]: pgmap v1078: 142 pgs: 1 active+undersized+remapped, 3 remapped+peering, 8 peering, 130 active+clean; 315 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 146 KiB/s, 32 objects/s recovering 2024-03-24T12:50:34.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[29919]: 6.1f scrub ok 2024-03-24T12:50:34.043 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:33 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:34.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:33 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:50:34.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:33 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:50:34.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:33 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:50:34.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:33 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:50:34.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:33 smithi190 ceph-mon[33748]: 6.1f scrub starts 2024-03-24T12:50:34.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:33 smithi190 ceph-mon[33748]: pgmap v1078: 142 pgs: 1 active+undersized+remapped, 3 remapped+peering, 8 peering, 130 active+clean; 315 KiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 7.5 KiB/s rd, 55 KiB/s wr, 9 op/s; 146 KiB/s, 32 objects/s recovering 2024-03-24T12:50:34.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:33 smithi190 ceph-mon[33748]: 6.1f scrub ok 2024-03-24T12:50:34.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:33 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:34.813 INFO:teuthology.orchestra.run.smithi029.stderr:set pool 1 pgp_num to 11 2024-03-24T12:50:35.073 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:34 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:50:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:34 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:50:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:34 smithi190 ceph-mon[33748]: osdmap e446: 8 total, 8 up, 8 in 2024-03-24T12:50:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:34 smithi190 ceph-mon[33748]: 4.2 scrub starts 2024-03-24T12:50:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:34 smithi190 ceph-mon[33748]: 4.2 scrub ok 2024-03-24T12:50:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:34 smithi190 ceph-mon[33748]: 4.f deep-scrub starts 2024-03-24T12:50:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:34 smithi190 ceph-mon[33748]: 4.f deep-scrub ok 2024-03-24T12:50:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:34 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2286596057' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:50:35.074 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:34 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:50:35.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[41500]: osdmap e446: 8 total, 8 up, 8 in 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[41500]: 4.2 scrub starts 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[41500]: 4.2 scrub ok 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[41500]: 4.f deep-scrub starts 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[41500]: 4.f deep-scrub ok 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2286596057' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:50:35.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[29919]: osdmap e446: 8 total, 8 up, 8 in 2024-03-24T12:50:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[29919]: 4.2 scrub starts 2024-03-24T12:50:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[29919]: 4.2 scrub ok 2024-03-24T12:50:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[29919]: 4.f deep-scrub starts 2024-03-24T12:50:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[29919]: 4.f deep-scrub ok 2024-03-24T12:50:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2286596057' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:50:35.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:34 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-03-24T12:50:35.342 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:55123 10.c 10:340df47c:::benchmark_data_smithi029_160664_object55122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e286)", 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:41:32.704477+0000", 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 542.640833993, 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.374830463, 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.343 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 55123 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.704477+0000", 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.704477+0000", 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.344 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.704478+0000", 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2359999999999999e-06 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.704539+0000", 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1402999999999995e-05 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.345 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.704550+0000", 2024-03-24T12:50:35.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0726e-05 2024-03-24T12:50:35.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.346 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.704556+0000", 2024-03-24T12:50:35.346 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8139999999999997e-06 2024-03-24T12:50:35.346 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.346 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.346 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.704594+0000", 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8447e-05 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.704687+0000", 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2521000000000004e-05 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 0,4", 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.704786+0000", 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.9071000000000001e-05 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.347 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.706117+0000", 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013312580000000001 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:32.706468+0000", 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00035138999999999998 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:33.079193+0000", 2024-03-24T12:50:35.348 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.37272466399999998 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:33.079248+0000", 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5365999999999998e-05 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:41:33.079307+0000", 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8566999999999997e-05 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.349 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:192917 10.13 10:ca5ffae4:::benchmark_data_smithi029_160664_object65943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.679981+0000", 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.66532921599998, 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.941135202, 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 192917 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.350 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.679981+0000", 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.679981+0000", 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.679983+0000", 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.24e-06 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.351 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.679986+0000", 2024-03-24T12:50:35.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7890000000000001e-06 2024-03-24T12:50:35.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.352 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.352 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.679995+0000", 2024-03-24T12:50:35.352 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.4910000000000005e-06 2024-03-24T12:50:35.352 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.352 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.680002+0000", 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7599999999999997e-06 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.680056+0000", 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4216999999999999e-05 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.680064+0000", 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8569999999999995e-06 2024-03-24T12:50:35.353 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.473325+0000", 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.79326086600000001 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.473459+0000", 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013458 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.354 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.473702+0000", 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00024251899999999999 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.477815+0000", 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.004113574 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.477862+0000", 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.6553e-05 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.355 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.621007+0000", 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.143145523 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.621072+0000", 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.4647999999999995e-05 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.621117+0000", 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4585000000000001e-05 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.356 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:192968 10.13 10:cae880e0:::benchmark_data_smithi029_160664_object65994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.717061+0000", 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.628249471, 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.90667096400000002, 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 192968 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.357 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.717061+0000", 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.717061+0000", 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.717062+0000", 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7399999999999991e-07 2024-03-24T12:50:35.358 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.717067+0000", 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0270000000000003e-06 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.717079+0000", 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1807999999999999e-05 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.717087+0000", 2024-03-24T12:50:35.359 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3410000000000003e-06 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.717149+0000", 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1397999999999997e-05 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.717158+0000", 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6379999999999999e-06 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.475448+0000", 2024-03-24T12:50:35.360 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.75828988799999997 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.475518+0000", 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9772999999999998e-05 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.475733+0000", 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000215188 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.361 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.477947+0000", 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022140710000000002 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.477980+0000", 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.2363000000000002e-05 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.623670+0000", 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.145690141 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.362 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.623703+0000", 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3284000000000002e-05 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.623732+0000", 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9070000000000001e-05 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.363 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:192972 10.13 10:cd47eca8:::benchmark_data_smithi029_160664_object65998:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.719935+0000", 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.62537564299998, 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.90391434699999995, 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 192972 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.364 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.719935+0000", 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.719935+0000", 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.719936+0000", 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8199999999999998e-07 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.365 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.719941+0000", 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8559999999999998e-06 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.719955+0000", 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4620000000000001e-05 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.719963+0000", 2024-03-24T12:50:35.366 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0730000000000005e-06 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.720012+0000", 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8364e-05 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.720019+0000", 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6190000000000002e-06 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.367 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.476676+0000", 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.75665634900000001 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.476788+0000", 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011239900000000001 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.477006+0000", 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021747100000000001 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.368 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.478030+0000", 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010246750000000001 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.478043+0000", 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2598e-05 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.623779+0000", 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14573630900000001 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.369 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.623810+0000", 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1031999999999997e-05 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.623849+0000", 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9100000000000002e-05 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.370 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.371 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.372 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:192993 10.13 10:c9b40310:::benchmark_data_smithi029_160664_object66019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.372 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.734017+0000", 2024-03-24T12:50:35.372 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.61129400700003, 2024-03-24T12:50:35.372 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.88990424700000004, 2024-03-24T12:50:35.372 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.372 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.372 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.372 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.372 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 192993 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734017+0000", 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734017+0000", 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.373 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734018+0000", 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1000000000000001e-06 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734022+0000", 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7310000000000001e-06 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734032+0000", 2024-03-24T12:50:35.374 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.3319999999999998e-06 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734038+0000", 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.852e-06 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734093+0000", 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5915999999999998e-05 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.375 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734103+0000", 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4129999999999995e-06 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.478070+0000", 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.74396675300000004 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.478139+0000", 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.8954000000000004e-05 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.376 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.478332+0000", 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019291499999999999 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.481470+0000", 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0031386719999999999 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.481487+0000", 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6789000000000001e-05 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.377 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.623872+0000", 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.142385492 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.623900+0000", 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7977999999999999e-05 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.623921+0000", 2024-03-24T12:50:35.378 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.035e-05 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:192995 10.13 10:cb21f75e:::benchmark_data_smithi029_160664_object66021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.734203+0000", 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.61110717000003, 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.89109651400000001, 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.379 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 192995 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734203+0000", 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734203+0000", 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.380 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734204+0000", 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3799999999999998e-07 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734206+0000", 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1569999999999998e-06 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734212+0000", 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2960000000000004e-06 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.381 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734216+0000", 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0489999999999997e-06 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734255+0000", 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8918000000000002e-05 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.734263+0000", 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.593e-06 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.382 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.479235+0000", 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.74497245899999998 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.479348+0000", 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011312899999999999 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.479557+0000", 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020836199999999999 2024-03-24T12:50:35.383 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.481510+0000", 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0019530490000000001 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.482697+0000", 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0011874780000000001 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.625210+0000", 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.142512638 2024-03-24T12:50:35.384 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.625260+0000", 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9558999999999998e-05 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.625300+0000", 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.0389000000000002e-05 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:193032 10.13 10:ccca8329:::benchmark_data_smithi029_160664_object66058:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.385 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.758314+0000", 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.58699663200002, 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.86711671400000001, 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 193032 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.386 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.758314+0000", 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.758314+0000", 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.758316+0000", 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.72e-06 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.387 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.758320+0000", 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7090000000000001e-06 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.758337+0000", 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6237999999999999e-05 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.758345+0000", 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.8030000000000003e-06 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.388 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.758405+0000", 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9500999999999998e-05 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.758415+0000", 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0125e-05 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.480413+0000", 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.72199826300000003 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.480480+0000", 2024-03-24T12:50:35.389 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.6722000000000003e-05 2024-03-24T12:50:35.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.480637+0000", 2024-03-24T12:50:35.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015694799999999999 2024-03-24T12:50:35.390 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.390 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.390 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.390 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.483848+0000", 2024-03-24T12:50:35.390 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032112310000000002 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.483896+0000", 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7712e-05 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.625343+0000", 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14144757399999999 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.391 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.625393+0000", 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9434999999999997e-05 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.625431+0000", 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7732999999999999e-05 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.392 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:193055 10.13 10:c896c6f5:::benchmark_data_smithi029_160664_object66081:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.775560+0000", 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.56975078400001, 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.85164539500000003, 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 193055 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.393 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.775560+0000", 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.775560+0000", 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.775561+0000", 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5510000000000001e-06 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.394 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.775566+0000", 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.9869999999999999e-06 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.775576+0000", 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0043e-05 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.775584+0000", 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0029999999999994e-06 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.395 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.775640+0000", 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5792999999999999e-05 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.775646+0000", 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8619999999999999e-06 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.481548+0000", 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.70590233300000005 2024-03-24T12:50:35.396 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.481658+0000", 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011011 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.481887+0000", 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000228352 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.483948+0000", 2024-03-24T12:50:35.397 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.002061046 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.483963+0000", 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5268999999999999e-05 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.627142+0000", 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14317865699999999 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.398 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.627180+0000", 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.7778999999999999e-05 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.627205+0000", 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.561e-05 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:193059 10.13 10:c8f5d155:::benchmark_data_smithi029_160664_object66085:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.399 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.776999+0000", 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.56831149099997, 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.85215823300000004, 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 193059 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.776999+0000", 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.400 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.776999+0000", 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.777000+0000", 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.9699999999999995e-07 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.777003+0000", 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.6100000000000002e-06 2024-03-24T12:50:35.401 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.777014+0000", 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0454e-05 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.777020+0000", 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7969999999999997e-06 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.777090+0000", 2024-03-24T12:50:35.402 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.0737999999999999e-05 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.777103+0000", 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2531000000000001e-05 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.482727+0000", 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.70562398299999995 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.482830+0000", 2024-03-24T12:50:35.403 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00010320899999999999 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.483048+0000", 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021757100000000001 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.486296+0000", 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032479570000000001 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.404 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.486309+0000", 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3227999999999999e-05 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.629058+0000", 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14274911400000001 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.629114+0000", 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5958999999999997e-05 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.405 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.629157+0000", 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.3384999999999999e-05 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:193114 10.13 10:cbfc4014:::benchmark_data_smithi029_160664_object66140:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.822830+0000", 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.522480433, 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.81643754999999996, 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.406 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 193114 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.822830+0000", 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.407 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.822830+0000", 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.822831+0000", 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2500000000000001e-06 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.822845+0000", 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3151e-05 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.408 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.822857+0000", 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2678e-05 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.822867+0000", 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7920000000000004e-06 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.822923+0000", 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6283999999999998e-05 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.409 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.822932+0000", 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9709999999999993e-06 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.483990+0000", 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.66105764300000003 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.484065+0000", 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4819000000000003e-05 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.410 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.484253+0000", 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000188123 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.486316+0000", 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020629770000000001 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.486364+0000", 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7873000000000001e-05 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.411 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639197+0000", 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15283349199999999 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639236+0000", 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8874000000000001e-05 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639268+0000", 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1622999999999999e-05 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.412 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:193224 10.13 10:ca0f6d1c:::benchmark_data_smithi029_160664_object66250:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.910626+0000", 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.43468483499998, 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.72874261799999995, 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 193224 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.413 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.910626+0000", 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.910626+0000", 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.414 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.910627+0000", 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.358e-06 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.910632+0000", 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0340000000000004e-06 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.910644+0000", 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2232000000000001e-05 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.415 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.910655+0000", 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1007e-05 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.910721+0000", 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5757000000000001e-05 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.910729+0000", 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0499999999999992e-06 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.416 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.485114+0000", 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.57438494299999998 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.485226+0000", 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00011167699999999999 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.485433+0000", 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020721800000000001 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.417 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.488680+0000", 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032468169999999999 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.488725+0000", 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.5392999999999997e-05 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639306+0000", 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15058073499999999 2024-03-24T12:50:35.418 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639341+0000", 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4987000000000002e-05 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639368+0000", 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.741e-05 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.419 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:193257 10.13 10:c96de69f:::benchmark_data_smithi029_160664_object66283:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.940563+0000", 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.40474732899997, 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.69889190099999998, 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 193257 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.940563+0000", 2024-03-24T12:50:35.420 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.940563+0000", 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.940565+0000", 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5099999999999999e-06 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.940570+0000", 2024-03-24T12:50:35.421 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.677e-06 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.940583+0000", 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2424e-05 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.940591+0000", 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.2190000000000004e-06 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.422 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.940653+0000", 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.2268999999999996e-05 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.940662+0000", 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9090000000000006e-06 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.486402+0000", 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.545740008 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.423 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.486515+0000", 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000112732 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.486722+0000", 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00020659899999999999 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.488778+0000", 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020567910000000001 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.424 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.488792+0000", 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3701e-05 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639401+0000", 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.15060879599999999 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639430+0000", 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.9419000000000001e-05 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.425 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639455+0000", 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4847000000000001e-05 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:193288 10.13 10:c9f17763:::benchmark_data_smithi029_160664_object66314:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:06.971593+0000", 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.37371723199999, 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.66793410099999995, 2024-03-24T12:50:35.426 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 193288 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.971593+0000", 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.427 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.971593+0000", 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.971595+0000", 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.26e-06 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.971599+0000", 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2440000000000004e-06 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.428 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.971616+0000", 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7195000000000001e-05 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.971624+0000", 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.3070000000000003e-06 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.971672+0000", 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.7874999999999998e-05 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.429 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:06.971678+0000", 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9379999999999997e-06 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.487564+0000", 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51588620600000001 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.487670+0000", 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000106079 2024-03-24T12:50:35.430 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.487880+0000", 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0002098 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.491109+0000", 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0032290370000000001 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.491152+0000", 2024-03-24T12:50:35.431 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.2837999999999999e-05 2024-03-24T12:50:35.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.432 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639479+0000", 2024-03-24T12:50:35.432 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.148326448 2024-03-24T12:50:35.432 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.432 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.432 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639506+0000", 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.7409000000000001e-05 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.639527+0000", 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1464999999999999e-05 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.433 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:193322 10.13 10:c8fbc85f:::benchmark_data_smithi029_160664_object66348:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:07.012859+0000", 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.33245187300002, 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.62759707200000003, 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 193322 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.434 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.012859+0000", 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.012859+0000", 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.012860+0000", 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0979999999999999e-06 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.435 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.012864+0000", 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8279999999999996e-06 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.012873+0000", 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.9330000000000007e-06 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.436 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.012878+0000", 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0130000000000001e-06 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.012926+0000", 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.8393999999999998e-05 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.012932+0000", 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5289999999999997e-06 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.437 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.488806+0000", 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.47587385599999998 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.488887+0000", 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0309000000000006e-05 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.489080+0000", 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019369499999999999 2024-03-24T12:50:35.438 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.491207+0000", 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0021265289999999998 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.491221+0000", 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3687e-05 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.640399+0000", 2024-03-24T12:50:35.439 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.149178432 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.640432+0000", 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.3284000000000002e-05 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.640456+0000", 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3485e-05 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.440 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:193333 10.13 10:c821e32a:::benchmark_data_smithi029_160664_object66359:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:07.026892+0000", 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.31841863199998, 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.61367667699999995, 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 193333 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.441 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.026892+0000", 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.026892+0000", 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.442 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.026893+0000", 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-06 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.026897+0000", 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4599999999999996e-06 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.026908+0000", 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0720000000000001e-05 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.443 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.026915+0000", 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3690000000000003e-06 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.026960+0000", 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4960999999999999e-05 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.026965+0000", 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8590000000000001e-06 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.444 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.489985+0000", 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.46301999799999999 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.490094+0000", 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000108782 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.490309+0000", 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000214699 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.445 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.492346+0000", 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0020371180000000001 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.492370+0000", 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4263000000000001e-05 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.640492+0000", 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.148121373 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.446 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.640530+0000", 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8609999999999998e-05 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.640569+0000", 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8393000000000003e-05 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.447 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.15279.0:193370 10.13 10:cbc547bb:::benchmark_data_smithi029_160664_object66396:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:44:07.122169+0000", 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 388.22314187199999, 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.51856897800000001, 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.15279", 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2019030380", 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 193370 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.448 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.122169+0000", 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.122169+0000", 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.122170+0000", 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.381e-06 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.449 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.122176+0000", 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1120000000000001e-06 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.122188+0000", 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1419e-05 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.122197+0000", 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1989999999999995e-06 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.450 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.122256+0000", 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9098000000000002e-05 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.122267+0000", 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1474999999999999e-05 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.491236+0000", 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.36896817300000001 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.451 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.491310+0000", 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.4586999999999996e-05 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1,2", 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.491481+0000", 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00017065 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.492512+0000", 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001031012 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.452 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.492523+0000", 2024-03-24T12:50:35.453 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1233000000000001e-05 2024-03-24T12:50:35.453 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.453 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.453 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.453 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.640618+0000", 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.14809544199999999 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.640699+0000", 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.0080000000000006e-05 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:44:07.640738+0000", 2024-03-24T12:50:35.454 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.9116999999999998e-05 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25328.0:162696 11.0 11:0353e9af:::benchmark_data_smithi029_192894_object57761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e417)", 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:48:24.992833+0000", 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 130.352477738, 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0350427149999999, 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25328", 2024-03-24T12:50:35.455 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 162696 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.992833+0000", 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.992833+0000", 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.456 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.992834+0000", 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3510000000000001e-06 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.992843+0000", 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2629999999999992e-06 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.992855+0000", 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1199000000000001e-05 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.457 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.992864+0000", 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.1039999999999996e-06 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.001337+0000", 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0084729239999999997 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.001340+0000", 2024-03-24T12:50:35.458 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.738e-06 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.024348+0000", 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0230071220000001 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.024480+0000", 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000132237 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1", 2024-03-24T12:50:35.459 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.024623+0000", 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014307 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.026870+0000", 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0022472220000000001 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.027716+0000", 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00084561200000000003 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.460 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.027773+0000", 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.7070000000000001e-05 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.027876+0000", 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000102803 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25328.0:162693 11.9 11:9011c58a:::benchmark_data_smithi029_192894_object57758:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:50:35.461 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:48:24.995520+0000", 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 130.349791049, 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0393089090000001, 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25328", 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 162693 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.462 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.995520+0000", 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.995520+0000", 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.995521+0000", 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2640000000000001e-06 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.463 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.995525+0000", 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.1910000000000002e-06 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.995535+0000", 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0283e-05 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.995542+0000", 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7750000000000004e-06 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.464 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.996461+0000", 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00091932199999999998 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:24.996470+0000", 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.5250000000000005e-06 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.029574+0000", 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.033104365 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.465 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for missing object", 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.029671+0000", 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6397000000000006e-05 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.034738+0000", 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0050674960000000003 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.034828+0000", 2024-03-24T12:50:35.466 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.0290999999999993e-05 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.25328.0:162706 11.9 11:9812b946:::benchmark_data_smithi029_192894_object57771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e418)", 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:48:25.041742+0000", 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 130.30356826799999, 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.99365104999999998, 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.25328", 2024-03-24T12:50:35.467 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/2122830000", 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 162706 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.041742+0000", 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.041742+0000", 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.468 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.041744+0000", 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1430000000000001e-06 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.041754+0000", 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7969999999999995e-06 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.041781+0000", 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6636e-05 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.469 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.041797+0000", 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5705999999999998e-05 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.041904+0000", 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000107462 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.041912+0000", 2024-03-24T12:50:35.470 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.1310000000000006e-06 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.029852+0000", 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.98793975899999997 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.029987+0000", 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013530700000000001 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for subops from 1", 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.030110+0000", 2024-03-24T12:50:35.471 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012285900000000001 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "op_commit", 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.034565+0000", 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0044553889999999997 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "sub_op_commit_rec", 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.035235+0000", 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00066923799999999995 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "commit_sent", 2024-03-24T12:50:35.472 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.035298+0000", 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3711e-05 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.035393+0000", 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.4912000000000002e-05 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24427.0:1328 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e417)", 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:48:25.160988+0000", 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 130.18432251999999, 2024-03-24T12:50:35.473 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.84552833699999996, 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24427", 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.190:0/4234972492", 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 1328 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.160988+0000", 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.474 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.160988+0000", 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.160990+0000", 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.277e-06 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.161003+0000", 2024-03-24T12:50:35.475 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.288e-05 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.161030+0000", 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6611999999999999e-05 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.161044+0000", 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4398e-05 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.161136+0000", 2024-03-24T12:50:35.476 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.2015000000000006e-05 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:25.161146+0000", 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6909999999999992e-06 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.006117+0000", 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.84497153899999999 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.006242+0000", 2024-03-24T12:50:35.477 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00012467399999999999 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:48:26.006516+0000", 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027425099999999999 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:35.478 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:35.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-24T12:50:35.990 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:36.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: 6.10 scrub starts 2024-03-24T12:50:36.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: 6.10 scrub ok 2024-03-24T12:50:36.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:50:36.112 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: osdmap e447: 8 total, 8 up, 8 in 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: pgmap v1081: 142 pgs: 1 active+undersized+remapped, 3 remapped+peering, 5 peering, 133 active+clean; 315 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 478 B/s, 26 objects/s recovering 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: 4.1d scrub starts 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: 4.1d scrub ok 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: 6.10 scrub starts 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: 6.10 scrub ok 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:50:36.113 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:50:36.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T12:50:36.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: osdmap e447: 8 total, 8 up, 8 in 2024-03-24T12:50:36.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: pgmap v1081: 142 pgs: 1 active+undersized+remapped, 3 remapped+peering, 5 peering, 133 active+clean; 315 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 478 B/s, 26 objects/s recovering 2024-03-24T12:50:36.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: 4.1d scrub starts 2024-03-24T12:50:36.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: 4.1d scrub ok 2024-03-24T12:50:36.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:50:36.114 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:35 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:50:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: 6.10 scrub starts 2024-03-24T12:50:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: 6.10 scrub ok 2024-03-24T12:50:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:50:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:50:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-03-24T12:50:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: osdmap e447: 8 total, 8 up, 8 in 2024-03-24T12:50:36.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: pgmap v1081: 142 pgs: 1 active+undersized+remapped, 3 remapped+peering, 5 peering, 133 active+clean; 315 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 478 B/s, 26 objects/s recovering 2024-03-24T12:50:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: 4.1d scrub starts 2024-03-24T12:50:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: 4.1d scrub ok 2024-03-24T12:50:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:50:36.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:35 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:50:36.266 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-24T12:50:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:36 smithi190 ceph-mon[33748]: 1.7 scrub starts 2024-03-24T12:50:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:36 smithi190 ceph-mon[33748]: 1.7 scrub ok 2024-03-24T12:50:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:36 smithi190 ceph-mon[33748]: 1.4 deep-scrub starts 2024-03-24T12:50:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:36 smithi190 ceph-mon[33748]: 1.4 deep-scrub ok 2024-03-24T12:50:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:36 smithi190 ceph-mon[33748]: 3.17 scrub starts 2024-03-24T12:50:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:36 smithi190 ceph-mon[33748]: 3.17 scrub ok 2024-03-24T12:50:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:36 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:50:37.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:36 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:50:37.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:36 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:50:37.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:36 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:50:37.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[29919]: 1.7 scrub starts 2024-03-24T12:50:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[29919]: 1.7 scrub ok 2024-03-24T12:50:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[29919]: 1.4 deep-scrub starts 2024-03-24T12:50:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[29919]: 1.4 deep-scrub ok 2024-03-24T12:50:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[29919]: 3.17 scrub starts 2024-03-24T12:50:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[29919]: 3.17 scrub ok 2024-03-24T12:50:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:50:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:50:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:50:37.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:50:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[41500]: 1.7 scrub starts 2024-03-24T12:50:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[41500]: 1.7 scrub ok 2024-03-24T12:50:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[41500]: 1.4 deep-scrub starts 2024-03-24T12:50:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[41500]: 1.4 deep-scrub ok 2024-03-24T12:50:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[41500]: 3.17 scrub starts 2024-03-24T12:50:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[41500]: 3.17 scrub ok 2024-03-24T12:50:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:50:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:50:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:50:37.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:36 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:50:37.304 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:37.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:37 smithi029 ceph-mon[29919]: 1.3 deep-scrub starts 2024-03-24T12:50:37.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:37 smithi029 ceph-mon[29919]: 1.3 deep-scrub ok 2024-03-24T12:50:37.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:37 smithi029 ceph-mon[29919]: pgmap v1082: 142 pgs: 1 active+undersized+remapped, 3 remapped+peering, 1 peering, 137 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 26 objects/s recovering 2024-03-24T12:50:37.846 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:37 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:50:37.847 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:50:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:50:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:50:37.847 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:37 smithi029 ceph-mon[41500]: 1.3 deep-scrub starts 2024-03-24T12:50:37.847 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:37 smithi029 ceph-mon[41500]: 1.3 deep-scrub ok 2024-03-24T12:50:37.847 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:37 smithi029 ceph-mon[41500]: pgmap v1082: 142 pgs: 1 active+undersized+remapped, 3 remapped+peering, 1 peering, 137 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 26 objects/s recovering 2024-03-24T12:50:37.847 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:37 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:50:37.859 INFO:teuthology.orchestra.run.smithi029.stdout:mon.a: {} 2024-03-24T12:50:37.872 INFO:teuthology.orchestra.run.smithi029.stdout:mon.b: {} 2024-03-24T12:50:37.882 INFO:teuthology.orchestra.run.smithi029.stdout:mon.c: {} 2024-03-24T12:50:37.883 INFO:teuthology.orchestra.run.smithi029.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T12:50:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T12:50:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-24T12:50:38.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:37 smithi190 ceph-mon[33748]: 1.3 deep-scrub starts 2024-03-24T12:50:38.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:37 smithi190 ceph-mon[33748]: 1.3 deep-scrub ok 2024-03-24T12:50:38.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:37 smithi190 ceph-mon[33748]: pgmap v1082: 142 pgs: 1 active+undersized+remapped, 3 remapped+peering, 1 peering, 137 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 26 objects/s recovering 2024-03-24T12:50:38.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:37 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:50:39.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:50:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:50:39.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:38 smithi190 ceph-mon[33748]: 5.15 scrub starts 2024-03-24T12:50:39.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:38 smithi190 ceph-mon[33748]: 5.15 scrub ok 2024-03-24T12:50:39.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:38 smithi190 ceph-mon[33748]: 4.3 scrub starts 2024-03-24T12:50:39.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:38 smithi190 ceph-mon[33748]: 4.3 scrub ok 2024-03-24T12:50:39.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:38 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/211984966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:50:39.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:38 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:50:39.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:38 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:50:39.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:38 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:50:39.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[41500]: 5.15 scrub starts 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[41500]: 5.15 scrub ok 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[41500]: 4.3 scrub starts 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[41500]: 4.3 scrub ok 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/211984966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[29919]: 5.15 scrub starts 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[29919]: 5.15 scrub ok 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[29919]: 4.3 scrub starts 2024-03-24T12:50:39.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[29919]: 4.3 scrub ok 2024-03-24T12:50:39.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/211984966' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T12:50:39.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:50:39.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:50:39.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:38 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:50:39.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd reweight 1 1 2024-03-24T12:50:40.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:39 smithi029 ceph-mon[29919]: 5.2 scrub starts 2024-03-24T12:50:40.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:39 smithi029 ceph-mon[29919]: 5.2 scrub ok 2024-03-24T12:50:40.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:39 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:40.124 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:39 smithi029 ceph-mon[29919]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 8 objects/s recovering 2024-03-24T12:50:40.125 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:39 smithi029 ceph-mon[41500]: 5.2 scrub starts 2024-03-24T12:50:40.125 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:39 smithi029 ceph-mon[41500]: 5.2 scrub ok 2024-03-24T12:50:40.125 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:39 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:40.125 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:39 smithi029 ceph-mon[41500]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 8 objects/s recovering 2024-03-24T12:50:40.131 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:39 smithi190 ceph-mon[33748]: 5.2 scrub starts 2024-03-24T12:50:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:39 smithi190 ceph-mon[33748]: 5.2 scrub ok 2024-03-24T12:50:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:39 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:39 smithi190 ceph-mon[33748]: pgmap v1083: 142 pgs: 142 active+clean; 582 KiB data, 5.1 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 8 objects/s recovering 2024-03-24T12:50:40.378 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:40.380 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:50:40.380 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:50:40.381 INFO:teuthology.orchestra.run.smithi029.stdout: "num_ops": 0 2024-03-24T12:50:40.381 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:50:41.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[41500]: 3.1f scrub starts 2024-03-24T12:50:41.059 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[41500]: 3.1f scrub ok 2024-03-24T12:50:41.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[41500]: 6.4 scrub starts 2024-03-24T12:50:41.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[41500]: 6.4 scrub ok 2024-03-24T12:50:41.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[41500]: 4.8 scrub starts 2024-03-24T12:50:41.060 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[41500]: 4.8 scrub ok 2024-03-24T12:50:41.060 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[29919]: 3.1f scrub starts 2024-03-24T12:50:41.060 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[29919]: 3.1f scrub ok 2024-03-24T12:50:41.060 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[29919]: 6.4 scrub starts 2024-03-24T12:50:41.060 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[29919]: 6.4 scrub ok 2024-03-24T12:50:41.060 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[29919]: 4.8 scrub starts 2024-03-24T12:50:41.060 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:40 smithi029 ceph-mon[29919]: 4.8 scrub ok 2024-03-24T12:50:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:40 smithi190 ceph-mon[33748]: 3.1f scrub starts 2024-03-24T12:50:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:40 smithi190 ceph-mon[33748]: 3.1f scrub ok 2024-03-24T12:50:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:40 smithi190 ceph-mon[33748]: 6.4 scrub starts 2024-03-24T12:50:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:40 smithi190 ceph-mon[33748]: 6.4 scrub ok 2024-03-24T12:50:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:40 smithi190 ceph-mon[33748]: 4.8 scrub starts 2024-03-24T12:50:41.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:40 smithi190 ceph-mon[33748]: 4.8 scrub ok 2024-03-24T12:50:41.948 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-24T12:50:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:41 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:50:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:41 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:50:42.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:41 smithi190 ceph-mon[33748]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-03-24T12:50:42.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:41 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:50:42.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:41 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:50:42.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:41 smithi029 ceph-mon[41500]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-03-24T12:50:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:41 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:50:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:41 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:50:42.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:41 smithi029 ceph-mon[29919]: pgmap v1084: 142 pgs: 142 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 6 objects/s recovering 2024-03-24T12:50:43.087 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:42 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:50:43.087 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:42 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:50:43.087 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:42 smithi029 ceph-mon[41500]: 3.8 scrub starts 2024-03-24T12:50:43.087 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:42 smithi029 ceph-mon[41500]: 3.8 scrub ok 2024-03-24T12:50:43.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:42 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:50:43.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:42 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:50:43.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:42 smithi029 ceph-mon[29919]: 3.8 scrub starts 2024-03-24T12:50:43.087 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:42 smithi029 ceph-mon[29919]: 3.8 scrub ok 2024-03-24T12:50:43.093 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:43.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:42 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:50:43.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:42 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:50:43.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:42 smithi190 ceph-mon[33748]: 3.8 scrub starts 2024-03-24T12:50:43.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:42 smithi190 ceph-mon[33748]: 3.8 scrub ok 2024-03-24T12:50:43.844 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.1 to 1 (10000) 2024-03-24T12:50:43.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:43 smithi029 ceph-mon[41500]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2024-03-24T12:50:43.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:43 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/156158381' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:43.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:43 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:43.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:43 smithi029 ceph-mon[29919]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2024-03-24T12:50:43.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:43 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/156158381' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:43.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:43 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:43 smithi190 ceph-mon[33748]: pgmap v1085: 142 pgs: 142 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 3 objects/s recovering 2024-03-24T12:50:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:43 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/156158381' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:43 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:44.640 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-24T12:50:44.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:44 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:50:44.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:44 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:50:44.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:44 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:50:44.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:44 smithi029 ceph-mon[41500]: osdmap e448: 8 total, 8 up, 8 in 2024-03-24T12:50:44.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:44 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:50:44.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:44 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:50:44.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:44 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:50:44.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:44 smithi029 ceph-mon[29919]: osdmap e448: 8 total, 8 up, 8 in 2024-03-24T12:50:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:44 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:50:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:44 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:50:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:44 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:50:45.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:44 smithi190 ceph-mon[33748]: osdmap e448: 8 total, 8 up, 8 in 2024-03-24T12:50:45.706 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:50:45.706 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [], 2024-03-24T12:50:45.706 INFO:teuthology.orchestra.run.smithi029.stdout: "complaint_time": 30, 2024-03-24T12:50:45.706 INFO:teuthology.orchestra.run.smithi029.stdout: "num_blocked_ops": 0 2024-03-24T12:50:45.706 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:50:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:45 smithi190 ceph-mon[33748]: 3.12 scrub starts 2024-03-24T12:50:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:45 smithi190 ceph-mon[33748]: 3.12 scrub ok 2024-03-24T12:50:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:45 smithi190 ceph-mon[33748]: 6.17 deep-scrub starts 2024-03-24T12:50:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:45 smithi190 ceph-mon[33748]: pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-24T12:50:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:45 smithi190 ceph-mon[33748]: 6.17 deep-scrub ok 2024-03-24T12:50:46.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:45 smithi190 ceph-mon[33748]: 6.3 scrub starts 2024-03-24T12:50:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[29919]: 3.12 scrub starts 2024-03-24T12:50:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[29919]: 3.12 scrub ok 2024-03-24T12:50:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[29919]: 6.17 deep-scrub starts 2024-03-24T12:50:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[29919]: pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-24T12:50:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[29919]: 6.17 deep-scrub ok 2024-03-24T12:50:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[29919]: 6.3 scrub starts 2024-03-24T12:50:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[41500]: 3.12 scrub starts 2024-03-24T12:50:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[41500]: 3.12 scrub ok 2024-03-24T12:50:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[41500]: 6.17 deep-scrub starts 2024-03-24T12:50:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[41500]: pgmap v1087: 142 pgs: 142 active+clean; 582 KiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 0 B/s, 2 objects/s recovering 2024-03-24T12:50:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[41500]: 6.17 deep-scrub ok 2024-03-24T12:50:46.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:45 smithi029 ceph-mon[41500]: 6.3 scrub starts 2024-03-24T12:50:46.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-24T12:50:46.823 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:46.992 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[29919]: 6.3 scrub ok 2024-03-24T12:50:46.992 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[29919]: 5.6 scrub starts 2024-03-24T12:50:46.992 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[29919]: 5.6 scrub ok 2024-03-24T12:50:46.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[29919]: 3.e scrub starts 2024-03-24T12:50:46.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[29919]: 3.e scrub ok 2024-03-24T12:50:46.993 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[29919]: 6.1d scrub starts 2024-03-24T12:50:46.993 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[41500]: 6.3 scrub ok 2024-03-24T12:50:46.993 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[41500]: 5.6 scrub starts 2024-03-24T12:50:46.993 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[41500]: 5.6 scrub ok 2024-03-24T12:50:46.993 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[41500]: 3.e scrub starts 2024-03-24T12:50:46.993 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[41500]: 3.e scrub ok 2024-03-24T12:50:46.993 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:46 smithi029 ceph-mon[41500]: 6.1d scrub starts 2024-03-24T12:50:47.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:46 smithi190 ceph-mon[33748]: 6.3 scrub ok 2024-03-24T12:50:47.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:46 smithi190 ceph-mon[33748]: 5.6 scrub starts 2024-03-24T12:50:47.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:46 smithi190 ceph-mon[33748]: 5.6 scrub ok 2024-03-24T12:50:47.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:46 smithi190 ceph-mon[33748]: 3.e scrub starts 2024-03-24T12:50:47.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:46 smithi190 ceph-mon[33748]: 3.e scrub ok 2024-03-24T12:50:47.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:46 smithi190 ceph-mon[33748]: 6.1d scrub starts 2024-03-24T12:50:47.699 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:47.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:50:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:50:47.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[41500]: 6.1d scrub ok 2024-03-24T12:50:47.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:50:47.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:50:47.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:50:47.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:50:47.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[41500]: pgmap v1088: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:47.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 2]}]: dispatch 2024-03-24T12:50:47.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 7]}]: dispatch 2024-03-24T12:50:47.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 6]}]: dispatch 2024-03-24T12:50:47.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[29919]: 6.1d scrub ok 2024-03-24T12:50:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:50:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:50:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:50:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:50:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[29919]: pgmap v1088: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 2]}]: dispatch 2024-03-24T12:50:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 7]}]: dispatch 2024-03-24T12:50:47.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:47 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 6]}]: dispatch 2024-03-24T12:50:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:47 smithi190 ceph-mon[33748]: 6.1d scrub ok 2024-03-24T12:50:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:47 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:50:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:47 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:50:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:47 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:50:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:47 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:50:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:47 smithi190 ceph-mon[33748]: pgmap v1088: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:50:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 2]}]: dispatch 2024-03-24T12:50:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 7]}]: dispatch 2024-03-24T12:50:48.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:47 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 6]}]: dispatch 2024-03-24T12:50:48.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[41500]: 3.18 deep-scrub starts 2024-03-24T12:50:48.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[41500]: 3.18 deep-scrub ok 2024-03-24T12:50:48.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 2]}]': finished 2024-03-24T12:50:48.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 7]}]': finished 2024-03-24T12:50:48.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 6]}]': finished 2024-03-24T12:50:48.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[41500]: osdmap e449: 8 total, 8 up, 8 in 2024-03-24T12:50:48.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:50:48.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:50:48.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:48.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[29919]: 3.18 deep-scrub starts 2024-03-24T12:50:48.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[29919]: 3.18 deep-scrub ok 2024-03-24T12:50:48.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 2]}]': finished 2024-03-24T12:50:48.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 7]}]': finished 2024-03-24T12:50:48.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 6]}]': finished 2024-03-24T12:50:48.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[29919]: osdmap e449: 8 total, 8 up, 8 in 2024-03-24T12:50:48.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:50:48.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:50:48.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:48 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:49.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:50:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:50:49.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:48 smithi190 ceph-mon[33748]: 3.18 deep-scrub starts 2024-03-24T12:50:49.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:48 smithi190 ceph-mon[33748]: 3.18 deep-scrub ok 2024-03-24T12:50:49.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 2]}]': finished 2024-03-24T12:50:49.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 7]}]': finished 2024-03-24T12:50:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.8", "id": [0, 6]}]': finished 2024-03-24T12:50:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:48 smithi190 ceph-mon[33748]: osdmap e449: 8 total, 8 up, 8 in 2024-03-24T12:50:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:48 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:50:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:48 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:50:49.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:48 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:50:50.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:49 smithi029 ceph-mon[29919]: 5.3 scrub starts 2024-03-24T12:50:50.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:49 smithi029 ceph-mon[29919]: 5.3 scrub ok 2024-03-24T12:50:50.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:49 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:50.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:49 smithi029 ceph-mon[29919]: osdmap e450: 8 total, 8 up, 8 in 2024-03-24T12:50:50.010 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:49 smithi029 ceph-mon[29919]: pgmap v1091: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:50:50.010 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:49 smithi029 ceph-mon[41500]: 5.3 scrub starts 2024-03-24T12:50:50.010 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:49 smithi029 ceph-mon[41500]: 5.3 scrub ok 2024-03-24T12:50:50.010 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:49 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:50.011 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:49 smithi029 ceph-mon[41500]: osdmap e450: 8 total, 8 up, 8 in 2024-03-24T12:50:50.011 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:49 smithi029 ceph-mon[41500]: pgmap v1091: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:50:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:49 smithi190 ceph-mon[33748]: 5.3 scrub starts 2024-03-24T12:50:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:49 smithi190 ceph-mon[33748]: 5.3 scrub ok 2024-03-24T12:50:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:49 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:50:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:49 smithi190 ceph-mon[33748]: osdmap e450: 8 total, 8 up, 8 in 2024-03-24T12:50:50.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:49 smithi190 ceph-mon[33748]: pgmap v1091: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:50:50.474 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: "size": 20, 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 600, 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: "ops": [ 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:4688 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 93828934255616 gen 6] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:53.491995+0000", 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 56.986190137000001, 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.87008702299999996, 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.475 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4688 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.491995+0000", 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.491995+0000", 2024-03-24T12:50:50.476 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.491996+0000", 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0759999999999999e-06 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492000+0000", 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.6909999999999999e-06 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.477 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492009+0000", 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6519999999999995e-06 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492018+0000", 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.0380000000000005e-06 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492074+0000", 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6449000000000001e-05 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.478 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492087+0000", 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2904999999999999e-05 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.342183+0000", 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.85009591799999995 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.342475+0000", 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00029212500000000001 2024-03-24T12:50:50.479 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.362082+0000", 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.019607169000000001 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:4689 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 93828934249856 gen 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e433)", 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:53.492040+0000", 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 56.986145264000001, 2024-03-24T12:50:50.480 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.87207518100000003, 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4689 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492040+0000", 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.481 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492040+0000", 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492041+0000", 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7299999999999996e-07 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.482 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492043+0000", 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.5660000000000001e-06 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492048+0000", 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2549999999999997e-06 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492054+0000", 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0329999999999997e-06 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.483 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492100+0000", 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.5577000000000001e-05 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:53.492109+0000", 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8149999999999995e-06 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.361903+0000", 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.86979369500000003 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.484 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.362053+0000", 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000150679 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.364115+0000", 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0020619879999999998 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.485 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "pg_update_log_missing(5.e epoch 434/433 rep_tid 78059 entries 434'13 (0'0) error 5:7759931f:::notify.3:head by client.24344.0:4690 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:54.342370+0000", 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 56.135815483999998, 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.011161753999999999, 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.342370+0000", 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.486 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.342370+0000", 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.342370+0000", 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.1199999999999998e-07 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.342376+0000", 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.631e-06 2024-03-24T12:50:50.487 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.342387+0000", 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1517e-05 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.342401+0000", 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3431000000000001e-05 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.342507+0000", 2024-03-24T12:50:50.488 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000106591 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.353532+0000", 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.011024172 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "pg_update_log_missing_reply(5.15 epoch 434/433 rep_tid 0 lcod 434'13)", 2024-03-24T12:50:50.489 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:49:54.346595+0000", 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 56.131590481000003, 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026415079999999999, 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.346595+0000", 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.490 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.346595+0000", 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.346595+0000", 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.1900000000000003e-07 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.346599+0000", 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2940000000000001e-06 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.491 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.346606+0000", 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.2810000000000003e-06 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.346621+0000", 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5153e-05 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.346710+0000", 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.8609000000000005e-05 2024-03-24T12:50:50.492 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:49:54.349236+0000", 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0025266519999999999 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "MOSDScrubReserve(4.19,QREQUEST,e:435,from:2,reservation_nonce:2)", 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:06.012455+0000", 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 44.46573025, 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0021950339999999998, 2024-03-24T12:50:50.493 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.012455+0000", 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.012455+0000", 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.494 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.012456+0000", 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2890000000000001e-06 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.012464+0000", 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7360000000000001e-06 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.012480+0000", 2024-03-24T12:50:50.495 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5491000000000001e-05 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.012492+0000", 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2763e-05 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.012557+0000", 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.4573999999999995e-05 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op must wait for map", 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.012562+0000", 2024-03-24T12:50:50.496 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.502e-06 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.014291+0000", 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0017281040000000001 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.014321+0000", 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.0896999999999999e-05 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.497 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:06.014650+0000", 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000328678 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_op(client.24344.0:4755 5.3 5:c609908c:::notify.5:head [watch reconnect cookie 93828934249856 gen 10] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:26.771454+0000", 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 23.706730812, 2024-03-24T12:50:50.498 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0693268009999999, 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: "client_info": { 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: "client": "client.24344", 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: "tid": 4755 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:26.771454+0000", 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.499 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:26.771454+0000", 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:26.771456+0000", 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.719e-06 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:26.771464+0000", 2024-03-24T12:50:50.500 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7160000000000003e-06 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:26.771488+0000", 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.3759000000000001e-05 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:26.771502+0000", 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3825000000000001e-05 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.501 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:26.771601+0000", 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9450000000000005e-05 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "op must wait for map", 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:26.771610+0000", 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.5590000000000006e-06 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:26.792099+0000", 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.020489944 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.502 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for peered", 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:26.792111+0000", 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1959e-05 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.812097+0000", 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0199850850000001 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "waiting for degraded object", 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.812539+0000", 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00044278999999999998 2024-03-24T12:50:50.503 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.504 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.504 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.504 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.824680+0000", 2024-03-24T12:50:50.504 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012140822000000001 2024-03-24T12:50:50.504 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.504 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.824761+0000", 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0538999999999996e-05 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.840781+0000", 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.016020633999999999 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "pg_update_log_missing(5.e epoch 440/439 rep_tid 78060 entries 440'14 (0'0) error 5:7759931f:::notify.3:head by client.24344.0:4757 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-24T12:50:50.505 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:27.810070+0000", 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 22.668115068999999, 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0077965960000000003, 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.810070+0000", 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.506 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.810070+0000", 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.810071+0000", 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.8800000000000002e-07 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.810079+0000", 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.2800000000000003e-06 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.507 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.810466+0000", 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00038665100000000001 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.810484+0000", 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7816999999999999e-05 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.817501+0000", 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0070175180000000004 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.508 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.817867+0000", 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00036574200000000001 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "MOSDPGPull(5.15 e440/439 cost 1049576)", 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:27.811509+0000", 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 22.666676542000001, 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0028184920000000001, 2024-03-24T12:50:50.509 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.811509+0000", 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.811509+0000", 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.510 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.811509+0000", 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.65e-07 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.811517+0000", 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.4349999999999997e-06 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.811523+0000", 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.276e-06 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.811528+0000", 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.7529999999999998e-06 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.814217+0000", 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026880850000000002 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.814327+0000", 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.000110678 2024-03-24T12:50:50.512 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "pg_update_log_missing(5.15 epoch 440/439 rep_tid 0 entries 440'14 (0'0) error 5:a93a5511:::notify.2:head by client.24344.0:4754 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:27.825573+0000", 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 22.652612529999999, 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.016084153, 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.513 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.825573+0000", 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.825573+0000", 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.825573+0000", 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.49e-07 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.514 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.825578+0000", 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.2429999999999996e-06 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.825587+0000", 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.6899999999999998e-06 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.825597+0000", 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9930000000000007e-06 2024-03-24T12:50:50.515 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.838468+0000", 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012871494000000001 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.841657+0000", 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0031885839999999999 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.516 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "pg_update_log_missing_reply(5.3 epoch 440/439 rep_tid 2 lcod 440'19)", 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:27.826347+0000", 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 22.65183824, 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012549334000000001, 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826347+0000", 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.517 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826347+0000", 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826348+0000", 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.5700000000000002e-07 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826351+0000", 2024-03-24T12:50:50.518 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.92e-06 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826362+0000", 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1297e-05 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826368+0000", 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.195e-06 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.838878+0000", 2024-03-24T12:50:50.519 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012509579 2024-03-24T12:50:50.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.521 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.521 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.521 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.838896+0000", 2024-03-24T12:50:50.521 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.8785999999999999e-05 2024-03-24T12:50:50.521 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.521 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.521 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.521 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "pg_update_log_missing_reply(5.3 epoch 440/439 rep_tid 2 lcod 440'19)", 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:27.826423+0000", 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 22.651762777999998, 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.012550364, 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826423+0000", 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.522 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826423+0000", 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826423+0000", 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.9099999999999999e-07 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826425+0000", 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.294e-06 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.523 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826432+0000", 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.5699999999999998e-06 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.826437+0000", 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0239999999999997e-06 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.838957+0000", 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.01252027 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.524 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.838973+0000", 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5815e-05 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "MOSDPGPull(4.15 e440/439 cost 1049576)", 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:27.829725+0000", 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 22.648460650000001, 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0095404159999999995, 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.525 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.829725+0000", 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.829725+0000", 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.829725+0000", 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.1299999999999999e-07 2024-03-24T12:50:50.526 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.829739+0000", 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.3200999999999999e-05 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.829755+0000", 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6722999999999999e-05 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.829764+0000", 2024-03-24T12:50:50.527 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.7169999999999997e-06 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.839181+0000", 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0094171840000000003 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.839265+0000", 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3777999999999996e-05 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.528 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "MOSDPGPull(2.2 e440/439 cost 1049576)", 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:27.837943+0000", 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 22.640242564000001, 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002566357, 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "reached pg", 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.837943+0000", 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.529 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.837943+0000", 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.837944+0000", 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2200000000000002e-07 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.837959+0000", 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5551999999999999e-05 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.530 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.837970+0000", 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0966e-05 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.837980+0000", 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.8600000000000005e-06 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.838041+0000", 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.1154000000000004e-05 2024-03-24T12:50:50.531 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:27.840509+0000", 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002467903 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "rep_scrubmap(6.11 e439 from shard 2)", 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:29.634033+0000", 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 20.844152548, 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.002677699, 2024-03-24T12:50:50.532 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634033+0000", 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634033+0000", 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.533 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634033+0000", 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4300000000000003e-07 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634037+0000", 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.4539999999999999e-06 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.534 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634047+0000", 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0248000000000001e-05 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634054+0000", 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.7689999999999999e-06 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.636682+0000", 2024-03-24T12:50:50.535 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026281149999999999 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.636693+0000", 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.1556000000000001e-05 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.636710+0000", 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.7014e-05 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.536 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "rep_scrubmap(6.11 e439 from shard 5)", 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:29.634066+0000", 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 20.844119483, 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026853990000000002, 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634066+0000", 2024-03-24T12:50:50.537 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634066+0000", 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634066+0000", 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.58e-07 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.538 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634070+0000", 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.72e-06 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634086+0000", 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.5611999999999999e-05 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.634092+0000", 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.8220000000000001e-06 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.539 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.636725+0000", 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0026322490000000001 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.636730+0000", 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.4709999999999998e-06 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:29.636751+0000", 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.1067e-05 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.540 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "rep_scrubmap(6.9 e433 from shard 2)", 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:31.625789+0000", 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 18.852396606999999, 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0069052209999999996, 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:31.625789+0000", 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.541 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:31.625789+0000", 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:31.625790+0000", 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.015e-06 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:31.625795+0000", 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.6370000000000004e-06 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.542 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:31.625808+0000", 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.2649999999999999e-05 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:31.625816+0000", 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.9589999999999995e-06 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:31.632667+0000", 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.006851289 2024-03-24T12:50:50.543 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:31.632677+0000", 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.9189999999999994e-06 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:31.632694+0000", 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.6752000000000002e-05 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.544 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "rep_scrubmap(6.10 e439 from shard 1)", 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:34.712126+0000", 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 15.766059221000001, 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0025165090000000001, 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712126+0000", 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.545 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712126+0000", 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712127+0000", 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.3799999999999996e-07 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712130+0000", 2024-03-24T12:50:50.546 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2320000000000001e-06 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712138+0000", 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.3550000000000005e-06 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712144+0000", 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.631e-06 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.547 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.714627+0000", 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0024830099999999999 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.714633+0000", 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 6.0270000000000001e-06 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.714643+0000", 2024-03-24T12:50:50.548 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.516e-06 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "rep_scrubmap(6.10 e439 from shard 5)", 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:34.712512+0000", 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 15.765672949000001, 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0021830859999999999, 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.549 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712512+0000", 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712512+0000", 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712513+0000", 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.9599999999999999e-07 2024-03-24T12:50:50.550 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712516+0000", 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 3.2559999999999998e-06 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712524+0000", 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.678e-06 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.551 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.712529+0000", 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.8019999999999999e-06 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.714664+0000", 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0021349580000000002 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.714671+0000", 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5379999999999997e-06 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:34.714695+0000", 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 2.4258000000000001e-05 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(osd.0.0:4 4.0 e450/439 ondisk, result = 0)", 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:49.620969+0000", 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.85721596600000005, 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0019692009999999999, 2024-03-24T12:50:50.553 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.620969+0000", 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.620969+0000", 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.554 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.620970+0000", 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.5700000000000002e-07 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.620975+0000", 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.7829999999999998e-06 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.620984+0000", 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.499e-06 2024-03-24T12:50:50.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.620992+0000", 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 7.7789999999999996e-06 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.622801+0000", 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.001809204 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.622812+0000", 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0718e-05 2024-03-24T12:50:50.556 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.622939+0000", 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.00012646099999999999 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: "description": "osd_repop_reply(osd.0.0:5 4.0 e450/439 ondisk, result = 0)", 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: "initiated_at": "2024-03-24T12:50:49.620997+0000", 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: "age": 0.85718835800000004, 2024-03-24T12:50:50.557 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0019964919999999999, 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: "type_data": { 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: "flag_point": "started", 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: "events": [ 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "initiated", 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.620997+0000", 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "header_read", 2024-03-24T12:50:50.558 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.620997+0000", 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "throttled", 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.620998+0000", 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 9.2900000000000002e-07 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "all_read", 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.621002+0000", 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 4.0470000000000004e-06 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.559 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "dispatched", 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.621013+0000", 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0723999999999999e-05 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "queued_for_pg", 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.621021+0000", 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 8.0010000000000001e-06 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "reached_pg", 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.622977+0000", 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 0.0019567400000000002 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.560 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "started", 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.622982+0000", 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 5.0760000000000002e-06 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: "event": "done", 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: "time": "2024-03-24T12:50:49.622993+0000", 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: "duration": 1.0975000000000001e-05 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: } 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2024-03-24T12:50:50.561 INFO:teuthology.orchestra.run.smithi029.stdout:} 2024-03-24T12:50:51.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:50 smithi190 ceph-mon[33748]: 4.0 deep-scrub starts 2024-03-24T12:50:51.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:50 smithi190 ceph-mon[33748]: 4.0 deep-scrub ok 2024-03-24T12:50:51.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:50 smithi190 ceph-mon[33748]: 5.e deep-scrub starts 2024-03-24T12:50:51.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:50 smithi190 ceph-mon[33748]: 5.e deep-scrub ok 2024-03-24T12:50:51.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:50 smithi029 ceph-mon[41500]: 4.0 deep-scrub starts 2024-03-24T12:50:51.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:50 smithi029 ceph-mon[41500]: 4.0 deep-scrub ok 2024-03-24T12:50:51.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:50 smithi029 ceph-mon[41500]: 5.e deep-scrub starts 2024-03-24T12:50:51.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:50 smithi029 ceph-mon[41500]: 5.e deep-scrub ok 2024-03-24T12:50:51.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:50 smithi029 ceph-mon[29919]: 4.0 deep-scrub starts 2024-03-24T12:50:51.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:50 smithi029 ceph-mon[29919]: 4.0 deep-scrub ok 2024-03-24T12:50:51.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:50 smithi029 ceph-mon[29919]: 5.e deep-scrub starts 2024-03-24T12:50:51.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:50 smithi029 ceph-mon[29919]: 5.e deep-scrub ok 2024-03-24T12:50:51.626 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-24T12:50:51.882 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-24T12:50:51.886 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:52.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:51 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:50:52.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:51 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:50:52.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:51 smithi190 ceph-mon[33748]: 3.b scrub starts 2024-03-24T12:50:52.008 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:51 smithi190 ceph-mon[33748]: 3.b scrub ok 2024-03-24T12:50:52.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:51 smithi190 ceph-mon[33748]: pgmap v1092: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 719 B/s rd, 0 op/s 2024-03-24T12:50:52.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:51 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2102462986' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:52.009 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:51 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:52.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:50:52.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:50:52.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[29919]: 3.b scrub starts 2024-03-24T12:50:52.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[29919]: 3.b scrub ok 2024-03-24T12:50:52.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[29919]: pgmap v1092: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 719 B/s rd, 0 op/s 2024-03-24T12:50:52.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2102462986' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:52.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:52.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:50:52.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:50:52.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[41500]: 3.b scrub starts 2024-03-24T12:50:52.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[41500]: 3.b scrub ok 2024-03-24T12:50:52.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[41500]: pgmap v1092: 142 pgs: 1 peering, 141 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 719 B/s rd, 0 op/s 2024-03-24T12:50:52.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2102462986' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:52.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:51 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-24T12:50:53.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:52 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:50:53.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:52 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:50:53.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:52 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:50:53.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:52 smithi190 ceph-mon[33748]: osdmap e451: 8 total, 8 up, 8 in 2024-03-24T12:50:53.051 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:52 smithi190 ceph-mon[33748]: 6.d scrub starts 2024-03-24T12:50:53.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd reweight 2 1 2024-03-24T12:50:53.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:52 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:50:53.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:52 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:50:53.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:52 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:50:53.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:52 smithi029 ceph-mon[29919]: osdmap e451: 8 total, 8 up, 8 in 2024-03-24T12:50:53.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:52 smithi029 ceph-mon[29919]: 6.d scrub starts 2024-03-24T12:50:53.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:52 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:50:53.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:52 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:50:53.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:52 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-24T12:50:53.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:52 smithi029 ceph-mon[41500]: osdmap e451: 8 total, 8 up, 8 in 2024-03-24T12:50:53.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:52 smithi029 ceph-mon[41500]: 6.d scrub starts 2024-03-24T12:50:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:53.590 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:53.590 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:50:53.590 INFO:teuthology.orchestra.run.smithi190.stdout: "num_ops": 0 2024-03-24T12:50:53.590 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:54.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:50:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:50:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[29919]: 6.d scrub ok 2024-03-24T12:50:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[29919]: 1.9 scrub starts 2024-03-24T12:50:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[29919]: 1.9 scrub ok 2024-03-24T12:50:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[29919]: osdmap e452: 8 total, 8 up, 8 in 2024-03-24T12:50:54.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[29919]: pgmap v1095: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:50:54.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:50:54.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:50:54.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[41500]: 6.d scrub ok 2024-03-24T12:50:54.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[41500]: 1.9 scrub starts 2024-03-24T12:50:54.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[41500]: 1.9 scrub ok 2024-03-24T12:50:54.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[41500]: osdmap e452: 8 total, 8 up, 8 in 2024-03-24T12:50:54.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:53 smithi029 ceph-mon[41500]: pgmap v1095: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:50:54.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:53 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:50:54.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:53 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:50:54.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:53 smithi190 ceph-mon[33748]: 6.d scrub ok 2024-03-24T12:50:54.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:53 smithi190 ceph-mon[33748]: 1.9 scrub starts 2024-03-24T12:50:54.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:53 smithi190 ceph-mon[33748]: 1.9 scrub ok 2024-03-24T12:50:54.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:53 smithi190 ceph-mon[33748]: osdmap e452: 8 total, 8 up, 8 in 2024-03-24T12:50:54.258 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:53 smithi190 ceph-mon[33748]: pgmap v1095: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 2.0 KiB/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:50:54.286 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-24T12:50:54.536 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:54.909 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:54 smithi190 ceph-mon[33748]: 6.c scrub starts 2024-03-24T12:50:54.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:54 smithi190 ceph-mon[33748]: 6.c scrub ok 2024-03-24T12:50:54.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:54 smithi190 ceph-mon[33748]: 3.1b scrub starts 2024-03-24T12:50:54.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:54 smithi190 ceph-mon[33748]: 3.1b scrub ok 2024-03-24T12:50:54.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:54 smithi190 ceph-mon[33748]: 1.3 scrub starts 2024-03-24T12:50:54.910 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:54 smithi190 ceph-mon[33748]: 1.3 scrub ok 2024-03-24T12:50:55.076 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[41500]: 6.c scrub starts 2024-03-24T12:50:55.076 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[41500]: 6.c scrub ok 2024-03-24T12:50:55.077 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[41500]: 3.1b scrub starts 2024-03-24T12:50:55.077 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[41500]: 3.1b scrub ok 2024-03-24T12:50:55.077 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[41500]: 1.3 scrub starts 2024-03-24T12:50:55.077 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[41500]: 1.3 scrub ok 2024-03-24T12:50:55.077 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[29919]: 6.c scrub starts 2024-03-24T12:50:55.077 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[29919]: 6.c scrub ok 2024-03-24T12:50:55.077 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[29919]: 3.1b scrub starts 2024-03-24T12:50:55.077 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[29919]: 3.1b scrub ok 2024-03-24T12:50:55.077 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[29919]: 1.3 scrub starts 2024-03-24T12:50:55.077 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:54 smithi029 ceph-mon[29919]: 1.3 scrub ok 2024-03-24T12:50:56.034 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:55 smithi190 ceph-mon[33748]: 6.19 scrub starts 2024-03-24T12:50:56.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:55 smithi190 ceph-mon[33748]: 6.19 scrub ok 2024-03-24T12:50:56.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:55 smithi190 ceph-mon[33748]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 839 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:50:56.035 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:55 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/732617347' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:50:56.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:55 smithi029 ceph-mon[41500]: 6.19 scrub starts 2024-03-24T12:50:56.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:55 smithi029 ceph-mon[41500]: 6.19 scrub ok 2024-03-24T12:50:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:55 smithi029 ceph-mon[41500]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 839 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:50:56.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:55 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/732617347' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:50:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:55 smithi029 ceph-mon[29919]: 6.19 scrub starts 2024-03-24T12:50:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:55 smithi029 ceph-mon[29919]: 6.19 scrub ok 2024-03-24T12:50:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:55 smithi029 ceph-mon[29919]: pgmap v1096: 142 pgs: 142 active+clean; 582 KiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 839 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-03-24T12:50:56.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:55 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/732617347' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:50:56.294 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:56.294 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [], 2024-03-24T12:50:56.294 INFO:teuthology.orchestra.run.smithi190.stdout: "complaint_time": 30, 2024-03-24T12:50:56.294 INFO:teuthology.orchestra.run.smithi190.stdout: "num_blocked_ops": 0 2024-03-24T12:50:56.294 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.2 to 1 (10000) 2024-03-24T12:50:57.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:56 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:50:57.097 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:56 smithi190 ceph-mon[33748]: 5.19 deep-scrub starts 2024-03-24T12:50:57.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:56 smithi190 ceph-mon[33748]: 5.19 deep-scrub ok 2024-03-24T12:50:57.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:56 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:50:57.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:56 smithi190 ceph-mon[33748]: 3.5 scrub starts 2024-03-24T12:50:57.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:56 smithi190 ceph-mon[33748]: 3.5 scrub ok 2024-03-24T12:50:57.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:56 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/732617347' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:50:57.098 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:56 smithi190 ceph-mon[33748]: osdmap e453: 8 total, 8 up, 8 in 2024-03-24T12:50:57.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:50:57.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[29919]: 5.19 deep-scrub starts 2024-03-24T12:50:57.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[29919]: 5.19 deep-scrub ok 2024-03-24T12:50:57.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:50:57.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[29919]: 3.5 scrub starts 2024-03-24T12:50:57.122 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[29919]: 3.5 scrub ok 2024-03-24T12:50:57.123 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/732617347' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:50:57.123 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[29919]: osdmap e453: 8 total, 8 up, 8 in 2024-03-24T12:50:57.123 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:50:57.123 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[41500]: 5.19 deep-scrub starts 2024-03-24T12:50:57.123 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[41500]: 5.19 deep-scrub ok 2024-03-24T12:50:57.123 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:50:57.123 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[41500]: 3.5 scrub starts 2024-03-24T12:50:57.123 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[41500]: 3.5 scrub ok 2024-03-24T12:50:57.123 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/732617347' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:50:57.123 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:56 smithi029 ceph-mon[41500]: osdmap e453: 8 total, 8 up, 8 in 2024-03-24T12:50:57.124 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-24T12:50:57.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-24T12:50:57.364 INFO:teuthology.orchestra.run.smithi190.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T12:50:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:50:57.796 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:50:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:50:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:50:58.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:57 smithi190 ceph-mon[33748]: pgmap v1098: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:50:58.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:57 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:50:58.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:57 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:50:58.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:57 smithi029 ceph-mon[41500]: pgmap v1098: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:50:58.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:57 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:50:58.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:57 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:50:58.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:57 smithi029 ceph-mon[29919]: pgmap v1098: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-24T12:50:58.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:57 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:50:58.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:57 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:50:59.011 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:58 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:50:59.011 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:58 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:50:59.011 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:58 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:50:59.011 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:58 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:50:59.091 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:50:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:50:59.091 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:58 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:50:59.091 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:58 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:50:59.112 INFO:teuthology.orchestra.run.smithi190.stdout:{ 2024-03-24T12:50:59.112 INFO:teuthology.orchestra.run.smithi190.stdout: "size": 20, 2024-03-24T12:50:59.112 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 600, 2024-03-24T12:50:59.112 INFO:teuthology.orchestra.run.smithi190.stdout: "ops": [ 2024-03-24T12:50:59.112 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "osd_op(client.24344.0:4754 5.15 5:a93a5511:::notify.2:head [watch reconnect cookie 93828934255616 gen 7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e439)", 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:26.772152+0000", 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 32.343277700000002, 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.069981573, 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "client_info": { 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "client": "client.24344", 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "client_addr": "172.21.15.29:0/4003367372", 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "tid": 4754 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.113 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:26.772152+0000", 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:26.772152+0000", 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:26.772153+0000", 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1200000000000001e-06 2024-03-24T12:50:59.114 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:26.772157+0000", 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.4589999999999998e-06 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:26.772164+0000", 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7859999999999997e-06 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:26.772172+0000", 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5630000000000003e-06 2024-03-24T12:50:59.115 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:26.785099+0000", 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.012927162000000001 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for peered", 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:26.785109+0000", 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.5370000000000003e-06 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.805604+0000", 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.020494969 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.116 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for missing object", 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.805739+0000", 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000135635 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.822356+0000", 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.016617064000000001 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "waiting for rw locks", 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.822462+0000", 2024-03-24T12:50:59.117 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000105739 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.823544+0000", 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001082059 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.823617+0000", 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3206000000000003e-05 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.118 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.842134+0000", 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.018516273999999999 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.11 440/439 [PushOp(4:8eb99c53:gc::gc.22:head, version: 74'3, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:8eb99c53:gc::gc.22:head@74'3, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.795952+0000", 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.319477375999998, 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001613894, 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.119 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.795952+0000", 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.795952+0000", 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.795953+0000", 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9100000000000004e-07 2024-03-24T12:50:59.120 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.795967+0000", 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3645000000000001e-05 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.795980+0000", 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3740000000000001e-05 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.795999+0000", 2024-03-24T12:50:59.121 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8678999999999999e-05 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.797324+0000", 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013247599999999999 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.797337+0000", 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3641e-05 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.797566+0000", 2024-03-24T12:50:59.122 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00022883800000000001 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.10 440/439 [PushOp(4:0fd950d8:::obj_delete_at_hint.0000000015:head, version: 390'10, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:0fd950d8:::obj_delete_at_hint.0000000015:head@390'10, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.798576+0000", 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.316853219999999, 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0074335360000000001, 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.123 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.798576+0000", 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.798576+0000", 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.798577+0000", 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4299999999999998e-07 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.124 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.798588+0000", 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1569000000000001e-05 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.798606+0000", 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.7588000000000002e-05 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.798623+0000", 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6759e-05 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.125 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.805862+0000", 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0072392309999999996 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.805874+0000", 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1909000000000001e-05 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.806010+0000", 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00013603700000000001 2024-03-24T12:50:59.126 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.11 440/439 [PushOp(4:88f5d812:::obj_delete_at_hint.0000000036:head, version: 390'21, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:88f5d812:::obj_delete_at_hint.0000000036:head@390'21, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.801665+0000", 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.313764621000001, 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00072847199999999997, 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.127 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.801665+0000", 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.801665+0000", 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.801666+0000", 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.8299999999999997e-07 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.128 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.801678+0000", 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2723e-05 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802157+0000", 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047859299999999998 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802179+0000", 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1630000000000001e-05 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.129 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802242+0000", 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.3807999999999996e-05 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802250+0000", 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6240000000000002e-06 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802393+0000", 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000143511 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.130 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(6.1c 440/439 [PushOp(6:3f97cd91:users.uid::dashboard:head, version: 74'1, data_included: [0~370], data_size: 370, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(6:3f97cd91:users.uid::dashboard:head@74'1, size: 370, copy_subset: [(0, 370)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 370, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.802765+0000", 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.312664404, 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.003873526, 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802765+0000", 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.131 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802765+0000", 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802766+0000", 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.6e-07 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802775+0000", 2024-03-24T12:50:59.132 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6579999999999997e-06 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802788+0000", 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2578999999999999e-05 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.802808+0000", 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9950999999999999e-05 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.133 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.806474+0000", 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0036665539999999998 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.806486+0000", 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1858999999999999e-05 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.806639+0000", 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015266499999999999 2024-03-24T12:50:59.134 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.15 440/439 [PushOp(4:ab569f81:gc::gc.12:head, version: 88'10, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:ab569f81:gc::gc.12:head@88'10, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.808184+0000", 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.307245197, 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00067572300000000004, 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.135 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.808184+0000", 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.808184+0000", 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.808185+0000", 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.0400000000000002e-07 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.136 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.808197+0000", 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2044e-05 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.808341+0000", 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00014455699999999999 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.808365+0000", 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.3388999999999999e-05 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.137 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.808462+0000", 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.7065000000000007e-05 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.808470+0000", 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8539999999999997e-06 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.808860+0000", 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00039051 2024-03-24T12:50:59.138 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.11 440/439 [PushOp(4:8a7625ba:::obj_delete_at_hint.0000000103:head, version: 390'25, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:8a7625ba:::obj_delete_at_hint.0000000103:head@390'25, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.809297+0000", 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.306132539, 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00065897999999999998, 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.139 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.809297+0000", 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.809297+0000", 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.809298+0000", 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.9299999999999998e-07 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.809454+0000", 2024-03-24T12:50:59.140 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00015638100000000001 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.809470+0000", 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6053e-05 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.809527+0000", 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6808000000000002e-05 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.141 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.809776+0000", 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000248891 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.809788+0000", 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2432000000000001e-05 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.809956+0000", 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00016782200000000001 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.142 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "pg_update_log_missing(5.e epoch 440/439 rep_tid 78060 entries 440'14 (0'0) error 5:7759931f:::notify.3:head by client.24344.0:4757 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.810626+0000", 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.304803567, 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071568399999999996, 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.810626+0000", 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.143 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.810626+0000", 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.810627+0000", 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.6100000000000001e-07 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.810634+0000", 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.6930000000000007e-06 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.144 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.810647+0000", 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.309e-05 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.810660+0000", 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2337999999999999e-05 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.810727+0000", 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.7219999999999997e-05 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.145 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.811342+0000", 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00061478200000000002 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(5.d 440/439 [PushOp(5:b4812045:::notify.4:head, version: 74'8, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:b4812045:::notify.4:head@74'8, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.811002+0000", 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.304427253, 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00084503599999999996, 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.146 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.147 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.147 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.147 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.811002+0000", 2024-03-24T12:50:59.147 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.147 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.147 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.811002+0000", 2024-03-24T12:50:59.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.148 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.148 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.148 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.811003+0000", 2024-03-24T12:50:59.148 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5800000000000001e-07 2024-03-24T12:50:59.148 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.811018+0000", 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5841e-05 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.811032+0000", 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3400999999999999e-05 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.149 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.811046+0000", 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4266e-05 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.811119+0000", 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.3186e-05 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.811131+0000", 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1241e-05 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.150 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.811847+0000", 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00071684299999999999 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.13 440/439 [PushOp(4:c8333aab:gc::gc.11:head, version: 88'12, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:c8333aab:gc::gc.11:head@88'12, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.812107+0000", 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.303322789999999, 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.001223252, 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.151 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.812107+0000", 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.812107+0000", 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.812107+0000", 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.1600000000000002e-07 2024-03-24T12:50:59.152 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.812117+0000", 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0061000000000001e-05 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.812131+0000", 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4121000000000001e-05 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.812146+0000", 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4324e-05 2024-03-24T12:50:59.153 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.812233+0000", 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.7131000000000003e-05 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.812244+0000", 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1378e-05 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.813330+0000", 2024-03-24T12:50:59.154 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0010859210000000001 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "pg_update_log_missing(5.d epoch 440/439 rep_tid 122158 entries 440'13 (0'0) error 5:b4812045:::notify.4:head by client.24344.0:4756 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.816145+0000", 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.299285146999999, 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00070335700000000005, 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.155 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.816145+0000", 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.816145+0000", 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.816145+0000", 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0900000000000002e-07 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.156 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.816160+0000", 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5296999999999999e-05 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.816174+0000", 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3502e-05 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.816199+0000", 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5448000000000001e-05 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.157 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.816569+0000", 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00036963600000000001 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.816848+0000", 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000278965 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(5.3 440/439 [PushOp(5:c609908c:::notify.5:head, version: 318'14, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(5:c609908c:::notify.5:head@318'14, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.821469+0000", 2024-03-24T12:50:59.158 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.293960671000001, 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00064837300000000005, 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821469+0000", 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821469+0000", 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.159 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821469+0000", 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 4.4000000000000002e-07 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821480+0000", 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0866999999999999e-05 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821495+0000", 2024-03-24T12:50:59.160 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4224e-05 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821516+0000", 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.1481e-05 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.822007+0000", 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000491261 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.822017+0000", 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.8660000000000001e-06 2024-03-24T12:50:59.161 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.822117+0000", 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000100234 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.1d 440/439 [PushOp(4:beaa4be4:gc::gc.7:head, version: 74'4, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:beaa4be4:gc::gc.7:head@74'4, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.821661+0000", 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.293768707000002, 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00073103899999999995, 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.162 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821661+0000", 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821661+0000", 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.163 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821661+0000", 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.0100000000000005e-07 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821671+0000", 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.397e-06 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821683+0000", 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2284e-05 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.164 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821695+0000", 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1977999999999999e-05 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.822184+0000", 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00048932399999999997 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.822196+0000", 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1253000000000001e-05 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.165 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.822392+0000", 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00019630200000000001 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.7 440/439 [PushOp(4:e07a6548:gc::gc.15:head, version: 84'14, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:e07a6548:gc::gc.15:head@84'14, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.821714+0000", 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.293715727999999, 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.0013820830000000001, 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.166 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821714+0000", 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821714+0000", 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821714+0000", 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 3.8099999999999998e-07 2024-03-24T12:50:59.167 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821722+0000", 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.9759999999999995e-06 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821733+0000", 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0530000000000001e-05 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.821745+0000", 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2437e-05 2024-03-24T12:50:59.168 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.822619+0000", 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00087367599999999999 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.822630+0000", 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0903999999999999e-05 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.823096+0000", 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000466179 2024-03-24T12:50:59.169 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.15 440/439 [PushOp(4:ae95b544:::obj_delete_at_hint.0000000020:head, version: 390'21, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ae95b544:::obj_delete_at_hint.0000000020:head@390'21, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.824163+0000", 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.291266791000002, 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000698623, 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.170 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.824163+0000", 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.824163+0000", 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.824163+0000", 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.4999999999999999e-07 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.171 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.824179+0000", 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.5607000000000002e-05 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.824193+0000", 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.4409999999999999e-05 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.824209+0000", 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.579e-05 2024-03-24T12:50:59.172 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.824642+0000", 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00043356200000000002 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.824650+0000", 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.7749999999999993e-06 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.824861+0000", 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00021122900000000001 2024-03-24T12:50:59.173 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.15 440/439 [PushOp(4:ad7a3a02:::obj_delete_at_hint.0000000048:head, version: 390'23, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:ad7a3a02:::obj_delete_at_hint.0000000048:head@390'23, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.825551+0000", 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.289878327, 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000647082, 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.174 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.825551+0000", 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.825551+0000", 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.825552+0000", 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.5899999999999996e-07 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.175 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.825562+0000", 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.031e-05 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.825575+0000", 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2578999999999999e-05 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.825595+0000", 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.0530000000000002e-05 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.176 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.825867+0000", 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00027132099999999999 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.825879+0000", 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2354000000000001e-05 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.826198+0000", 2024-03-24T12:50:59.177 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00031932899999999998 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(4.8 440/439 [PushOp(4:1111b601:::obj_delete_at_hint.0000000069:head, version: 390'37, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:1111b601:::obj_delete_at_hint.0000000069:head@390'37, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:27.860294+0000", 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 31.255135614, 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00074661699999999998, 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.178 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.860294+0000", 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.860294+0000", 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.860295+0000", 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4099999999999999e-07 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.179 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.860303+0000", 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 8.6479999999999992e-06 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.860315+0000", 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1647999999999999e-05 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.860325+0000", 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0358e-05 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.180 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.860400+0000", 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.5024000000000005e-05 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.860412+0000", 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1939e-05 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:27.861041+0000", 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00062845900000000005 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.181 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(1.8 441/440 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 22'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@22'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:28.786048+0000", 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.329381827999999, 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00085964699999999995, 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.182 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786048+0000", 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786048+0000", 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786048+0000", 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.4499999999999997e-07 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.183 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786354+0000", 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00030547499999999998 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786373+0000", 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.8771999999999999e-05 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786386+0000", 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.3619e-05 2024-03-24T12:50:59.184 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786447+0000", 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 6.1131999999999997e-05 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786458+0000", 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.0597999999999999e-05 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786907+0000", 2024-03-24T12:50:59.185 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00044950600000000002 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "MOSDPGPush(1.4 441/440 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 83'128, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@83'128, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:28.786646+0000", 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 30.328784040999999, 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00084575999999999998, 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "started", 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.186 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786646+0000", 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786646+0000", 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.786646+0000", 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 5.5899999999999996e-07 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.187 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.787123+0000", 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00047656899999999998 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.787139+0000", 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.6045999999999998e-05 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.787150+0000", 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1669e-05 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.188 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.787229+0000", 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 7.8434000000000002e-05 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "started", 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.787254+0000", 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 2.5364999999999999e-05 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:28.787491+0000", 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.000237118 2024-03-24T12:50:59.189 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: "description": "pg_update_log_missing(5.3 epoch 445/443 rep_tid 3 entries 445'20 (0'0) error 5:c609908c:::notify.5:head by client.24344.0:4778 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: "initiated_at": "2024-03-24T12:50:33.499211+0000", 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: "age": 25.616218903, 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00082409900000000001, 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: "type_data": { 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: "flag_point": "reached pg", 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: "events": [ 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.190 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "initiated", 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:33.499211+0000", 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "header_read", 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:33.499211+0000", 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "throttled", 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:33.499212+0000", 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 9.6200000000000006e-07 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.191 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "all_read", 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:33.499224+0000", 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.2607e-05 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "dispatched", 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:33.499244+0000", 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.9636e-05 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "queued_for_pg", 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:33.499255+0000", 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 1.1481000000000001e-05 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "reached_pg", 2024-03-24T12:50:59.192 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:33.499516+0000", 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00026076199999999999 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: }, 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: { 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: "event": "done", 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: "time": "2024-03-24T12:50:33.500035+0000", 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: "duration": 0.00051865099999999996 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: } 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout: ] 2024-03-24T12:50:59.193 INFO:teuthology.orchestra.run.smithi190.stdout:} 2024-03-24T12:50:59.908 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-24T12:51:00.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: 6.1a scrub starts 2024-03-24T12:51:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: 6.1a scrub ok 2024-03-24T12:51:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:51:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: 4.10 scrub starts 2024-03-24T12:51:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: 4.10 scrub ok 2024-03-24T12:51:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:51:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:51:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2033902407' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:51:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:51:00.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: 6.1a scrub starts 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: 6.1a scrub ok 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: 4.10 scrub starts 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: 4.10 scrub ok 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2033902407' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:51:00.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:50:59 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:51:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: 6.1a scrub starts 2024-03-24T12:51:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: 6.1a scrub ok 2024-03-24T12:51:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: pgmap v1099: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T12:51:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: 4.10 scrub starts 2024-03-24T12:51:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: 4.10 scrub ok 2024-03-24T12:51:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:51:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:51:00.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2033902407' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-24T12:51:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:51:00.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:50:59 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:51:00.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd reweight 3 1 2024-03-24T12:51:00.880 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:01.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:00 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2033902407' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:51:01.165 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:00 smithi029 ceph-mon[29919]: osdmap e454: 8 total, 8 up, 8 in 2024-03-24T12:51:01.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:00 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2033902407' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:51:01.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:00 smithi029 ceph-mon[41500]: osdmap e454: 8 total, 8 up, 8 in 2024-03-24T12:51:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:00 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2033902407' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-24T12:51:01.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:00 smithi190 ceph-mon[33748]: osdmap e454: 8 total, 8 up, 8 in 2024-03-24T12:51:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:01 smithi029 ceph-mon[29919]: 6.1c scrub starts 2024-03-24T12:51:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:01 smithi029 ceph-mon[29919]: 6.1c scrub ok 2024-03-24T12:51:02.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:01 smithi029 ceph-mon[29919]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-03-24T12:51:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:01 smithi029 ceph-mon[41500]: 6.1c scrub starts 2024-03-24T12:51:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:01 smithi029 ceph-mon[41500]: 6.1c scrub ok 2024-03-24T12:51:02.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:01 smithi029 ceph-mon[41500]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-03-24T12:51:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:01 smithi190 ceph-mon[33748]: 6.1c scrub starts 2024-03-24T12:51:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:01 smithi190 ceph-mon[33748]: 6.1c scrub ok 2024-03-24T12:51:02.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:01 smithi190 ceph-mon[33748]: pgmap v1101: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 383 B/s rd, 0 op/s 2024-03-24T12:51:03.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:02 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:51:03.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:02 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:51:03.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:02 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:51:03.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:02 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:51:03.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:02 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:51:03.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:02 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:51:03.918 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.3 to 1 (10000) 2024-03-24T12:51:04.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:51:04.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:51:04.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[41500]: pgmap v1102: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:51:04.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4195240950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:51:04.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[41500]: 3.1c scrub starts 2024-03-24T12:51:04.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[41500]: 3.1c scrub ok 2024-03-24T12:51:04.308 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:04.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:51:04.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:51:04.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[29919]: pgmap v1102: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:51:04.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4195240950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:51:04.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[29919]: 3.1c scrub starts 2024-03-24T12:51:04.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[29919]: 3.1c scrub ok 2024-03-24T12:51:04.309 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:03 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:03 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:51:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:03 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:51:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:03 smithi190 ceph-mon[33748]: pgmap v1102: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:51:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:03 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4195240950' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:51:04.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:03 smithi190 ceph-mon[33748]: 3.1c scrub starts 2024-03-24T12:51:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:03 smithi190 ceph-mon[33748]: 3.1c scrub ok 2024-03-24T12:51:04.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:03 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:04.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-24T12:51:05.009 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:05.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:04 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4195240950' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:51:05.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:04 smithi029 ceph-mon[41500]: osdmap e455: 8 total, 8 up, 8 in 2024-03-24T12:51:05.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:04 smithi029 ceph-mon[41500]: 6.7 scrub starts 2024-03-24T12:51:05.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:04 smithi029 ceph-mon[41500]: 6.7 scrub ok 2024-03-24T12:51:05.135 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:04 smithi029 ceph-mon[41500]: osdmap e456: 8 total, 8 up, 8 in 2024-03-24T12:51:05.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:04 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4195240950' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:51:05.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:04 smithi029 ceph-mon[29919]: osdmap e455: 8 total, 8 up, 8 in 2024-03-24T12:51:05.135 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:04 smithi029 ceph-mon[29919]: 6.7 scrub starts 2024-03-24T12:51:05.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:04 smithi029 ceph-mon[29919]: 6.7 scrub ok 2024-03-24T12:51:05.136 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:04 smithi029 ceph-mon[29919]: osdmap e456: 8 total, 8 up, 8 in 2024-03-24T12:51:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:04 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4195240950' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:51:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:04 smithi190 ceph-mon[33748]: osdmap e455: 8 total, 8 up, 8 in 2024-03-24T12:51:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:04 smithi190 ceph-mon[33748]: 6.7 scrub starts 2024-03-24T12:51:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:04 smithi190 ceph-mon[33748]: 6.7 scrub ok 2024-03-24T12:51:05.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:04 smithi190 ceph-mon[33748]: osdmap e456: 8 total, 8 up, 8 in 2024-03-24T12:51:06.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:05 smithi029 ceph-mon[29919]: 3.11 scrub starts 2024-03-24T12:51:06.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:05 smithi029 ceph-mon[29919]: 3.11 scrub ok 2024-03-24T12:51:06.330 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:05 smithi029 ceph-mon[29919]: pgmap v1105: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:51:06.331 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:05 smithi029 ceph-mon[29919]: osdmap e457: 8 total, 8 up, 8 in 2024-03-24T12:51:06.331 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:05 smithi029 ceph-mon[41500]: 3.11 scrub starts 2024-03-24T12:51:06.331 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:05 smithi029 ceph-mon[41500]: 3.11 scrub ok 2024-03-24T12:51:06.331 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:05 smithi029 ceph-mon[41500]: pgmap v1105: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:51:06.331 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:05 smithi029 ceph-mon[41500]: osdmap e457: 8 total, 8 up, 8 in 2024-03-24T12:51:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:05 smithi190 ceph-mon[33748]: 3.11 scrub starts 2024-03-24T12:51:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:05 smithi190 ceph-mon[33748]: 3.11 scrub ok 2024-03-24T12:51:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:05 smithi190 ceph-mon[33748]: pgmap v1105: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:51:06.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:05 smithi190 ceph-mon[33748]: osdmap e457: 8 total, 8 up, 8 in 2024-03-24T12:51:07.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:51:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:51:07.928 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-24T12:51:08.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[41500]: osdmap e458: 8 total, 8 up, 8 in 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[41500]: pgmap v1108: 142 pgs: 2 remapped+peering, 140 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 335 B/s, 4 objects/s recovering 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[41500]: 6.6 deep-scrub starts 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[41500]: 6.6 deep-scrub ok 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[41500]: 3.1a scrub starts 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[41500]: 3.1a scrub ok 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1336650248' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[29919]: osdmap e458: 8 total, 8 up, 8 in 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[29919]: pgmap v1108: 142 pgs: 2 remapped+peering, 140 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 335 B/s, 4 objects/s recovering 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[29919]: 6.6 deep-scrub starts 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[29919]: 6.6 deep-scrub ok 2024-03-24T12:51:08.320 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[29919]: 3.1a scrub starts 2024-03-24T12:51:08.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[29919]: 3.1a scrub ok 2024-03-24T12:51:08.321 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:07 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1336650248' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:51:08.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:07 smithi190 ceph-mon[33748]: osdmap e458: 8 total, 8 up, 8 in 2024-03-24T12:51:08.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:07 smithi190 ceph-mon[33748]: pgmap v1108: 142 pgs: 2 remapped+peering, 140 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 335 B/s, 4 objects/s recovering 2024-03-24T12:51:08.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:07 smithi190 ceph-mon[33748]: 6.6 deep-scrub starts 2024-03-24T12:51:08.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:07 smithi190 ceph-mon[33748]: 6.6 deep-scrub ok 2024-03-24T12:51:08.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:07 smithi190 ceph-mon[33748]: 3.1a scrub starts 2024-03-24T12:51:08.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:07 smithi190 ceph-mon[33748]: 3.1a scrub ok 2024-03-24T12:51:08.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:07 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1336650248' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-24T12:51:08.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd reweight 4 1 2024-03-24T12:51:09.057 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:09.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:51:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:51:09.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:08 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1336650248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:51:09.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:08 smithi190 ceph-mon[33748]: osdmap e459: 8 total, 8 up, 8 in 2024-03-24T12:51:09.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:08 smithi190 ceph-mon[33748]: 3.1d scrub starts 2024-03-24T12:51:09.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:08 smithi190 ceph-mon[33748]: 3.1d scrub ok 2024-03-24T12:51:09.159 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:08 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1336650248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:51:09.159 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:08 smithi029 ceph-mon[41500]: osdmap e459: 8 total, 8 up, 8 in 2024-03-24T12:51:09.159 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:08 smithi029 ceph-mon[41500]: 3.1d scrub starts 2024-03-24T12:51:09.159 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:08 smithi029 ceph-mon[41500]: 3.1d scrub ok 2024-03-24T12:51:09.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:08 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1336650248' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-24T12:51:09.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:08 smithi029 ceph-mon[29919]: osdmap e459: 8 total, 8 up, 8 in 2024-03-24T12:51:09.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:08 smithi029 ceph-mon[29919]: 3.1d scrub starts 2024-03-24T12:51:09.160 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:08 smithi029 ceph-mon[29919]: 3.1d scrub ok 2024-03-24T12:51:10.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:10.256 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:09 smithi029 ceph-mon[41500]: pgmap v1110: 142 pgs: 1 remapped+peering, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s; 80 KiB/s, 13 objects/s recovering 2024-03-24T12:51:10.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:10.257 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:09 smithi029 ceph-mon[29919]: pgmap v1110: 142 pgs: 1 remapped+peering, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s; 80 KiB/s, 13 objects/s recovering 2024-03-24T12:51:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:10.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:09 smithi190 ceph-mon[33748]: pgmap v1110: 142 pgs: 1 remapped+peering, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1012 B/s rd, 0 op/s; 80 KiB/s, 13 objects/s recovering 2024-03-24T12:51:11.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[29919]: 3.2 scrub starts 2024-03-24T12:51:11.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[29919]: 3.2 scrub ok 2024-03-24T12:51:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[29919]: 5.12 scrub starts 2024-03-24T12:51:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[29919]: 5.12 scrub ok 2024-03-24T12:51:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[29919]: 3.16 scrub starts 2024-03-24T12:51:11.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[29919]: 3.16 scrub ok 2024-03-24T12:51:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[41500]: 3.2 scrub starts 2024-03-24T12:51:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[41500]: 3.2 scrub ok 2024-03-24T12:51:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[41500]: 5.12 scrub starts 2024-03-24T12:51:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[41500]: 5.12 scrub ok 2024-03-24T12:51:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[41500]: 3.16 scrub starts 2024-03-24T12:51:11.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:10 smithi029 ceph-mon[41500]: 3.16 scrub ok 2024-03-24T12:51:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:10 smithi190 ceph-mon[33748]: 3.2 scrub starts 2024-03-24T12:51:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:10 smithi190 ceph-mon[33748]: 3.2 scrub ok 2024-03-24T12:51:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:10 smithi190 ceph-mon[33748]: 5.12 scrub starts 2024-03-24T12:51:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:10 smithi190 ceph-mon[33748]: 5.12 scrub ok 2024-03-24T12:51:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:10 smithi190 ceph-mon[33748]: 3.16 scrub starts 2024-03-24T12:51:11.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:10 smithi190 ceph-mon[33748]: 3.16 scrub ok 2024-03-24T12:51:11.942 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.4 to 1 (10000) 2024-03-24T12:51:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:11 smithi190 ceph-mon[33748]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 96 KiB/s, 8 objects/s recovering 2024-03-24T12:51:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:11 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4124341912' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:51:12.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:11 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:51:12.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:11 smithi029 ceph-mon[41500]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 96 KiB/s, 8 objects/s recovering 2024-03-24T12:51:12.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:11 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4124341912' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:51:12.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:11 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:51:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:11 smithi029 ceph-mon[29919]: pgmap v1111: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 96 KiB/s, 8 objects/s recovering 2024-03-24T12:51:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:11 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4124341912' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:51:12.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:11 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:51:12.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-24T12:51:13.141 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:13.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:51:13.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:51:13.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[29919]: 3.1e scrub starts 2024-03-24T12:51:13.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[29919]: 3.1e scrub ok 2024-03-24T12:51:13.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:51:13.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[29919]: osdmap e460: 8 total, 8 up, 8 in 2024-03-24T12:51:13.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:51:13.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:51:13.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[41500]: 3.1e scrub starts 2024-03-24T12:51:13.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[41500]: 3.1e scrub ok 2024-03-24T12:51:13.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:51:13.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:12 smithi029 ceph-mon[41500]: osdmap e460: 8 total, 8 up, 8 in 2024-03-24T12:51:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:12 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:51:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:12 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:51:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:12 smithi190 ceph-mon[33748]: 3.1e scrub starts 2024-03-24T12:51:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:12 smithi190 ceph-mon[33748]: 3.1e scrub ok 2024-03-24T12:51:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:12 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:51:13.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:12 smithi190 ceph-mon[33748]: osdmap e460: 8 total, 8 up, 8 in 2024-03-24T12:51:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:13 smithi190 ceph-mon[33748]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 96 KiB/s, 5 objects/s recovering 2024-03-24T12:51:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:13 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:51:14.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:13 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:51:14.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:13 smithi029 ceph-mon[41500]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 96 KiB/s, 5 objects/s recovering 2024-03-24T12:51:14.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:13 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:51:14.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:13 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:51:14.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:13 smithi029 ceph-mon[29919]: pgmap v1113: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 96 KiB/s, 5 objects/s recovering 2024-03-24T12:51:14.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:13 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:51:14.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:13 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:51:15.043 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:14 smithi029 ceph-mon[41500]: 4.17 scrub starts 2024-03-24T12:51:15.044 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:14 smithi029 ceph-mon[41500]: 4.17 scrub ok 2024-03-24T12:51:15.044 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:14 smithi029 ceph-mon[29919]: 4.17 scrub starts 2024-03-24T12:51:15.044 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:14 smithi029 ceph-mon[29919]: 4.17 scrub ok 2024-03-24T12:51:15.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:14 smithi190 ceph-mon[33748]: 4.17 scrub starts 2024-03-24T12:51:15.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:14 smithi190 ceph-mon[33748]: 4.17 scrub ok 2024-03-24T12:51:15.949 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-24T12:51:16.318 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:15 smithi029 ceph-mon[41500]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 72 KiB/s, 4 objects/s recovering 2024-03-24T12:51:16.319 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:15 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/664694574' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:51:16.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:15 smithi029 ceph-mon[29919]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 72 KiB/s, 4 objects/s recovering 2024-03-24T12:51:16.319 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:15 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/664694574' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:51:16.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:15 smithi190 ceph-mon[33748]: pgmap v1114: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 72 KiB/s, 4 objects/s recovering 2024-03-24T12:51:16.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:15 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/664694574' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-24T12:51:16.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd reweight 6 1 2024-03-24T12:51:16.909 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:17.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:16 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/664694574' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:51:17.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:16 smithi029 ceph-mon[41500]: osdmap e461: 8 total, 8 up, 8 in 2024-03-24T12:51:17.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:16 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:51:17.266 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:16 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:51:17.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:16 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/664694574' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:51:17.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:16 smithi029 ceph-mon[29919]: osdmap e461: 8 total, 8 up, 8 in 2024-03-24T12:51:17.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:16 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:51:17.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:16 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:51:17.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:16 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/664694574' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-24T12:51:17.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:16 smithi190 ceph-mon[33748]: osdmap e461: 8 total, 8 up, 8 in 2024-03-24T12:51:17.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:16 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:51:17.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:16 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:51:17.884 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:51:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:51:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:51:18.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:17 smithi029 ceph-mon[41500]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-24T12:51:18.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:17 smithi029 ceph-mon[29919]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-24T12:51:18.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:17 smithi190 ceph-mon[33748]: pgmap v1116: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 32 KiB/s, 0 objects/s recovering 2024-03-24T12:51:19.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:51:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:51:19.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:18 smithi190 ceph-mon[33748]: 4.6 scrub starts 2024-03-24T12:51:19.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:18 smithi190 ceph-mon[33748]: 4.6 scrub ok 2024-03-24T12:51:19.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:18 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:51:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:18 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:51:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:18 smithi190 ceph-mon[33748]: 1.a scrub starts 2024-03-24T12:51:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:18 smithi190 ceph-mon[33748]: 1.a scrub ok 2024-03-24T12:51:19.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:18 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:19.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[29919]: 4.6 scrub starts 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[29919]: 4.6 scrub ok 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[29919]: 1.a scrub starts 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[29919]: 1.a scrub ok 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[41500]: 4.6 scrub starts 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[41500]: 4.6 scrub ok 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:51:19.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[41500]: 1.a scrub starts 2024-03-24T12:51:19.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[41500]: 1.a scrub ok 2024-03-24T12:51:19.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:18 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:19.971 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.6 to 1 (10000) 2024-03-24T12:51:20.328 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:20.328 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[41500]: pgmap v1117: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:20.328 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[41500]: 6.13 deep-scrub starts 2024-03-24T12:51:20.328 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2398337587' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:20.328 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[41500]: 6.13 deep-scrub ok 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[41500]: 5.7 scrub starts 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[41500]: 5.7 scrub ok 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[29919]: pgmap v1117: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[29919]: 6.13 deep-scrub starts 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2398337587' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[29919]: 6.13 deep-scrub ok 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[29919]: 5.7 scrub starts 2024-03-24T12:51:20.329 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:19 smithi029 ceph-mon[29919]: 5.7 scrub ok 2024-03-24T12:51:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:19 smithi190 ceph-mon[33748]: pgmap v1117: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:19 smithi190 ceph-mon[33748]: 6.13 deep-scrub starts 2024-03-24T12:51:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:19 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2398337587' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:19 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:19 smithi190 ceph-mon[33748]: 6.13 deep-scrub ok 2024-03-24T12:51:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:19 smithi190 ceph-mon[33748]: 5.7 scrub starts 2024-03-24T12:51:20.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:19 smithi190 ceph-mon[33748]: 5.7 scrub ok 2024-03-24T12:51:20.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-24T12:51:20.956 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:21.229 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[29919]: 1.2 scrub starts 2024-03-24T12:51:21.230 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[29919]: 1.2 scrub ok 2024-03-24T12:51:21.230 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[29919]: 6.12 scrub starts 2024-03-24T12:51:21.230 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[29919]: 6.12 scrub ok 2024-03-24T12:51:21.230 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:51:21.230 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[29919]: osdmap e462: 8 total, 8 up, 8 in 2024-03-24T12:51:21.230 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:51:21.230 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:51:21.230 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[41500]: 1.2 scrub starts 2024-03-24T12:51:21.230 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[41500]: 1.2 scrub ok 2024-03-24T12:51:21.230 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[41500]: 6.12 scrub starts 2024-03-24T12:51:21.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[41500]: 6.12 scrub ok 2024-03-24T12:51:21.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:51:21.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[41500]: osdmap e462: 8 total, 8 up, 8 in 2024-03-24T12:51:21.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:51:21.231 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:20 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:51:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:20 smithi190 ceph-mon[33748]: 1.2 scrub starts 2024-03-24T12:51:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:20 smithi190 ceph-mon[33748]: 1.2 scrub ok 2024-03-24T12:51:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:20 smithi190 ceph-mon[33748]: 6.12 scrub starts 2024-03-24T12:51:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:20 smithi190 ceph-mon[33748]: 6.12 scrub ok 2024-03-24T12:51:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:20 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:51:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:20 smithi190 ceph-mon[33748]: osdmap e462: 8 total, 8 up, 8 in 2024-03-24T12:51:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:20 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:51:21.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:20 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:51:22.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:21 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:51:22.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:21 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:51:22.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:21 smithi029 ceph-mon[29919]: pgmap v1119: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:51:22.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:21 smithi029 ceph-mon[29919]: 5.b scrub starts 2024-03-24T12:51:22.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:21 smithi029 ceph-mon[29919]: 5.b scrub ok 2024-03-24T12:51:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:21 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:51:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:21 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:51:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:21 smithi029 ceph-mon[41500]: pgmap v1119: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:51:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:21 smithi029 ceph-mon[41500]: 5.b scrub starts 2024-03-24T12:51:22.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:21 smithi029 ceph-mon[41500]: 5.b scrub ok 2024-03-24T12:51:22.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:21 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:51:22.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:21 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:51:22.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:21 smithi190 ceph-mon[33748]: pgmap v1119: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:51:22.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:21 smithi190 ceph-mon[33748]: 5.b scrub starts 2024-03-24T12:51:22.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:21 smithi190 ceph-mon[33748]: 5.b scrub ok 2024-03-24T12:51:23.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:22 smithi190 ceph-mon[33748]: 4.b scrub starts 2024-03-24T12:51:23.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:22 smithi190 ceph-mon[33748]: 4.b scrub ok 2024-03-24T12:51:23.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:22 smithi190 ceph-mon[33748]: 5.a scrub starts 2024-03-24T12:51:23.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:22 smithi190 ceph-mon[33748]: 5.a scrub ok 2024-03-24T12:51:23.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:22 smithi190 ceph-mon[33748]: 6.a scrub starts 2024-03-24T12:51:23.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:22 smithi190 ceph-mon[33748]: 6.a scrub ok 2024-03-24T12:51:23.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[29919]: 4.b scrub starts 2024-03-24T12:51:23.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[29919]: 4.b scrub ok 2024-03-24T12:51:23.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[29919]: 5.a scrub starts 2024-03-24T12:51:23.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[29919]: 5.a scrub ok 2024-03-24T12:51:23.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[29919]: 6.a scrub starts 2024-03-24T12:51:23.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[29919]: 6.a scrub ok 2024-03-24T12:51:23.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[41500]: 4.b scrub starts 2024-03-24T12:51:23.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[41500]: 4.b scrub ok 2024-03-24T12:51:23.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[41500]: 5.a scrub starts 2024-03-24T12:51:23.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[41500]: 5.a scrub ok 2024-03-24T12:51:23.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[41500]: 6.a scrub starts 2024-03-24T12:51:23.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:22 smithi029 ceph-mon[41500]: 6.a scrub ok 2024-03-24T12:51:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-24T12:51:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:23 smithi190 ceph-mon[33748]: 6.15 scrub starts 2024-03-24T12:51:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:23 smithi190 ceph-mon[33748]: 6.15 scrub ok 2024-03-24T12:51:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:23 smithi190 ceph-mon[33748]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:23 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2119511881' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:23 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:23 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:51:24.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:23 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:51:24.397 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[41500]: 6.15 scrub starts 2024-03-24T12:51:24.397 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[41500]: 6.15 scrub ok 2024-03-24T12:51:24.397 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[41500]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:24.397 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2119511881' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:24.397 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:24.398 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:51:24.398 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:51:24.398 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[29919]: 6.15 scrub starts 2024-03-24T12:51:24.398 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[29919]: 6.15 scrub ok 2024-03-24T12:51:24.398 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[29919]: pgmap v1120: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:24.398 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2119511881' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:24.398 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-24T12:51:24.398 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:51:24.398 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:23 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:51:24.759 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd reweight 7 1 2024-03-24T12:51:25.014 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:25.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:51:25.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:51:25.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:51:25.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[29919]: osdmap e463: 8 total, 8 up, 8 in 2024-03-24T12:51:25.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[29919]: 4.12 scrub starts 2024-03-24T12:51:25.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[29919]: 4.12 scrub ok 2024-03-24T12:51:25.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[29919]: 3.14 scrub starts 2024-03-24T12:51:25.269 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[29919]: 3.14 scrub ok 2024-03-24T12:51:25.269 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:51:25.270 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:51:25.270 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:51:25.270 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[41500]: osdmap e463: 8 total, 8 up, 8 in 2024-03-24T12:51:25.270 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[41500]: 4.12 scrub starts 2024-03-24T12:51:25.270 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[41500]: 4.12 scrub ok 2024-03-24T12:51:25.270 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[41500]: 3.14 scrub starts 2024-03-24T12:51:25.270 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:24 smithi029 ceph-mon[41500]: 3.14 scrub ok 2024-03-24T12:51:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:24 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:51:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:24 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:51:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:24 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-24T12:51:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:24 smithi190 ceph-mon[33748]: osdmap e463: 8 total, 8 up, 8 in 2024-03-24T12:51:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:24 smithi190 ceph-mon[33748]: 4.12 scrub starts 2024-03-24T12:51:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:24 smithi190 ceph-mon[33748]: 4.12 scrub ok 2024-03-24T12:51:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:24 smithi190 ceph-mon[33748]: 3.14 scrub starts 2024-03-24T12:51:25.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:24 smithi190 ceph-mon[33748]: 3.14 scrub ok 2024-03-24T12:51:26.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:51:26.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:51:26.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[41500]: pgmap v1122: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:26.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[41500]: osdmap e464: 8 total, 8 up, 8 in 2024-03-24T12:51:26.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[41500]: 3.10 scrub starts 2024-03-24T12:51:26.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[41500]: 3.10 scrub ok 2024-03-24T12:51:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:51:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:51:26.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[29919]: pgmap v1122: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:26.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[29919]: osdmap e464: 8 total, 8 up, 8 in 2024-03-24T12:51:26.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[29919]: 3.10 scrub starts 2024-03-24T12:51:26.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:25 smithi029 ceph-mon[29919]: 3.10 scrub ok 2024-03-24T12:51:26.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:25 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:51:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:25 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:51:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:25 smithi190 ceph-mon[33748]: pgmap v1122: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:25 smithi190 ceph-mon[33748]: osdmap e464: 8 total, 8 up, 8 in 2024-03-24T12:51:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:25 smithi190 ceph-mon[33748]: 3.10 scrub starts 2024-03-24T12:51:26.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:25 smithi190 ceph-mon[33748]: 3.10 scrub ok 2024-03-24T12:51:27.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:26 smithi029 ceph-mon[41500]: 3.d scrub starts 2024-03-24T12:51:27.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:26 smithi029 ceph-mon[41500]: 3.d scrub ok 2024-03-24T12:51:27.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:26 smithi029 ceph-mon[41500]: 4.d scrub starts 2024-03-24T12:51:27.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:26 smithi029 ceph-mon[41500]: 4.d scrub ok 2024-03-24T12:51:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:26 smithi029 ceph-mon[29919]: 3.d scrub starts 2024-03-24T12:51:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:26 smithi029 ceph-mon[29919]: 3.d scrub ok 2024-03-24T12:51:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:26 smithi029 ceph-mon[29919]: 4.d scrub starts 2024-03-24T12:51:27.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:26 smithi029 ceph-mon[29919]: 4.d scrub ok 2024-03-24T12:51:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:26 smithi190 ceph-mon[33748]: 3.d scrub starts 2024-03-24T12:51:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:26 smithi190 ceph-mon[33748]: 3.d scrub ok 2024-03-24T12:51:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:26 smithi190 ceph-mon[33748]: 4.d scrub starts 2024-03-24T12:51:27.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:26 smithi190 ceph-mon[33748]: 4.d scrub ok 2024-03-24T12:51:27.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:51:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:51:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:51:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.7 to 1 (10000) 2024-03-24T12:51:28.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:51:28.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:51:28.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: 1.0 scrub starts 2024-03-24T12:51:28.385 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: 1.0 scrub ok 2024-03-24T12:51:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-03-24T12:51:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: osdmap e465: 8 total, 8 up, 8 in 2024-03-24T12:51:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3706728689' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:51:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: 3.9 scrub starts 2024-03-24T12:51:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: 3.9 scrub ok 2024-03-24T12:51:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: 1.6 deep-scrub starts 2024-03-24T12:51:28.386 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-mon[33748]: 1.6 deep-scrub ok 2024-03-24T12:51:28.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:51:28.439 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:51:28.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: 1.0 scrub starts 2024-03-24T12:51:28.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: 1.0 scrub ok 2024-03-24T12:51:28.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-03-24T12:51:28.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: osdmap e465: 8 total, 8 up, 8 in 2024-03-24T12:51:28.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3706728689' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:51:28.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: 3.9 scrub starts 2024-03-24T12:51:28.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: 3.9 scrub ok 2024-03-24T12:51:28.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: 1.6 deep-scrub starts 2024-03-24T12:51:28.440 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[41500]: 1.6 deep-scrub ok 2024-03-24T12:51:28.440 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:51:28.441 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:51:28.441 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: 1.0 scrub starts 2024-03-24T12:51:28.441 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: 1.0 scrub ok 2024-03-24T12:51:28.441 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: pgmap v1124: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2024-03-24T12:51:28.441 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: osdmap e465: 8 total, 8 up, 8 in 2024-03-24T12:51:28.441 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3706728689' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:51:28.441 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: 3.9 scrub starts 2024-03-24T12:51:28.441 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: 3.9 scrub ok 2024-03-24T12:51:28.441 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: 1.6 deep-scrub starts 2024-03-24T12:51:28.441 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:28 smithi029 ceph-mon[29919]: 1.6 deep-scrub ok 2024-03-24T12:51:28.803 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-24T12:51:29.069 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:29.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:51:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:51:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:29 smithi190 ceph-mon[33748]: 2.0 scrub starts 2024-03-24T12:51:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:29 smithi190 ceph-mon[33748]: 2.0 scrub ok 2024-03-24T12:51:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:29 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3706728689' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:51:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:29 smithi190 ceph-mon[33748]: osdmap e466: 8 total, 8 up, 8 in 2024-03-24T12:51:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:29 smithi190 ceph-mon[33748]: 5.d scrub starts 2024-03-24T12:51:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:29 smithi190 ceph-mon[33748]: 5.d scrub ok 2024-03-24T12:51:29.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:29 smithi190 ceph-mon[33748]: osdmap e467: 8 total, 8 up, 8 in 2024-03-24T12:51:29.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[41500]: 2.0 scrub starts 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[41500]: 2.0 scrub ok 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3706728689' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[41500]: osdmap e466: 8 total, 8 up, 8 in 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[41500]: 5.d scrub starts 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[41500]: 5.d scrub ok 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[41500]: osdmap e467: 8 total, 8 up, 8 in 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[29919]: 2.0 scrub starts 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[29919]: 2.0 scrub ok 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3706728689' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[29919]: osdmap e466: 8 total, 8 up, 8 in 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[29919]: 5.d scrub starts 2024-03-24T12:51:29.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[29919]: 5.d scrub ok 2024-03-24T12:51:29.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:29 smithi029 ceph-mon[29919]: osdmap e467: 8 total, 8 up, 8 in 2024-03-24T12:51:30.333 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:30 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:30.333 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:30 smithi190 ceph-mon[33748]: pgmap v1127: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:51:30.333 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:30 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:51:30.333 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:30 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:51:30.333 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:30 smithi190 ceph-mon[33748]: osdmap e468: 8 total, 8 up, 8 in 2024-03-24T12:51:30.387 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:30 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:30.388 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:30 smithi029 ceph-mon[29919]: pgmap v1127: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:51:30.388 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:30 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:51:30.388 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:30 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:51:30.388 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:30 smithi029 ceph-mon[29919]: osdmap e468: 8 total, 8 up, 8 in 2024-03-24T12:51:30.388 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:30 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:30.388 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:30 smithi029 ceph-mon[41500]: pgmap v1127: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:51:30.388 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:30 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:51:30.388 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:30 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:51:30.388 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:30 smithi029 ceph-mon[41500]: osdmap e468: 8 total, 8 up, 8 in 2024-03-24T12:51:31.327 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-24T12:51:31.328 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:31 smithi029 ceph-mon[41500]: 4.11 scrub starts 2024-03-24T12:51:31.328 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:31 smithi029 ceph-mon[41500]: 4.11 scrub ok 2024-03-24T12:51:31.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:31 smithi029 ceph-mon[29919]: 4.11 scrub starts 2024-03-24T12:51:31.328 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:31 smithi029 ceph-mon[29919]: 4.11 scrub ok 2024-03-24T12:51:31.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:31 smithi190 ceph-mon[33748]: 4.11 scrub starts 2024-03-24T12:51:31.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:31 smithi190 ceph-mon[33748]: 4.11 scrub ok 2024-03-24T12:51:32.053 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd reweight 0 1 2024-03-24T12:51:32.304 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:32 smithi190 ceph-mon[33748]: 4.1e scrub starts 2024-03-24T12:51:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:32 smithi190 ceph-mon[33748]: 4.1e scrub ok 2024-03-24T12:51:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:32 smithi190 ceph-mon[33748]: pgmap v1130: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-24T12:51:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:32 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1352825364' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:51:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:32 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:51:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:32 smithi190 ceph-mon[33748]: 5.17 scrub starts 2024-03-24T12:51:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:32 smithi190 ceph-mon[33748]: 5.17 scrub ok 2024-03-24T12:51:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:32 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:51:32.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:32 smithi190 ceph-mon[33748]: osdmap e469: 8 total, 8 up, 8 in 2024-03-24T12:51:32.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[41500]: 4.1e scrub starts 2024-03-24T12:51:32.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[41500]: 4.1e scrub ok 2024-03-24T12:51:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[41500]: pgmap v1130: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-24T12:51:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1352825364' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:51:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:51:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[41500]: 5.17 scrub starts 2024-03-24T12:51:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[41500]: 5.17 scrub ok 2024-03-24T12:51:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:51:32.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[41500]: osdmap e469: 8 total, 8 up, 8 in 2024-03-24T12:51:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[29919]: 4.1e scrub starts 2024-03-24T12:51:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[29919]: 4.1e scrub ok 2024-03-24T12:51:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[29919]: pgmap v1130: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-03-24T12:51:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1352825364' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:51:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-24T12:51:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[29919]: 5.17 scrub starts 2024-03-24T12:51:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[29919]: 5.17 scrub ok 2024-03-24T12:51:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-24T12:51:32.420 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:32 smithi029 ceph-mon[29919]: osdmap e469: 8 total, 8 up, 8 in 2024-03-24T12:51:33.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:33 smithi029 ceph-mon[41500]: 1.1 scrub starts 2024-03-24T12:51:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:33 smithi029 ceph-mon[41500]: 1.1 scrub ok 2024-03-24T12:51:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:33 smithi029 ceph-mon[41500]: 4.1a scrub starts 2024-03-24T12:51:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:33 smithi029 ceph-mon[41500]: 4.1a scrub ok 2024-03-24T12:51:33.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:33 smithi029 ceph-mon[41500]: osdmap e470: 8 total, 8 up, 8 in 2024-03-24T12:51:33.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:33 smithi029 ceph-mon[29919]: 1.1 scrub starts 2024-03-24T12:51:33.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:33 smithi029 ceph-mon[29919]: 1.1 scrub ok 2024-03-24T12:51:33.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:33 smithi029 ceph-mon[29919]: 4.1a scrub starts 2024-03-24T12:51:33.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:33 smithi029 ceph-mon[29919]: 4.1a scrub ok 2024-03-24T12:51:33.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:33 smithi029 ceph-mon[29919]: osdmap e470: 8 total, 8 up, 8 in 2024-03-24T12:51:33.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:33 smithi190 ceph-mon[33748]: 1.1 scrub starts 2024-03-24T12:51:33.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:33 smithi190 ceph-mon[33748]: 1.1 scrub ok 2024-03-24T12:51:33.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:33 smithi190 ceph-mon[33748]: 4.1a scrub starts 2024-03-24T12:51:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:33 smithi190 ceph-mon[33748]: 4.1a scrub ok 2024-03-24T12:51:33.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:33 smithi190 ceph-mon[33748]: osdmap e470: 8 total, 8 up, 8 in 2024-03-24T12:51:34.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[29919]: pgmap v1133: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 3 objects/s recovering 2024-03-24T12:51:34.266 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[29919]: 4.19 scrub starts 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[29919]: 4.19 scrub ok 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[29919]: 1.8 scrub starts 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[29919]: 1.8 scrub ok 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[29919]: 4.a scrub starts 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[29919]: 4.a scrub ok 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[41500]: pgmap v1133: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 3 objects/s recovering 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[41500]: 4.19 scrub starts 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[41500]: 4.19 scrub ok 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[41500]: 1.8 scrub starts 2024-03-24T12:51:34.267 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[41500]: 1.8 scrub ok 2024-03-24T12:51:34.268 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[41500]: 4.a scrub starts 2024-03-24T12:51:34.268 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[41500]: 4.a scrub ok 2024-03-24T12:51:34.268 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:34 smithi190 ceph-mon[33748]: pgmap v1133: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 54 B/s, 3 objects/s recovering 2024-03-24T12:51:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:34 smithi190 ceph-mon[33748]: 4.19 scrub starts 2024-03-24T12:51:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:34 smithi190 ceph-mon[33748]: 4.19 scrub ok 2024-03-24T12:51:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:34 smithi190 ceph-mon[33748]: 1.8 scrub starts 2024-03-24T12:51:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:34 smithi190 ceph-mon[33748]: 1.8 scrub ok 2024-03-24T12:51:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:34 smithi190 ceph-mon[33748]: 4.a scrub starts 2024-03-24T12:51:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:34 smithi190 ceph-mon[33748]: 4.a scrub ok 2024-03-24T12:51:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.0 to 1 (10000) 2024-03-24T12:51:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:35 smithi190 ceph-mon[33748]: 6.18 scrub starts 2024-03-24T12:51:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:35 smithi190 ceph-mon[33748]: 6.18 scrub ok 2024-03-24T12:51:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:35 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2993995638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:35 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:35.414 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:35 smithi029 ceph-mon[41500]: 6.18 scrub starts 2024-03-24T12:51:35.414 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:35 smithi029 ceph-mon[41500]: 6.18 scrub ok 2024-03-24T12:51:35.414 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:35 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2993995638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:35.414 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:35 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:35 smithi029 ceph-mon[29919]: 6.18 scrub starts 2024-03-24T12:51:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:35 smithi029 ceph-mon[29919]: 6.18 scrub ok 2024-03-24T12:51:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:35 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2993995638' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:35.414 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:35 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:35.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-24T12:51:36.139 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:36.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:36 smithi029 ceph-mon[29919]: 3.4 scrub starts 2024-03-24T12:51:36.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:36 smithi029 ceph-mon[29919]: 3.4 scrub ok 2024-03-24T12:51:36.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:36 smithi029 ceph-mon[29919]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 857 B/s rd, 0 op/s; 73 B/s, 4 objects/s recovering 2024-03-24T12:51:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:36 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:51:36.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:36 smithi029 ceph-mon[29919]: osdmap e471: 8 total, 8 up, 8 in 2024-03-24T12:51:36.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:36 smithi029 ceph-mon[41500]: 3.4 scrub starts 2024-03-24T12:51:36.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:36 smithi029 ceph-mon[41500]: 3.4 scrub ok 2024-03-24T12:51:36.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:36 smithi029 ceph-mon[41500]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 857 B/s rd, 0 op/s; 73 B/s, 4 objects/s recovering 2024-03-24T12:51:36.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:36 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:51:36.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:36 smithi029 ceph-mon[41500]: osdmap e471: 8 total, 8 up, 8 in 2024-03-24T12:51:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:36 smithi190 ceph-mon[33748]: 3.4 scrub starts 2024-03-24T12:51:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:36 smithi190 ceph-mon[33748]: 3.4 scrub ok 2024-03-24T12:51:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:36 smithi190 ceph-mon[33748]: pgmap v1134: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 857 B/s rd, 0 op/s; 73 B/s, 4 objects/s recovering 2024-03-24T12:51:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:36 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:51:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:36 smithi190 ceph-mon[33748]: osdmap e471: 8 total, 8 up, 8 in 2024-03-24T12:51:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:37 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:51:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:37 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:51:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:37 smithi190 ceph-mon[33748]: 4.7 scrub starts 2024-03-24T12:51:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:37 smithi190 ceph-mon[33748]: 4.7 scrub ok 2024-03-24T12:51:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:37 smithi190 ceph-mon[33748]: 4.1 scrub starts 2024-03-24T12:51:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:37 smithi190 ceph-mon[33748]: 3.c scrub starts 2024-03-24T12:51:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:37 smithi190 ceph-mon[33748]: 3.c scrub ok 2024-03-24T12:51:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:37 smithi190 ceph-mon[33748]: 4.1 scrub ok 2024-03-24T12:51:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:51:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:51:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[41500]: 4.7 scrub starts 2024-03-24T12:51:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[41500]: 4.7 scrub ok 2024-03-24T12:51:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[41500]: 4.1 scrub starts 2024-03-24T12:51:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[41500]: 3.c scrub starts 2024-03-24T12:51:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[41500]: 3.c scrub ok 2024-03-24T12:51:37.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[41500]: 4.1 scrub ok 2024-03-24T12:51:37.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:51:37.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:51:37.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[29919]: 4.7 scrub starts 2024-03-24T12:51:37.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[29919]: 4.7 scrub ok 2024-03-24T12:51:37.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[29919]: 4.1 scrub starts 2024-03-24T12:51:37.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[29919]: 3.c scrub starts 2024-03-24T12:51:37.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[29919]: 3.c scrub ok 2024-03-24T12:51:37.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-mon[29919]: 4.1 scrub ok 2024-03-24T12:51:37.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:51:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:51:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:51:38.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:38 smithi029 ceph-mon[41500]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54 B/s, 3 objects/s recovering 2024-03-24T12:51:38.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:38 smithi029 ceph-mon[41500]: 4.15 scrub starts 2024-03-24T12:51:38.252 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:38 smithi029 ceph-mon[41500]: 4.15 scrub ok 2024-03-24T12:51:38.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:38 smithi029 ceph-mon[29919]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54 B/s, 3 objects/s recovering 2024-03-24T12:51:38.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:38 smithi029 ceph-mon[29919]: 4.15 scrub starts 2024-03-24T12:51:38.253 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:38 smithi029 ceph-mon[29919]: 4.15 scrub ok 2024-03-24T12:51:38.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:38 smithi190 ceph-mon[33748]: pgmap v1136: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 54 B/s, 3 objects/s recovering 2024-03-24T12:51:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:38 smithi190 ceph-mon[33748]: 4.15 scrub starts 2024-03-24T12:51:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:38 smithi190 ceph-mon[33748]: 4.15 scrub ok 2024-03-24T12:51:39.040 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-24T12:51:39.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:51:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:51:39.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:39 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:51:39.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:39 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2936309779' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:39.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:39 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:39.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:39 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:51:39.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:39 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:51:39.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:39 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:51:39.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:51:39.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2936309779' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:39.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:39.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:51:39.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:51:39.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:51:39.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:51:39.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2936309779' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:39.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-24T12:51:39.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:51:39.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:51:39.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:39 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:51:39.880 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd reweight 5 1 2024-03-24T12:51:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:40 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:51:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:40 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:51:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:40 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:40 smithi190 ceph-mon[33748]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 43 B/s, 2 objects/s recovering 2024-03-24T12:51:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:40 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:51:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:40 smithi190 ceph-mon[33748]: osdmap e472: 8 total, 8 up, 8 in 2024-03-24T12:51:40.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:40 smithi190 ceph-mon[33748]: 1.5 deep-scrub starts 2024-03-24T12:51:40.133 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:40 smithi190 ceph-mon[33748]: 1.5 deep-scrub ok 2024-03-24T12:51:40.136 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:40.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:51:40.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[29919]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 43 B/s, 2 objects/s recovering 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[29919]: osdmap e472: 8 total, 8 up, 8 in 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[29919]: 1.5 deep-scrub starts 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[29919]: 1.5 deep-scrub ok 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[41500]: pgmap v1137: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 43 B/s, 2 objects/s recovering 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-24T12:51:40.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[41500]: osdmap e472: 8 total, 8 up, 8 in 2024-03-24T12:51:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[41500]: 1.5 deep-scrub starts 2024-03-24T12:51:40.168 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:40 smithi029 ceph-mon[41500]: 1.5 deep-scrub ok 2024-03-24T12:51:41.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:41 smithi190 ceph-mon[33748]: 5.1 scrub starts 2024-03-24T12:51:41.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:41 smithi190 ceph-mon[33748]: 5.1 scrub ok 2024-03-24T12:51:41.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:41 smithi190 ceph-mon[33748]: 4.1f scrub starts 2024-03-24T12:51:41.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:41 smithi190 ceph-mon[33748]: 4.1f scrub ok 2024-03-24T12:51:41.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:41 smithi029 ceph-mon[41500]: 5.1 scrub starts 2024-03-24T12:51:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:41 smithi029 ceph-mon[41500]: 5.1 scrub ok 2024-03-24T12:51:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:41 smithi029 ceph-mon[41500]: 4.1f scrub starts 2024-03-24T12:51:41.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:41 smithi029 ceph-mon[41500]: 4.1f scrub ok 2024-03-24T12:51:41.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:41 smithi029 ceph-mon[29919]: 5.1 scrub starts 2024-03-24T12:51:41.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:41 smithi029 ceph-mon[29919]: 5.1 scrub ok 2024-03-24T12:51:41.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:41 smithi029 ceph-mon[29919]: 4.1f scrub starts 2024-03-24T12:51:41.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:41 smithi029 ceph-mon[29919]: 4.1f scrub ok 2024-03-24T12:51:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:42 smithi190 ceph-mon[33748]: 5.14 scrub starts 2024-03-24T12:51:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:42 smithi190 ceph-mon[33748]: 5.14 scrub ok 2024-03-24T12:51:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:42 smithi190 ceph-mon[33748]: 6.10 scrub starts 2024-03-24T12:51:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:42 smithi190 ceph-mon[33748]: 6.10 scrub ok 2024-03-24T12:51:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:42 smithi190 ceph-mon[33748]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-03-24T12:51:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:42 smithi190 ceph-mon[33748]: 6.1e deep-scrub starts 2024-03-24T12:51:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:42 smithi190 ceph-mon[33748]: 6.1e deep-scrub ok 2024-03-24T12:51:42.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[41500]: 5.14 scrub starts 2024-03-24T12:51:42.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[41500]: 5.14 scrub ok 2024-03-24T12:51:42.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[41500]: 6.10 scrub starts 2024-03-24T12:51:42.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[41500]: 6.10 scrub ok 2024-03-24T12:51:42.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[41500]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-03-24T12:51:42.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[41500]: 6.1e deep-scrub starts 2024-03-24T12:51:42.418 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[41500]: 6.1e deep-scrub ok 2024-03-24T12:51:42.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[29919]: 5.14 scrub starts 2024-03-24T12:51:42.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[29919]: 5.14 scrub ok 2024-03-24T12:51:42.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[29919]: 6.10 scrub starts 2024-03-24T12:51:42.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[29919]: 6.10 scrub ok 2024-03-24T12:51:42.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[29919]: pgmap v1139: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 27 B/s, 1 objects/s recovering 2024-03-24T12:51:42.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[29919]: 6.1e deep-scrub starts 2024-03-24T12:51:42.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:42 smithi029 ceph-mon[29919]: 6.1e deep-scrub ok 2024-03-24T12:51:43.048 INFO:teuthology.orchestra.run.smithi029.stderr:reweighted osd.5 to 1 (10000) 2024-03-24T12:51:43.158 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[41500]: 6.11 deep-scrub starts 2024-03-24T12:51:43.158 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[41500]: 6.11 deep-scrub ok 2024-03-24T12:51:43.158 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[41500]: 2.1 scrub starts 2024-03-24T12:51:43.158 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[41500]: 2.1 scrub ok 2024-03-24T12:51:43.158 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/684758111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:51:43.158 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:51:43.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[29919]: 6.11 deep-scrub starts 2024-03-24T12:51:43.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[29919]: 6.11 deep-scrub ok 2024-03-24T12:51:43.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[29919]: 2.1 scrub starts 2024-03-24T12:51:43.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[29919]: 2.1 scrub ok 2024-03-24T12:51:43.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/684758111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:51:43.159 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:43 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:51:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:43 smithi190 ceph-mon[33748]: 6.11 deep-scrub starts 2024-03-24T12:51:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:43 smithi190 ceph-mon[33748]: 6.11 deep-scrub ok 2024-03-24T12:51:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:43 smithi190 ceph-mon[33748]: 2.1 scrub starts 2024-03-24T12:51:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:43 smithi190 ceph-mon[33748]: 2.1 scrub ok 2024-03-24T12:51:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:43 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/684758111' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:51:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:43 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:51:44.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-24T12:51:44.264 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:44.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:44 smithi190 ceph-mon[33748]: pgmap v1140: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:44.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:44 smithi190 ceph-mon[33748]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:51:44.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:44 smithi190 ceph-mon[33748]: osdmap e473: 8 total, 8 up, 8 in 2024-03-24T12:51:44.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:44 smithi190 ceph-mon[33748]: 3.a scrub starts 2024-03-24T12:51:44.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:44 smithi190 ceph-mon[33748]: 3.a scrub ok 2024-03-24T12:51:44.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:44 smithi029 ceph-mon[29919]: pgmap v1140: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:44.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:44 smithi029 ceph-mon[29919]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:51:44.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:44 smithi029 ceph-mon[29919]: osdmap e473: 8 total, 8 up, 8 in 2024-03-24T12:51:44.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:44 smithi029 ceph-mon[29919]: 3.a scrub starts 2024-03-24T12:51:44.385 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:44 smithi029 ceph-mon[29919]: 3.a scrub ok 2024-03-24T12:51:44.386 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:44 smithi029 ceph-mon[41500]: pgmap v1140: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:44.386 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:44 smithi029 ceph-mon[41500]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:51:44.386 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:44 smithi029 ceph-mon[41500]: osdmap e473: 8 total, 8 up, 8 in 2024-03-24T12:51:44.386 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:44 smithi029 ceph-mon[41500]: 3.a scrub starts 2024-03-24T12:51:44.386 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:44 smithi029 ceph-mon[41500]: 3.a scrub ok 2024-03-24T12:51:45.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:45 smithi190 ceph-mon[33748]: 4.3 scrub starts 2024-03-24T12:51:45.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:45 smithi190 ceph-mon[33748]: 4.3 scrub ok 2024-03-24T12:51:45.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:45 smithi190 ceph-mon[33748]: 5.18 scrub starts 2024-03-24T12:51:45.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:45 smithi190 ceph-mon[33748]: 5.18 scrub ok 2024-03-24T12:51:45.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:45 smithi029 ceph-mon[41500]: 4.3 scrub starts 2024-03-24T12:51:45.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:45 smithi029 ceph-mon[41500]: 4.3 scrub ok 2024-03-24T12:51:45.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:45 smithi029 ceph-mon[41500]: 5.18 scrub starts 2024-03-24T12:51:45.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:45 smithi029 ceph-mon[41500]: 5.18 scrub ok 2024-03-24T12:51:45.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:45 smithi029 ceph-mon[29919]: 4.3 scrub starts 2024-03-24T12:51:45.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:45 smithi029 ceph-mon[29919]: 4.3 scrub ok 2024-03-24T12:51:45.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:45 smithi029 ceph-mon[29919]: 5.18 scrub starts 2024-03-24T12:51:45.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:45 smithi029 ceph-mon[29919]: 5.18 scrub ok 2024-03-24T12:51:46.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:46 smithi029 ceph-mon[41500]: pgmap v1142: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:46.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:46 smithi029 ceph-mon[41500]: 6.1b scrub starts 2024-03-24T12:51:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:46 smithi029 ceph-mon[29919]: pgmap v1142: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:46.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:46 smithi029 ceph-mon[29919]: 6.1b scrub starts 2024-03-24T12:51:46.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:46 smithi190 ceph-mon[33748]: pgmap v1142: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:46.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:46 smithi190 ceph-mon[33748]: 6.1b scrub starts 2024-03-24T12:51:47.059 INFO:teuthology.orchestra.run.smithi029.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-24T12:51:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:47 smithi190 ceph-mon[33748]: 6.1b scrub ok 2024-03-24T12:51:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:47 smithi190 ceph-mon[33748]: 5.1b scrub starts 2024-03-24T12:51:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:47 smithi190 ceph-mon[33748]: 5.1b scrub ok 2024-03-24T12:51:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:47 smithi190 ceph-mon[33748]: 4.c scrub starts 2024-03-24T12:51:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:47 smithi190 ceph-mon[33748]: 4.c scrub ok 2024-03-24T12:51:47.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:47 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1175375395' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:51:47.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[29919]: 6.1b scrub ok 2024-03-24T12:51:47.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[29919]: 5.1b scrub starts 2024-03-24T12:51:47.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[29919]: 5.1b scrub ok 2024-03-24T12:51:47.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[29919]: 4.c scrub starts 2024-03-24T12:51:47.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[29919]: 4.c scrub ok 2024-03-24T12:51:47.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1175375395' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:51:47.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[41500]: 6.1b scrub ok 2024-03-24T12:51:47.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[41500]: 5.1b scrub starts 2024-03-24T12:51:47.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[41500]: 5.1b scrub ok 2024-03-24T12:51:47.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[41500]: 4.c scrub starts 2024-03-24T12:51:47.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[41500]: 4.c scrub ok 2024-03-24T12:51:47.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1175375395' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-24T12:51:47.904 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:51:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:51:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:51:47.937 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-24T12:51:47.937 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-24T12:51:47.937 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-24T12:51:47.937 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-24T12:51:47.937 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T12:51:47.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T12:51:48.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[41500]: 4.1c scrub starts 2024-03-24T12:51:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[41500]: 4.1c scrub ok 2024-03-24T12:51:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[41500]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:51:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1175375395' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:51:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[41500]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T12:51:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[41500]: 5.16 scrub starts 2024-03-24T12:51:48.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[41500]: 5.16 scrub ok 2024-03-24T12:51:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[29919]: 4.1c scrub starts 2024-03-24T12:51:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[29919]: 4.1c scrub ok 2024-03-24T12:51:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[29919]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:51:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1175375395' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:51:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[29919]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T12:51:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[29919]: 5.16 scrub starts 2024-03-24T12:51:48.168 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:48 smithi029 ceph-mon[29919]: 5.16 scrub ok 2024-03-24T12:51:48.190 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:48 smithi190 ceph-mon[33748]: 4.1c scrub starts 2024-03-24T12:51:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:48 smithi190 ceph-mon[33748]: 4.1c scrub ok 2024-03-24T12:51:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:48 smithi190 ceph-mon[33748]: pgmap v1143: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:51:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:48 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1175375395' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-24T12:51:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:48 smithi190 ceph-mon[33748]: osdmap e474: 8 total, 8 up, 8 in 2024-03-24T12:51:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:48 smithi190 ceph-mon[33748]: 5.16 scrub starts 2024-03-24T12:51:48.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:48 smithi190 ceph-mon[33748]: 5.16 scrub ok 2024-03-24T12:51:49.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:49 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:51:49.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:49 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:51:49.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:49 smithi190 ceph-mon[33748]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T12:51:49.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:49 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:49.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:51:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:51:49.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:49 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:51:49.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:49 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:51:49.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:49 smithi029 ceph-mon[41500]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T12:51:49.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:49 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:49.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:49 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:51:49.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:49 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:51:49.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:49 smithi029 ceph-mon[29919]: osdmap e475: 8 total, 8 up, 8 in 2024-03-24T12:51:49.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:49 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:51:50.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:50 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:50.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:50 smithi190 ceph-mon[33748]: pgmap v1146: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:51:50.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:50 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:50.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:50 smithi029 ceph-mon[29919]: pgmap v1146: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:51:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:50 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:50.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:50 smithi029 ceph-mon[41500]: pgmap v1146: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:51:50.470 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:51:50.471 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":476,"fsid":"bc017092-e9d8-11ee-95cd-87774f69a715","created":"2024-03-24T12:20:18.712998+0000","modified":"2024-03-24T12:51:50.318356+0000","last_up_change":"2024-03-24T12:50:15.724537+0000","last_in_change":"2024-03-24T12:50:30.774242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T12:23:41.491597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"447","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"293","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-24T12:26:01.465080+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-24T12:26:02.100021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T12:26:03.473862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T12:26:05.573828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T12:26:07.645083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0aa1c49b-7887-4f74-b03c-08cc0adec90b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":417,"up_from":433,"up_thru":468,"down_at":418,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6801","nonce":1184012970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1184012970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1184012970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6803","nonce":1184012970}]},"public_addr":"172.21.15.29:6801/1184012970","cluster_addr":"172.21.15.29:6802/1184012970","heartbeat_back_addr":"172.21.15.29:6804/1184012970","heartbeat_front_addr":"172.21.15.29:6803/1184012970","state":["exists","up"]},{"osd":1,"uuid":"99d8ef8f-eb59-4882-ab51-b398e6b53bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":455,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":2923396572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2923396572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2923396572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6807","nonce":2923396572}]},"public_addr":"172.21.15.29:6805/2923396572","cluster_addr":"172.21.15.29:6806/2923396572","heartbeat_back_addr":"172.21.15.29:6808/2923396572","heartbeat_front_addr":"172.21.15.29:6807/2923396572","state":["exists","up"]},{"osd":2,"uuid":"c3f83505-25ee-4217-b60a-081c36092e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":455,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":126032915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":126032915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":126032915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":126032915}]},"public_addr":"172.21.15.29:6809/126032915","cluster_addr":"172.21.15.29:6810/126032915","heartbeat_back_addr":"172.21.15.29:6812/126032915","heartbeat_front_addr":"172.21.15.29:6811/126032915","state":["exists","up"]},{"osd":3,"uuid":"f5e479a8-459a-4315-a808-711a7b84bb96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":466,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1080171824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1080171824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1080171824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":1080171824}]},"public_addr":"172.21.15.29:6813/1080171824","cluster_addr":"172.21.15.29:6814/1080171824","heartbeat_back_addr":"172.21.15.29:6816/1080171824","heartbeat_front_addr":"172.21.15.29:6815/1080171824","state":["exists","up"]},{"osd":4,"uuid":"5e420304-2798-4d11-9688-d0d89482008c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":466,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3690258869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":3690258869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":3690258869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3690258869}]},"public_addr":"172.21.15.190:6800/3690258869","cluster_addr":"172.21.15.190:6801/3690258869","heartbeat_back_addr":"172.21.15.190:6803/3690258869","heartbeat_front_addr":"172.21.15.190:6802/3690258869","state":["exists","up"]},{"osd":5,"uuid":"1213431a-652c-4cbf-b32a-36335430c666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":42,"last_clean_end":203,"up_from":437,"up_thru":466,"down_at":204,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":477033924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":477033924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":477033924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":477033924}]},"public_addr":"172.21.15.190:6804/477033924","cluster_addr":"172.21.15.190:6805/477033924","heartbeat_back_addr":"172.21.15.190:6807/477033924","heartbeat_front_addr":"172.21.15.190:6806/477033924","state":["exists","up"]},{"osd":6,"uuid":"0712e511-0257-44f3-a82f-ee09ab957c0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":466,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1295064976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":1295064976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":1295064976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1295064976}]},"public_addr":"172.21.15.190:6808/1295064976","cluster_addr":"172.21.15.190:6809/1295064976","heartbeat_back_addr":"172.21.15.190:6811/1295064976","heartbeat_front_addr":"172.21.15.190:6810/1295064976","state":["exists","up"]},{"osd":7,"uuid":"7e36bde4-14d0-4e7e-9a82-2bb97859dbda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":469,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":110963949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":110963949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":110963949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":110963949}]},"public_addr":"172.21.15.190:6812/110963949","cluster_addr":"172.21.15.190:6813/110963949","heartbeat_back_addr":"172.21.15.190:6815/110963949","heartbeat_front_addr":"172.21.15.190:6814/110963949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:50:58.996148+0000","dead_epoch":417},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:50:59.711094+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:51:26.327352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:51:38.430690+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:51:24.736511+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:51:23.434331+0000","dead_epoch":203},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:50:33.448041+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:51:47.553609+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.f","osds":[3,4,7]}],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":4,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":7}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":7}]},{"pgid":"5.8","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2633022206":"2024-03-25T12:26:46.364982+0000","172.21.15.29:6800/2612078930":"2024-03-25T12:26:46.364982+0000","172.21.15.29:0/2365028933":"2024-03-25T12:26:46.364982+0000","172.21.15.29:0/4113609697":"2024-03-25T12:26:46.364982+0000","172.21.15.29:0/4130171883":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/3697529912":"2024-03-25T12:26:46.364982+0000","172.21.15.29:0/2929430696":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/327626505":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/3715448582":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/861146182":"2024-03-25T12:26:46.364982+0000","172.21.15.29:6800/2109454792":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/1631806715":"2024-03-25T12:20:46.948976+0000","172.21.15.29:6800/852212806":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/1823252077":"2024-03-25T12:21:15.151901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T12:51:51.196 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T12:51:51.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T12:51:51.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[41500]: 3.6 scrub starts 2024-03-24T12:51:51.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[41500]: 3.6 scrub ok 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[41500]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/556802173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[41500]: 5.6 deep-scrub starts 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[41500]: 5.6 deep-scrub ok 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[41500]: pgmap v1148: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[29919]: 3.6 scrub starts 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[29919]: 3.6 scrub ok 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[29919]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/556802173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[29919]: 5.6 deep-scrub starts 2024-03-24T12:51:51.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[29919]: 5.6 deep-scrub ok 2024-03-24T12:51:51.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:51 smithi029 ceph-mon[29919]: pgmap v1148: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:51:51.447 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:51 smithi190 ceph-mon[33748]: 3.6 scrub starts 2024-03-24T12:51:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:51 smithi190 ceph-mon[33748]: 3.6 scrub ok 2024-03-24T12:51:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:51 smithi190 ceph-mon[33748]: osdmap e476: 8 total, 8 up, 8 in 2024-03-24T12:51:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:51 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/556802173' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:51:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:51 smithi190 ceph-mon[33748]: 5.6 deep-scrub starts 2024-03-24T12:51:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:51 smithi190 ceph-mon[33748]: 5.6 deep-scrub ok 2024-03-24T12:51:51.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:51 smithi190 ceph-mon[33748]: pgmap v1148: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:51:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:52 smithi190 ceph-mon[33748]: 3.7 scrub starts 2024-03-24T12:51:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:52 smithi190 ceph-mon[33748]: 3.7 scrub ok 2024-03-24T12:51:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:52 smithi190 ceph-mon[33748]: 4.e scrub starts 2024-03-24T12:51:52.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:52 smithi190 ceph-mon[33748]: 4.e scrub ok 2024-03-24T12:51:52.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:52 smithi029 ceph-mon[29919]: 3.7 scrub starts 2024-03-24T12:51:52.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:52 smithi029 ceph-mon[29919]: 3.7 scrub ok 2024-03-24T12:51:52.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:52 smithi029 ceph-mon[29919]: 4.e scrub starts 2024-03-24T12:51:52.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:52 smithi029 ceph-mon[29919]: 4.e scrub ok 2024-03-24T12:51:52.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:52 smithi029 ceph-mon[41500]: 3.7 scrub starts 2024-03-24T12:51:52.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:52 smithi029 ceph-mon[41500]: 3.7 scrub ok 2024-03-24T12:51:52.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:52 smithi029 ceph-mon[41500]: 4.e scrub starts 2024-03-24T12:51:52.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:52 smithi029 ceph-mon[41500]: 4.e scrub ok 2024-03-24T12:51:53.438 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:51:53.439 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":476,"fsid":"bc017092-e9d8-11ee-95cd-87774f69a715","created":"2024-03-24T12:20:18.712998+0000","modified":"2024-03-24T12:51:50.318356+0000","last_up_change":"2024-03-24T12:50:15.724537+0000","last_in_change":"2024-03-24T12:50:30.774242+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T12:23:41.491597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"447","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"293","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.1800000667572021,"score_stable":2.1800000667572021,"optimal_score":1,"raw_score_acting":2.1800000667572021,"raw_score_stable":2.1800000667572021,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-24T12:26:01.465080+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-24T12:26:02.100021+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-24T12:26:03.473862+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-24T12:26:05.573828+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-24T12:26:07.645083+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0aa1c49b-7887-4f74-b03c-08cc0adec90b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":417,"up_from":433,"up_thru":468,"down_at":418,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6801","nonce":1184012970}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":1184012970}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":1184012970}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6803","nonce":1184012970}]},"public_addr":"172.21.15.29:6801/1184012970","cluster_addr":"172.21.15.29:6802/1184012970","heartbeat_back_addr":"172.21.15.29:6804/1184012970","heartbeat_front_addr":"172.21.15.29:6803/1184012970","state":["exists","up"]},{"osd":1,"uuid":"99d8ef8f-eb59-4882-ab51-b398e6b53bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":455,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6805","nonce":2923396572}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":2923396572}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":2923396572}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6807","nonce":2923396572}]},"public_addr":"172.21.15.29:6805/2923396572","cluster_addr":"172.21.15.29:6806/2923396572","heartbeat_back_addr":"172.21.15.29:6808/2923396572","heartbeat_front_addr":"172.21.15.29:6807/2923396572","state":["exists","up"]},{"osd":2,"uuid":"c3f83505-25ee-4217-b60a-081c36092e92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":455,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6809","nonce":126032915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":126032915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6812","nonce":126032915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":126032915}]},"public_addr":"172.21.15.29:6809/126032915","cluster_addr":"172.21.15.29:6810/126032915","heartbeat_back_addr":"172.21.15.29:6812/126032915","heartbeat_front_addr":"172.21.15.29:6811/126032915","state":["exists","up"]},{"osd":3,"uuid":"f5e479a8-459a-4315-a808-711a7b84bb96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":466,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6813","nonce":1080171824}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":1080171824}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":1080171824}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6815","nonce":1080171824}]},"public_addr":"172.21.15.29:6813/1080171824","cluster_addr":"172.21.15.29:6814/1080171824","heartbeat_back_addr":"172.21.15.29:6816/1080171824","heartbeat_front_addr":"172.21.15.29:6815/1080171824","state":["exists","up"]},{"osd":4,"uuid":"5e420304-2798-4d11-9688-d0d89482008c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":466,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6800","nonce":3690258869}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6801","nonce":3690258869}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6803","nonce":3690258869}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6802","nonce":3690258869}]},"public_addr":"172.21.15.190:6800/3690258869","cluster_addr":"172.21.15.190:6801/3690258869","heartbeat_back_addr":"172.21.15.190:6803/3690258869","heartbeat_front_addr":"172.21.15.190:6802/3690258869","state":["exists","up"]},{"osd":5,"uuid":"1213431a-652c-4cbf-b32a-36335430c666","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":42,"last_clean_end":203,"up_from":437,"up_thru":466,"down_at":204,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6804","nonce":477033924}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6805","nonce":477033924}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6807","nonce":477033924}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6806","nonce":477033924}]},"public_addr":"172.21.15.190:6804/477033924","cluster_addr":"172.21.15.190:6805/477033924","heartbeat_back_addr":"172.21.15.190:6807/477033924","heartbeat_front_addr":"172.21.15.190:6806/477033924","state":["exists","up"]},{"osd":6,"uuid":"0712e511-0257-44f3-a82f-ee09ab957c0c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":466,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6808","nonce":1295064976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6809","nonce":1295064976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6811","nonce":1295064976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6810","nonce":1295064976}]},"public_addr":"172.21.15.190:6808/1295064976","cluster_addr":"172.21.15.190:6809/1295064976","heartbeat_back_addr":"172.21.15.190:6811/1295064976","heartbeat_front_addr":"172.21.15.190:6810/1295064976","state":["exists","up"]},{"osd":7,"uuid":"7e36bde4-14d0-4e7e-9a82-2bb97859dbda","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":469,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6812","nonce":110963949}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6813","nonce":110963949}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6815","nonce":110963949}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.190:6814","nonce":110963949}]},"public_addr":"172.21.15.190:6812/110963949","cluster_addr":"172.21.15.190:6813/110963949","heartbeat_back_addr":"172.21.15.190:6815/110963949","heartbeat_front_addr":"172.21.15.190:6814/110963949","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:50:58.996148+0000","dead_epoch":417},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:50:59.711094+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:51:26.327352+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:51:38.430690+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:51:24.736511+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:51:23.434331+0000","dead_epoch":203},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:50:33.448041+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T12:51:47.553609+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"5.f","osds":[3,4,7]}],"pg_upmap_items":[{"pgid":"3.1b","mappings":[{"from":4,"to":2}]},{"pgid":"4.2","mappings":[{"from":1,"to":7}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.1b","mappings":[{"from":1,"to":7}]},{"pgid":"5.8","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2633022206":"2024-03-25T12:26:46.364982+0000","172.21.15.29:6800/2612078930":"2024-03-25T12:26:46.364982+0000","172.21.15.29:0/2365028933":"2024-03-25T12:26:46.364982+0000","172.21.15.29:0/4113609697":"2024-03-25T12:26:46.364982+0000","172.21.15.29:0/4130171883":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/3697529912":"2024-03-25T12:26:46.364982+0000","172.21.15.29:0/2929430696":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/327626505":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/3715448582":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/861146182":"2024-03-25T12:26:46.364982+0000","172.21.15.29:6800/2109454792":"2024-03-25T12:20:46.948976+0000","172.21.15.29:0/1631806715":"2024-03-25T12:20:46.948976+0000","172.21.15.29:6800/852212806":"2024-03-25T12:21:15.151901+0000","172.21.15.29:0/1823252077":"2024-03-25T12:21:15.151901+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T12:51:53.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[29919]: 5.11 scrub starts 2024-03-24T12:51:53.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[29919]: 5.11 scrub ok 2024-03-24T12:51:53.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[29919]: pgmap v1149: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T12:51:53.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[29919]: 5.10 scrub starts 2024-03-24T12:51:53.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[29919]: 5.10 scrub ok 2024-03-24T12:51:53.611 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[29919]: 3.8 scrub starts 2024-03-24T12:51:53.612 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[29919]: 3.8 scrub ok 2024-03-24T12:51:53.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[41500]: 5.11 scrub starts 2024-03-24T12:51:53.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[41500]: 5.11 scrub ok 2024-03-24T12:51:53.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[41500]: pgmap v1149: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T12:51:53.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[41500]: 5.10 scrub starts 2024-03-24T12:51:53.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[41500]: 5.10 scrub ok 2024-03-24T12:51:53.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[41500]: 3.8 scrub starts 2024-03-24T12:51:53.612 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:53 smithi029 ceph-mon[41500]: 3.8 scrub ok 2024-03-24T12:51:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:53 smithi190 ceph-mon[33748]: 5.11 scrub starts 2024-03-24T12:51:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:53 smithi190 ceph-mon[33748]: 5.11 scrub ok 2024-03-24T12:51:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:53 smithi190 ceph-mon[33748]: pgmap v1149: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-24T12:51:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:53 smithi190 ceph-mon[33748]: 5.10 scrub starts 2024-03-24T12:51:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:53 smithi190 ceph-mon[33748]: 5.10 scrub ok 2024-03-24T12:51:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:53 smithi190 ceph-mon[33748]: 3.8 scrub starts 2024-03-24T12:51:53.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:53 smithi190 ceph-mon[33748]: 3.8 scrub ok 2024-03-24T12:51:54.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T12:51:54.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T12:51:54.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T12:51:54.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T12:51:54.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T12:51:54.159 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T12:51:54.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T12:51:54.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T12:51:54.418 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:54 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/46998535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:51:54.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:54 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:51:54.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:54 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:51:54.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:54 smithi029 ceph-mon[29919]: 5.4 scrub starts 2024-03-24T12:51:54.419 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:54 smithi029 ceph-mon[29919]: 5.4 scrub ok 2024-03-24T12:51:54.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:54 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/46998535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:51:54.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:54 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:51:54.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:54 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:51:54.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:54 smithi029 ceph-mon[41500]: 5.4 scrub starts 2024-03-24T12:51:54.419 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:54 smithi029 ceph-mon[41500]: 5.4 scrub ok 2024-03-24T12:51:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:54 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/46998535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T12:51:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:54 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:51:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:54 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:51:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:54 smithi190 ceph-mon[33748]: 5.4 scrub starts 2024-03-24T12:51:54.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:54 smithi190 ceph-mon[33748]: 5.4 scrub ok 2024-03-24T12:51:54.716 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:54.718 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:54.718 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:54.719 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:54.729 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:54.729 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:54.729 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:54.745 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:51:55.501 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[29919]: 5.1d scrub starts 2024-03-24T12:51:55.501 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[29919]: 5.1d scrub ok 2024-03-24T12:51:55.501 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[29919]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T12:51:55.501 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[29919]: 5.1a deep-scrub starts 2024-03-24T12:51:55.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[29919]: 5.1a deep-scrub ok 2024-03-24T12:51:55.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[29919]: pgmap v1151: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-03-24T12:51:55.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[29919]: 5.f scrub starts 2024-03-24T12:51:55.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[29919]: 5.f scrub ok 2024-03-24T12:51:55.502 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[29919]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T12:51:55.502 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[41500]: 5.1d scrub starts 2024-03-24T12:51:55.502 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[41500]: 5.1d scrub ok 2024-03-24T12:51:55.502 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[41500]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T12:51:55.502 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[41500]: 5.1a deep-scrub starts 2024-03-24T12:51:55.502 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[41500]: 5.1a deep-scrub ok 2024-03-24T12:51:55.503 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[41500]: pgmap v1151: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-03-24T12:51:55.503 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[41500]: 5.f scrub starts 2024-03-24T12:51:55.503 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[41500]: 5.f scrub ok 2024-03-24T12:51:55.503 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:55 smithi029 ceph-mon[41500]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T12:51:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:55 smithi190 ceph-mon[33748]: 5.1d scrub starts 2024-03-24T12:51:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:55 smithi190 ceph-mon[33748]: 5.1d scrub ok 2024-03-24T12:51:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:55 smithi190 ceph-mon[33748]: osdmap e477: 8 total, 8 up, 8 in 2024-03-24T12:51:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:55 smithi190 ceph-mon[33748]: 5.1a deep-scrub starts 2024-03-24T12:51:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:55 smithi190 ceph-mon[33748]: 5.1a deep-scrub ok 2024-03-24T12:51:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:55 smithi190 ceph-mon[33748]: pgmap v1151: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-03-24T12:51:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:55 smithi190 ceph-mon[33748]: 5.f scrub starts 2024-03-24T12:51:55.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:55 smithi190 ceph-mon[33748]: 5.f scrub ok 2024-03-24T12:51:55.633 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:55 smithi190 ceph-mon[33748]: osdmap e478: 8 total, 8 up, 8 in 2024-03-24T12:51:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:56 smithi190 ceph-mon[33748]: 6.9 scrub starts 2024-03-24T12:51:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:56 smithi190 ceph-mon[33748]: 6.9 scrub ok 2024-03-24T12:51:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:56 smithi190 ceph-mon[33748]: 3.19 scrub starts 2024-03-24T12:51:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:56 smithi190 ceph-mon[33748]: 3.19 scrub ok 2024-03-24T12:51:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:56 smithi190 ceph-mon[33748]: 4.18 scrub starts 2024-03-24T12:51:56.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:56 smithi190 ceph-mon[33748]: 4.18 scrub ok 2024-03-24T12:51:56.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[41500]: 6.9 scrub starts 2024-03-24T12:51:56.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[41500]: 6.9 scrub ok 2024-03-24T12:51:56.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[41500]: 3.19 scrub starts 2024-03-24T12:51:56.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[41500]: 3.19 scrub ok 2024-03-24T12:51:56.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[41500]: 4.18 scrub starts 2024-03-24T12:51:56.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[41500]: 4.18 scrub ok 2024-03-24T12:51:56.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[29919]: 6.9 scrub starts 2024-03-24T12:51:56.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[29919]: 6.9 scrub ok 2024-03-24T12:51:56.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[29919]: 3.19 scrub starts 2024-03-24T12:51:56.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[29919]: 3.19 scrub ok 2024-03-24T12:51:56.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[29919]: 4.18 scrub starts 2024-03-24T12:51:56.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:56 smithi029 ceph-mon[29919]: 4.18 scrub ok 2024-03-24T12:51:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:57 smithi190 ceph-mon[33748]: 5.1e deep-scrub starts 2024-03-24T12:51:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:57 smithi190 ceph-mon[33748]: 5.1e deep-scrub ok 2024-03-24T12:51:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:57 smithi190 ceph-mon[33748]: 4.f scrub starts 2024-03-24T12:51:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:57 smithi190 ceph-mon[33748]: 4.f scrub ok 2024-03-24T12:51:57.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:57 smithi190 ceph-mon[33748]: pgmap v1153: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 769 B/s rd, 0 op/s 2024-03-24T12:51:57.666 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:51:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:51:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-mon[41500]: 5.1e deep-scrub starts 2024-03-24T12:51:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-mon[41500]: 5.1e deep-scrub ok 2024-03-24T12:51:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-mon[41500]: 4.f scrub starts 2024-03-24T12:51:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-mon[41500]: 4.f scrub ok 2024-03-24T12:51:57.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-mon[41500]: pgmap v1153: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 769 B/s rd, 0 op/s 2024-03-24T12:51:57.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-mon[29919]: 5.1e deep-scrub starts 2024-03-24T12:51:57.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-mon[29919]: 5.1e deep-scrub ok 2024-03-24T12:51:57.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-mon[29919]: 4.f scrub starts 2024-03-24T12:51:57.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-mon[29919]: 4.f scrub ok 2024-03-24T12:51:57.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:57 smithi029 ceph-mon[29919]: pgmap v1153: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 769 B/s rd, 0 op/s 2024-03-24T12:51:58.392 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:58 smithi029 ceph-mon[41500]: 4.8 scrub starts 2024-03-24T12:51:58.392 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:58 smithi029 ceph-mon[41500]: 4.8 scrub ok 2024-03-24T12:51:58.392 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:58 smithi029 ceph-mon[41500]: 6.f scrub starts 2024-03-24T12:51:58.393 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:58 smithi029 ceph-mon[41500]: 6.f scrub ok 2024-03-24T12:51:58.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:58 smithi029 ceph-mon[29919]: 4.8 scrub starts 2024-03-24T12:51:58.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:58 smithi029 ceph-mon[29919]: 4.8 scrub ok 2024-03-24T12:51:58.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:58 smithi029 ceph-mon[29919]: 6.f scrub starts 2024-03-24T12:51:58.393 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:58 smithi029 ceph-mon[29919]: 6.f scrub ok 2024-03-24T12:51:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:58 smithi190 ceph-mon[33748]: 4.8 scrub starts 2024-03-24T12:51:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:58 smithi190 ceph-mon[33748]: 4.8 scrub ok 2024-03-24T12:51:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:58 smithi190 ceph-mon[33748]: 6.f scrub starts 2024-03-24T12:51:58.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:58 smithi190 ceph-mon[33748]: 6.f scrub ok 2024-03-24T12:51:59.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:51:58 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:51:59.412 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:59 smithi190 ceph-mon[33748]: 3.1 scrub starts 2024-03-24T12:51:59.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:59 smithi190 ceph-mon[33748]: 3.1 scrub ok 2024-03-24T12:51:59.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:59 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:59.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:59 smithi190 ceph-mon[33748]: 4.2 deep-scrub starts 2024-03-24T12:51:59.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:59 smithi190 ceph-mon[33748]: 4.2 deep-scrub ok 2024-03-24T12:51:59.413 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:51:59 smithi190 ceph-mon[33748]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:59.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[41500]: 3.1 scrub starts 2024-03-24T12:51:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[41500]: 3.1 scrub ok 2024-03-24T12:51:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[41500]: 4.2 deep-scrub starts 2024-03-24T12:51:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[41500]: 4.2 deep-scrub ok 2024-03-24T12:51:59.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[41500]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:51:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[29919]: 3.1 scrub starts 2024-03-24T12:51:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[29919]: 3.1 scrub ok 2024-03-24T12:51:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:51:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[29919]: 4.2 deep-scrub starts 2024-03-24T12:51:59.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[29919]: 4.2 deep-scrub ok 2024-03-24T12:51:59.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:51:59 smithi029 ceph-mon[29919]: pgmap v1154: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:00.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:00 smithi029 ceph-mon[29919]: 3.1f scrub starts 2024-03-24T12:52:00.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:00 smithi029 ceph-mon[29919]: 3.1f scrub ok 2024-03-24T12:52:00.417 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:00 smithi029 ceph-mon[29919]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T12:52:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:00 smithi029 ceph-mon[41500]: 3.1f scrub starts 2024-03-24T12:52:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:00 smithi029 ceph-mon[41500]: 3.1f scrub ok 2024-03-24T12:52:00.417 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:00 smithi029 ceph-mon[41500]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T12:52:00.631 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:00 smithi190 ceph-mon[33748]: 3.1f scrub starts 2024-03-24T12:52:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:00 smithi190 ceph-mon[33748]: 3.1f scrub ok 2024-03-24T12:52:00.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:00 smithi190 ceph-mon[33748]: osdmap e479: 8 total, 8 up, 8 in 2024-03-24T12:52:01.470 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:01 smithi029 ceph-mon[29919]: 3.17 scrub starts 2024-03-24T12:52:01.470 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:01 smithi029 ceph-mon[29919]: 3.17 scrub ok 2024-03-24T12:52:01.470 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:01 smithi029 ceph-mon[29919]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-03-24T12:52:01.470 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:01 smithi029 ceph-mon[41500]: 3.17 scrub starts 2024-03-24T12:52:01.470 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:01 smithi029 ceph-mon[41500]: 3.17 scrub ok 2024-03-24T12:52:01.471 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:01 smithi029 ceph-mon[41500]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-03-24T12:52:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:01 smithi190 ceph-mon[33748]: 3.17 scrub starts 2024-03-24T12:52:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:01 smithi190 ceph-mon[33748]: 3.17 scrub ok 2024-03-24T12:52:01.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:01 smithi190 ceph-mon[33748]: pgmap v1156: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 772 B/s rd, 0 op/s 2024-03-24T12:52:02.624 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[41500]: 4.16 scrub starts 2024-03-24T12:52:02.624 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[41500]: 4.16 scrub ok 2024-03-24T12:52:02.625 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[41500]: 1.7 scrub starts 2024-03-24T12:52:02.625 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[41500]: 1.7 scrub ok 2024-03-24T12:52:02.625 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[41500]: 6.14 scrub starts 2024-03-24T12:52:02.625 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[41500]: 6.14 scrub ok 2024-03-24T12:52:02.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[29919]: 4.16 scrub starts 2024-03-24T12:52:02.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[29919]: 4.16 scrub ok 2024-03-24T12:52:02.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[29919]: 1.7 scrub starts 2024-03-24T12:52:02.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[29919]: 1.7 scrub ok 2024-03-24T12:52:02.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[29919]: 6.14 scrub starts 2024-03-24T12:52:02.625 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:02 smithi029 ceph-mon[29919]: 6.14 scrub ok 2024-03-24T12:52:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:02 smithi190 ceph-mon[33748]: 4.16 scrub starts 2024-03-24T12:52:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:02 smithi190 ceph-mon[33748]: 4.16 scrub ok 2024-03-24T12:52:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:02 smithi190 ceph-mon[33748]: 1.7 scrub starts 2024-03-24T12:52:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:02 smithi190 ceph-mon[33748]: 1.7 scrub ok 2024-03-24T12:52:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:02 smithi190 ceph-mon[33748]: 6.14 scrub starts 2024-03-24T12:52:02.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:02 smithi190 ceph-mon[33748]: 6.14 scrub ok 2024-03-24T12:52:03.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:03 smithi029 ceph-mon[29919]: 4.4 scrub starts 2024-03-24T12:52:03.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:03 smithi029 ceph-mon[29919]: 4.4 scrub ok 2024-03-24T12:52:03.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:03 smithi029 ceph-mon[29919]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:03.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:03 smithi029 ceph-mon[29919]: 5.1c scrub starts 2024-03-24T12:52:03.535 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:03 smithi029 ceph-mon[29919]: 5.1c scrub ok 2024-03-24T12:52:03.536 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:03 smithi029 ceph-mon[41500]: 4.4 scrub starts 2024-03-24T12:52:03.536 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:03 smithi029 ceph-mon[41500]: 4.4 scrub ok 2024-03-24T12:52:03.536 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:03 smithi029 ceph-mon[41500]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:03.536 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:03 smithi029 ceph-mon[41500]: 5.1c scrub starts 2024-03-24T12:52:03.536 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:03 smithi029 ceph-mon[41500]: 5.1c scrub ok 2024-03-24T12:52:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:03 smithi190 ceph-mon[33748]: 4.4 scrub starts 2024-03-24T12:52:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:03 smithi190 ceph-mon[33748]: 4.4 scrub ok 2024-03-24T12:52:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:03 smithi190 ceph-mon[33748]: pgmap v1157: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:03 smithi190 ceph-mon[33748]: 5.1c scrub starts 2024-03-24T12:52:03.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:03 smithi190 ceph-mon[33748]: 5.1c scrub ok 2024-03-24T12:52:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:04 smithi190 ceph-mon[33748]: 6.1 scrub starts 2024-03-24T12:52:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:04 smithi190 ceph-mon[33748]: 6.1 scrub ok 2024-03-24T12:52:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:04 smithi190 ceph-mon[33748]: 5.2 deep-scrub starts 2024-03-24T12:52:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:04 smithi190 ceph-mon[33748]: 5.2 deep-scrub ok 2024-03-24T12:52:04.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:04 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:52:04.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:04 smithi029 ceph-mon[41500]: 6.1 scrub starts 2024-03-24T12:52:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:04 smithi029 ceph-mon[41500]: 6.1 scrub ok 2024-03-24T12:52:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:04 smithi029 ceph-mon[41500]: 5.2 deep-scrub starts 2024-03-24T12:52:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:04 smithi029 ceph-mon[41500]: 5.2 deep-scrub ok 2024-03-24T12:52:04.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:04 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:52:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:04 smithi029 ceph-mon[29919]: 6.1 scrub starts 2024-03-24T12:52:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:04 smithi029 ceph-mon[29919]: 6.1 scrub ok 2024-03-24T12:52:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:04 smithi029 ceph-mon[29919]: 5.2 deep-scrub starts 2024-03-24T12:52:04.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:04 smithi029 ceph-mon[29919]: 5.2 deep-scrub ok 2024-03-24T12:52:04.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:04 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:52:05.626 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:05 smithi029 ceph-mon[41500]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:52:05.627 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:05 smithi029 ceph-mon[41500]: 5.0 scrub starts 2024-03-24T12:52:05.627 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:05 smithi029 ceph-mon[41500]: 5.0 scrub ok 2024-03-24T12:52:05.627 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:05 smithi029 ceph-mon[41500]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T12:52:05.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:05 smithi029 ceph-mon[29919]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:52:05.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:05 smithi029 ceph-mon[29919]: 5.0 scrub starts 2024-03-24T12:52:05.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:05 smithi029 ceph-mon[29919]: 5.0 scrub ok 2024-03-24T12:52:05.627 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:05 smithi029 ceph-mon[29919]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T12:52:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:05 smithi190 ceph-mon[33748]: pgmap v1158: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-24T12:52:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:05 smithi190 ceph-mon[33748]: 5.0 scrub starts 2024-03-24T12:52:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:05 smithi190 ceph-mon[33748]: 5.0 scrub ok 2024-03-24T12:52:05.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:05 smithi190 ceph-mon[33748]: osdmap e480: 8 total, 8 up, 8 in 2024-03-24T12:52:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:06 smithi190 ceph-mon[33748]: 6.4 scrub starts 2024-03-24T12:52:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:06 smithi190 ceph-mon[33748]: 6.4 scrub ok 2024-03-24T12:52:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:06 smithi190 ceph-mon[33748]: 5.15 scrub starts 2024-03-24T12:52:06.632 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:06 smithi190 ceph-mon[33748]: 5.15 scrub ok 2024-03-24T12:52:06.650 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:06 smithi029 ceph-mon[41500]: 6.4 scrub starts 2024-03-24T12:52:06.650 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:06 smithi029 ceph-mon[41500]: 6.4 scrub ok 2024-03-24T12:52:06.650 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:06 smithi029 ceph-mon[41500]: 5.15 scrub starts 2024-03-24T12:52:06.650 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:06 smithi029 ceph-mon[41500]: 5.15 scrub ok 2024-03-24T12:52:06.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:06 smithi029 ceph-mon[29919]: 6.4 scrub starts 2024-03-24T12:52:06.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:06 smithi029 ceph-mon[29919]: 6.4 scrub ok 2024-03-24T12:52:06.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:06 smithi029 ceph-mon[29919]: 5.15 scrub starts 2024-03-24T12:52:06.651 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:06 smithi029 ceph-mon[29919]: 5.15 scrub ok 2024-03-24T12:52:06.651 INFO:teuthology.orchestra.run.smithi029.stdout:1876900708375 2024-03-24T12:52:06.652 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T12:52:07.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:07 smithi029 ceph-mon[41500]: pgmap v1160: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:52:07.416 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:07 smithi029 ceph-mon[29919]: pgmap v1160: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:52:07.757 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:52:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:52:07.881 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:07 smithi190 ceph-mon[33748]: pgmap v1160: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:52:08.426 INFO:teuthology.orchestra.run.smithi029.stdout:55834575198 2024-03-24T12:52:08.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T12:52:08.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[29919]: 3.12 deep-scrub starts 2024-03-24T12:52:08.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[29919]: 3.12 deep-scrub ok 2024-03-24T12:52:08.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[29919]: 1.3 scrub starts 2024-03-24T12:52:08.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[29919]: 1.3 scrub ok 2024-03-24T12:52:08.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[29919]: 3.13 deep-scrub starts 2024-03-24T12:52:08.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[29919]: 3.13 deep-scrub ok 2024-03-24T12:52:08.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[29919]: 5.e scrub starts 2024-03-24T12:52:08.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[29919]: 5.e scrub ok 2024-03-24T12:52:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[41500]: 3.12 deep-scrub starts 2024-03-24T12:52:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[41500]: 3.12 deep-scrub ok 2024-03-24T12:52:08.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[41500]: 1.3 scrub starts 2024-03-24T12:52:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[41500]: 1.3 scrub ok 2024-03-24T12:52:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[41500]: 3.13 deep-scrub starts 2024-03-24T12:52:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[41500]: 3.13 deep-scrub ok 2024-03-24T12:52:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[41500]: 5.e scrub starts 2024-03-24T12:52:08.668 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:08 smithi029 ceph-mon[41500]: 5.e scrub ok 2024-03-24T12:52:08.862 INFO:teuthology.orchestra.run.smithi029.stdout:257698038076 2024-03-24T12:52:08.863 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T12:52:08.863 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:08 smithi190 ceph-mon[33748]: 3.12 deep-scrub starts 2024-03-24T12:52:08.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:08 smithi190 ceph-mon[33748]: 3.12 deep-scrub ok 2024-03-24T12:52:08.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:08 smithi190 ceph-mon[33748]: 1.3 scrub starts 2024-03-24T12:52:08.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:08 smithi190 ceph-mon[33748]: 1.3 scrub ok 2024-03-24T12:52:08.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:08 smithi190 ceph-mon[33748]: 3.13 deep-scrub starts 2024-03-24T12:52:08.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:08 smithi190 ceph-mon[33748]: 3.13 deep-scrub ok 2024-03-24T12:52:08.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:08 smithi190 ceph-mon[33748]: 5.e scrub starts 2024-03-24T12:52:08.864 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:08 smithi190 ceph-mon[33748]: 5.e scrub ok 2024-03-24T12:52:09.132 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:52:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:52:09.662 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:09 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:09.662 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:09 smithi029 ceph-mon[41500]: pgmap v1161: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:09.662 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:09 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:09.662 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:09 smithi029 ceph-mon[29919]: pgmap v1161: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:09.838 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:09 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:09.839 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:09 smithi190 ceph-mon[33748]: pgmap v1161: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:10.070 INFO:teuthology.orchestra.run.smithi029.stdout:115964117328 2024-03-24T12:52:10.070 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T12:52:10.556 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:10 smithi029 ceph-mon[41500]: 5.13 scrub starts 2024-03-24T12:52:10.556 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:10 smithi029 ceph-mon[41500]: 5.13 scrub ok 2024-03-24T12:52:10.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:10 smithi029 ceph-mon[29919]: 5.13 scrub starts 2024-03-24T12:52:10.557 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:10 smithi029 ceph-mon[29919]: 5.13 scrub ok 2024-03-24T12:52:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:10 smithi190 ceph-mon[33748]: 5.13 scrub starts 2024-03-24T12:52:10.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:10 smithi190 ceph-mon[33748]: 5.13 scrub ok 2024-03-24T12:52:11.649 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:11 smithi029 ceph-mon[41500]: 5.1f scrub starts 2024-03-24T12:52:11.649 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:11 smithi029 ceph-mon[41500]: 5.1f scrub ok 2024-03-24T12:52:11.650 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:11 smithi029 ceph-mon[41500]: pgmap v1162: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:11.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:11 smithi029 ceph-mon[29919]: 5.1f scrub starts 2024-03-24T12:52:11.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:11 smithi029 ceph-mon[29919]: 5.1f scrub ok 2024-03-24T12:52:11.650 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:11 smithi029 ceph-mon[29919]: pgmap v1162: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:11.881 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:11 smithi190 ceph-mon[33748]: 5.1f scrub starts 2024-03-24T12:52:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:11 smithi190 ceph-mon[33748]: 5.1f scrub ok 2024-03-24T12:52:11.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:11 smithi190 ceph-mon[33748]: pgmap v1162: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:12.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:12 smithi029 ceph-mon[41500]: 1.9 scrub starts 2024-03-24T12:52:12.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:12 smithi029 ceph-mon[41500]: 1.9 scrub ok 2024-03-24T12:52:12.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:12 smithi029 ceph-mon[41500]: 6.d scrub starts 2024-03-24T12:52:12.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:12 smithi029 ceph-mon[41500]: 6.d scrub ok 2024-03-24T12:52:12.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:12 smithi029 ceph-mon[29919]: 1.9 scrub starts 2024-03-24T12:52:12.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:12 smithi029 ceph-mon[29919]: 1.9 scrub ok 2024-03-24T12:52:12.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:12 smithi029 ceph-mon[29919]: 6.d scrub starts 2024-03-24T12:52:12.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:12 smithi029 ceph-mon[29919]: 6.d scrub ok 2024-03-24T12:52:12.881 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:12 smithi190 ceph-mon[33748]: 1.9 scrub starts 2024-03-24T12:52:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:12 smithi190 ceph-mon[33748]: 1.9 scrub ok 2024-03-24T12:52:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:12 smithi190 ceph-mon[33748]: 6.d scrub starts 2024-03-24T12:52:12.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:12 smithi190 ceph-mon[33748]: 6.d scrub ok 2024-03-24T12:52:12.968 INFO:teuthology.orchestra.run.smithi029.stdout:77309411673 2024-03-24T12:52:12.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T12:52:13.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[41500]: 6.5 scrub starts 2024-03-24T12:52:13.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[41500]: 6.5 scrub ok 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[41500]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[41500]: 4.14 deep-scrub starts 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[41500]: 4.14 deep-scrub ok 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[41500]: 4.1b scrub starts 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[41500]: 4.1b scrub ok 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[29919]: 6.5 scrub starts 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[29919]: 6.5 scrub ok 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[29919]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[29919]: 4.14 deep-scrub starts 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[29919]: 4.14 deep-scrub ok 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[29919]: 4.1b scrub starts 2024-03-24T12:52:13.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:13 smithi029 ceph-mon[29919]: 4.1b scrub ok 2024-03-24T12:52:13.783 INFO:teuthology.orchestra.run.smithi029.stdout:1859720839197 2024-03-24T12:52:13.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T12:52:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:13 smithi190 ceph-mon[33748]: 6.5 scrub starts 2024-03-24T12:52:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:13 smithi190 ceph-mon[33748]: 6.5 scrub ok 2024-03-24T12:52:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:13 smithi190 ceph-mon[33748]: pgmap v1163: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:13 smithi190 ceph-mon[33748]: 4.14 deep-scrub starts 2024-03-24T12:52:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:13 smithi190 ceph-mon[33748]: 4.14 deep-scrub ok 2024-03-24T12:52:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:13 smithi190 ceph-mon[33748]: 4.1b scrub starts 2024-03-24T12:52:13.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:13 smithi190 ceph-mon[33748]: 4.1b scrub ok 2024-03-24T12:52:14.227 INFO:teuthology.orchestra.run.smithi029.stdout:219043332420 2024-03-24T12:52:14.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T12:52:14.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:14 smithi029 ceph-mon[41500]: 5.9 scrub starts 2024-03-24T12:52:14.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:14 smithi029 ceph-mon[41500]: 5.9 scrub ok 2024-03-24T12:52:14.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:14 smithi029 ceph-mon[41500]: 6.7 scrub starts 2024-03-24T12:52:14.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:14 smithi029 ceph-mon[41500]: 6.7 scrub ok 2024-03-24T12:52:14.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:14 smithi029 ceph-mon[29919]: 5.9 scrub starts 2024-03-24T12:52:14.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:14 smithi029 ceph-mon[29919]: 5.9 scrub ok 2024-03-24T12:52:14.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:14 smithi029 ceph-mon[29919]: 6.7 scrub starts 2024-03-24T12:52:14.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:14 smithi029 ceph-mon[29919]: 6.7 scrub ok 2024-03-24T12:52:14.797 INFO:teuthology.orchestra.run.smithi029.stdout:141733921102 2024-03-24T12:52:14.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T12:52:14.879 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:14 smithi190 ceph-mon[33748]: 5.9 scrub starts 2024-03-24T12:52:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:14 smithi190 ceph-mon[33748]: 5.9 scrub ok 2024-03-24T12:52:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:14 smithi190 ceph-mon[33748]: 6.7 scrub starts 2024-03-24T12:52:14.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:14 smithi190 ceph-mon[33748]: 6.7 scrub ok 2024-03-24T12:52:14.898 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:14.912 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:15.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:15 smithi029 ceph-mon[29919]: 3.1b scrub starts 2024-03-24T12:52:15.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:15 smithi029 ceph-mon[29919]: 3.1b scrub ok 2024-03-24T12:52:15.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:15 smithi029 ceph-mon[29919]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:15 smithi029 ceph-mon[41500]: 3.1b scrub starts 2024-03-24T12:52:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:15 smithi029 ceph-mon[41500]: 3.1b scrub ok 2024-03-24T12:52:15.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:15 smithi029 ceph-mon[41500]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:15 smithi190 ceph-mon[33748]: 3.1b scrub starts 2024-03-24T12:52:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:15 smithi190 ceph-mon[33748]: 3.1b scrub ok 2024-03-24T12:52:15.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:15 smithi190 ceph-mon[33748]: pgmap v1164: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:16.385 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:16.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:52:16.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[41500]: 6.1d scrub starts 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[41500]: 6.1d scrub ok 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[41500]: 6.16 scrub starts 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[41500]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[41500]: 6.16 scrub ok 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[41500]: 3.1d scrub starts 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[41500]: 3.1d scrub ok 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[41500]: 6.b scrub starts 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:52:16.667 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[29919]: 6.1d scrub starts 2024-03-24T12:52:16.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[29919]: 6.1d scrub ok 2024-03-24T12:52:16.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[29919]: 6.16 scrub starts 2024-03-24T12:52:16.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[29919]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T12:52:16.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[29919]: 6.16 scrub ok 2024-03-24T12:52:16.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[29919]: 3.1d scrub starts 2024-03-24T12:52:16.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[29919]: 3.1d scrub ok 2024-03-24T12:52:16.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:16 smithi029 ceph-mon[29919]: 6.b scrub starts 2024-03-24T12:52:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:16 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:52:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:16 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:52:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:16 smithi190 ceph-mon[33748]: 6.1d scrub starts 2024-03-24T12:52:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:16 smithi190 ceph-mon[33748]: 6.1d scrub ok 2024-03-24T12:52:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:16 smithi190 ceph-mon[33748]: 6.16 scrub starts 2024-03-24T12:52:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:16 smithi190 ceph-mon[33748]: osdmap e481: 8 total, 8 up, 8 in 2024-03-24T12:52:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:16 smithi190 ceph-mon[33748]: 6.16 scrub ok 2024-03-24T12:52:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:16 smithi190 ceph-mon[33748]: 3.1d scrub starts 2024-03-24T12:52:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:16 smithi190 ceph-mon[33748]: 3.1d scrub ok 2024-03-24T12:52:16.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:16 smithi190 ceph-mon[33748]: 6.b scrub starts 2024-03-24T12:52:17.603 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:17 smithi029 ceph-mon[41500]: 6.b scrub ok 2024-03-24T12:52:17.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:17 smithi029 ceph-mon[41500]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:17.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:17 smithi029 ceph-mon[41500]: 1.4 scrub starts 2024-03-24T12:52:17.604 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:17 smithi029 ceph-mon[41500]: 1.4 scrub ok 2024-03-24T12:52:17.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:17 smithi029 ceph-mon[29919]: 6.b scrub ok 2024-03-24T12:52:17.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:17 smithi029 ceph-mon[29919]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:17.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:17 smithi029 ceph-mon[29919]: 1.4 scrub starts 2024-03-24T12:52:17.604 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:17 smithi029 ceph-mon[29919]: 1.4 scrub ok 2024-03-24T12:52:17.881 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:17 smithi190 ceph-mon[33748]: 6.b scrub ok 2024-03-24T12:52:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:17 smithi190 ceph-mon[33748]: pgmap v1166: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:17 smithi190 ceph-mon[33748]: 1.4 scrub starts 2024-03-24T12:52:17.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:17 smithi190 ceph-mon[33748]: 1.4 scrub ok 2024-03-24T12:52:17.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:17 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:52:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:52:18.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:18 smithi190 ceph-mon[33748]: 2.2 scrub starts 2024-03-24T12:52:18.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:18 smithi190 ceph-mon[33748]: 2.2 scrub ok 2024-03-24T12:52:18.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:18 smithi190 ceph-mon[33748]: 3.3 scrub starts 2024-03-24T12:52:18.874 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:18 smithi190 ceph-mon[33748]: 3.3 scrub ok 2024-03-24T12:52:18.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:18 smithi029 ceph-mon[41500]: 2.2 scrub starts 2024-03-24T12:52:18.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:18 smithi029 ceph-mon[41500]: 2.2 scrub ok 2024-03-24T12:52:18.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:18 smithi029 ceph-mon[41500]: 3.3 scrub starts 2024-03-24T12:52:18.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:18 smithi029 ceph-mon[41500]: 3.3 scrub ok 2024-03-24T12:52:18.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:18 smithi029 ceph-mon[29919]: 2.2 scrub starts 2024-03-24T12:52:18.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:18 smithi029 ceph-mon[29919]: 2.2 scrub ok 2024-03-24T12:52:18.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:18 smithi029 ceph-mon[29919]: 3.3 scrub starts 2024-03-24T12:52:18.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:18 smithi029 ceph-mon[29919]: 3.3 scrub ok 2024-03-24T12:52:19.131 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:52:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:52:19.641 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[41500]: 5.c scrub starts 2024-03-24T12:52:19.641 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[41500]: 5.c scrub ok 2024-03-24T12:52:19.641 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:52:19.641 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:19.642 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[41500]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:19.642 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[41500]: 6.1f scrub starts 2024-03-24T12:52:19.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[29919]: 5.c scrub starts 2024-03-24T12:52:19.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[29919]: 5.c scrub ok 2024-03-24T12:52:19.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:52:19.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:19.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[29919]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:19.642 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:19 smithi029 ceph-mon[29919]: 6.1f scrub starts 2024-03-24T12:52:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:19 smithi190 ceph-mon[33748]: 5.c scrub starts 2024-03-24T12:52:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:19 smithi190 ceph-mon[33748]: 5.c scrub ok 2024-03-24T12:52:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:19 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:52:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:19 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:19 smithi190 ceph-mon[33748]: pgmap v1167: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:19.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:19 smithi190 ceph-mon[33748]: 6.1f scrub starts 2024-03-24T12:52:20.014 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:20.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:20 smithi029 ceph-mon[29919]: 6.1f scrub ok 2024-03-24T12:52:20.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:20 smithi029 ceph-mon[29919]: 4.1d scrub starts 2024-03-24T12:52:20.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:20 smithi029 ceph-mon[29919]: 4.1d scrub ok 2024-03-24T12:52:20.666 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:20 smithi029 ceph-mon[29919]: 6.17 deep-scrub starts 2024-03-24T12:52:20.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:20 smithi029 ceph-mon[41500]: 6.1f scrub ok 2024-03-24T12:52:20.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:20 smithi029 ceph-mon[41500]: 4.1d scrub starts 2024-03-24T12:52:20.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:20 smithi029 ceph-mon[41500]: 4.1d scrub ok 2024-03-24T12:52:20.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:20 smithi029 ceph-mon[41500]: 6.17 deep-scrub starts 2024-03-24T12:52:20.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:20 smithi190 ceph-mon[33748]: 6.1f scrub ok 2024-03-24T12:52:20.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:20 smithi190 ceph-mon[33748]: 4.1d scrub starts 2024-03-24T12:52:20.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:20 smithi190 ceph-mon[33748]: 4.1d scrub ok 2024-03-24T12:52:20.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:20 smithi190 ceph-mon[33748]: 6.17 deep-scrub starts 2024-03-24T12:52:21.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:21 smithi190 ceph-mon[33748]: 6.17 deep-scrub ok 2024-03-24T12:52:21.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:21 smithi190 ceph-mon[33748]: 5.3 scrub starts 2024-03-24T12:52:21.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:21 smithi190 ceph-mon[33748]: 5.3 scrub ok 2024-03-24T12:52:21.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:21 smithi190 ceph-mon[33748]: 3.15 scrub starts 2024-03-24T12:52:21.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:21 smithi190 ceph-mon[33748]: 3.15 scrub ok 2024-03-24T12:52:21.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:21 smithi190 ceph-mon[33748]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:21.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:21 smithi190 ceph-mon[33748]: 3.18 deep-scrub starts 2024-03-24T12:52:21.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:21 smithi190 ceph-mon[33748]: 3.18 deep-scrub ok 2024-03-24T12:52:21.916 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[41500]: 6.17 deep-scrub ok 2024-03-24T12:52:21.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[41500]: 5.3 scrub starts 2024-03-24T12:52:21.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[41500]: 5.3 scrub ok 2024-03-24T12:52:21.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[41500]: 3.15 scrub starts 2024-03-24T12:52:21.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[41500]: 3.15 scrub ok 2024-03-24T12:52:21.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[41500]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:21.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[41500]: 3.18 deep-scrub starts 2024-03-24T12:52:21.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[41500]: 3.18 deep-scrub ok 2024-03-24T12:52:21.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[29919]: 6.17 deep-scrub ok 2024-03-24T12:52:21.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[29919]: 5.3 scrub starts 2024-03-24T12:52:21.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[29919]: 5.3 scrub ok 2024-03-24T12:52:21.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[29919]: 3.15 scrub starts 2024-03-24T12:52:21.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[29919]: 3.15 scrub ok 2024-03-24T12:52:21.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[29919]: pgmap v1168: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:21.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[29919]: 3.18 deep-scrub starts 2024-03-24T12:52:21.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:21 smithi029 ceph-mon[29919]: 3.18 deep-scrub ok 2024-03-24T12:52:22.498 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:22 smithi029 ceph-mon[41500]: 5.19 scrub starts 2024-03-24T12:52:22.499 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:22 smithi029 ceph-mon[41500]: 5.19 scrub ok 2024-03-24T12:52:22.499 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:22 smithi029 ceph-mon[41500]: 6.e scrub starts 2024-03-24T12:52:22.499 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:22 smithi029 ceph-mon[41500]: 6.e scrub ok 2024-03-24T12:52:22.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:22 smithi029 ceph-mon[29919]: 5.19 scrub starts 2024-03-24T12:52:22.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:22 smithi029 ceph-mon[29919]: 5.19 scrub ok 2024-03-24T12:52:22.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:22 smithi029 ceph-mon[29919]: 6.e scrub starts 2024-03-24T12:52:22.499 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:22 smithi029 ceph-mon[29919]: 6.e scrub ok 2024-03-24T12:52:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:22 smithi190 ceph-mon[33748]: 5.19 scrub starts 2024-03-24T12:52:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:22 smithi190 ceph-mon[33748]: 5.19 scrub ok 2024-03-24T12:52:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:22 smithi190 ceph-mon[33748]: 6.e scrub starts 2024-03-24T12:52:22.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:22 smithi190 ceph-mon[33748]: 6.e scrub ok 2024-03-24T12:52:23.028 INFO:teuthology.orchestra.run.smithi029.stdout:1859720839199 2024-03-24T12:52:23.160 INFO:teuthology.orchestra.run.smithi029.stdout:257698038080 2024-03-24T12:52:23.713 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[41500]: 6.3 scrub starts 2024-03-24T12:52:23.713 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[41500]: 6.3 scrub ok 2024-03-24T12:52:23.713 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[41500]: pgmap v1169: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:23.713 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/686828586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T12:52:23.713 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[41500]: 4.0 scrub starts 2024-03-24T12:52:23.713 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[41500]: 4.0 scrub ok 2024-03-24T12:52:23.713 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/472605929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T12:52:23.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[29919]: 6.3 scrub starts 2024-03-24T12:52:23.713 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[29919]: 6.3 scrub ok 2024-03-24T12:52:23.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[29919]: pgmap v1169: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:23.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/686828586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T12:52:23.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[29919]: 4.0 scrub starts 2024-03-24T12:52:23.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[29919]: 4.0 scrub ok 2024-03-24T12:52:23.714 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:23 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/472605929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T12:52:23.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:23 smithi190 ceph-mon[33748]: 6.3 scrub starts 2024-03-24T12:52:23.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:23 smithi190 ceph-mon[33748]: 6.3 scrub ok 2024-03-24T12:52:23.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:23 smithi190 ceph-mon[33748]: pgmap v1169: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:23.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:23 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/686828586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T12:52:23.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:23 smithi190 ceph-mon[33748]: 4.0 scrub starts 2024-03-24T12:52:23.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:23 smithi190 ceph-mon[33748]: 4.0 scrub ok 2024-03-24T12:52:23.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:23 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/472605929' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T12:52:24.621 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698038076 got 257698038080 for osd.7 2024-03-24T12:52:24.621 DEBUG:teuthology.parallel:result is None 2024-03-24T12:52:24.722 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[41500]: 4.5 scrub starts 2024-03-24T12:52:24.722 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[41500]: 4.5 scrub ok 2024-03-24T12:52:24.722 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[41500]: 6.19 scrub starts 2024-03-24T12:52:24.722 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[41500]: 6.19 scrub ok 2024-03-24T12:52:24.722 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[41500]: 3.1a deep-scrub starts 2024-03-24T12:52:24.722 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[41500]: 3.1a deep-scrub ok 2024-03-24T12:52:24.722 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[29919]: 4.5 scrub starts 2024-03-24T12:52:24.723 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[29919]: 4.5 scrub ok 2024-03-24T12:52:24.723 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[29919]: 6.19 scrub starts 2024-03-24T12:52:24.723 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[29919]: 6.19 scrub ok 2024-03-24T12:52:24.723 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[29919]: 3.1a deep-scrub starts 2024-03-24T12:52:24.723 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:24 smithi029 ceph-mon[29919]: 3.1a deep-scrub ok 2024-03-24T12:52:24.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:24 smithi190 ceph-mon[33748]: 4.5 scrub starts 2024-03-24T12:52:24.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:24 smithi190 ceph-mon[33748]: 4.5 scrub ok 2024-03-24T12:52:24.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:24 smithi190 ceph-mon[33748]: 6.19 scrub starts 2024-03-24T12:52:24.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:24 smithi190 ceph-mon[33748]: 6.19 scrub ok 2024-03-24T12:52:24.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:24 smithi190 ceph-mon[33748]: 3.1a deep-scrub starts 2024-03-24T12:52:24.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:24 smithi190 ceph-mon[33748]: 3.1a deep-scrub ok 2024-03-24T12:52:25.076 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:25.273 INFO:teuthology.orchestra.run.smithi029.stdout:77309411675 2024-03-24T12:52:25.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:25 smithi190 ceph-mon[33748]: 3.11 scrub starts 2024-03-24T12:52:25.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:25 smithi190 ceph-mon[33748]: 3.11 scrub ok 2024-03-24T12:52:25.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:25 smithi190 ceph-mon[33748]: pgmap v1170: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:25.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:25 smithi190 ceph-mon[33748]: 3.16 scrub starts 2024-03-24T12:52:25.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:25 smithi190 ceph-mon[33748]: 3.16 scrub ok 2024-03-24T12:52:25.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:25 smithi190 ceph-mon[33748]: 6.6 deep-scrub starts 2024-03-24T12:52:25.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:25 smithi190 ceph-mon[33748]: 6.6 deep-scrub ok 2024-03-24T12:52:25.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:25 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2034358886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T12:52:25.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[29919]: 3.11 scrub starts 2024-03-24T12:52:25.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[29919]: 3.11 scrub ok 2024-03-24T12:52:25.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[29919]: pgmap v1170: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[29919]: 3.16 scrub starts 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[29919]: 3.16 scrub ok 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[29919]: 6.6 deep-scrub starts 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[29919]: 6.6 deep-scrub ok 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2034358886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[41500]: 3.11 scrub starts 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[41500]: 3.11 scrub ok 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[41500]: pgmap v1170: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[41500]: 3.16 scrub starts 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[41500]: 3.16 scrub ok 2024-03-24T12:52:25.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[41500]: 6.6 deep-scrub starts 2024-03-24T12:52:25.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[41500]: 6.6 deep-scrub ok 2024-03-24T12:52:25.918 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:25 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2034358886' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T12:52:26.740 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:52:26.741 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:52:26.741 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[41500]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T12:52:26.741 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:52:26.741 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:52:26.741 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[41500]: 3.e deep-scrub starts 2024-03-24T12:52:26.741 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[41500]: 3.e deep-scrub ok 2024-03-24T12:52:26.741 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[41500]: 6.1a scrub starts 2024-03-24T12:52:26.741 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:52:26.742 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:52:26.742 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[29919]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T12:52:26.742 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:52:26.742 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:52:26.742 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[29919]: 3.e deep-scrub starts 2024-03-24T12:52:26.742 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[29919]: 3.e deep-scrub ok 2024-03-24T12:52:26.742 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:26 smithi029 ceph-mon[29919]: 6.1a scrub starts 2024-03-24T12:52:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:26 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:52:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:26 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:52:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:26 smithi190 ceph-mon[33748]: osdmap e482: 8 total, 8 up, 8 in 2024-03-24T12:52:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:26 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:52:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:26 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:52:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:26 smithi190 ceph-mon[33748]: 3.e deep-scrub starts 2024-03-24T12:52:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:26 smithi190 ceph-mon[33748]: 3.e deep-scrub ok 2024-03-24T12:52:26.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:26 smithi190 ceph-mon[33748]: 6.1a scrub starts 2024-03-24T12:52:26.955 INFO:teuthology.orchestra.run.smithi029.stdout:219043332423 2024-03-24T12:52:27.198 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1859720839197 got 1859720839199 for osd.0 2024-03-24T12:52:27.198 DEBUG:teuthology.parallel:result is None 2024-03-24T12:52:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:27 smithi190 ceph-mon[33748]: 6.1a scrub ok 2024-03-24T12:52:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:27 smithi190 ceph-mon[33748]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T12:52:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:27 smithi190 ceph-mon[33748]: 6.8 scrub starts 2024-03-24T12:52:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:27 smithi190 ceph-mon[33748]: 6.8 scrub ok 2024-03-24T12:52:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:27 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/2970538525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T12:52:27.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:27 smithi190 ceph-mon[33748]: pgmap v1173: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:52:27.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:52:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:52:27.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[41500]: 6.1a scrub ok 2024-03-24T12:52:27.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[41500]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T12:52:27.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[41500]: 6.8 scrub starts 2024-03-24T12:52:27.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[41500]: 6.8 scrub ok 2024-03-24T12:52:27.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/2970538525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T12:52:27.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[41500]: pgmap v1173: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:52:27.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[29919]: 6.1a scrub ok 2024-03-24T12:52:27.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[29919]: osdmap e483: 8 total, 8 up, 8 in 2024-03-24T12:52:27.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[29919]: 6.8 scrub starts 2024-03-24T12:52:27.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[29919]: 6.8 scrub ok 2024-03-24T12:52:27.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/2970538525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T12:52:27.918 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:27 smithi029 ceph-mon[29919]: pgmap v1173: 142 pgs: 1 active+clean+scrubbing, 141 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-24T12:52:28.747 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:52:28.747 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:52:28.747 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[41500]: 3.5 scrub starts 2024-03-24T12:52:28.747 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[41500]: 3.5 scrub ok 2024-03-24T12:52:28.747 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[41500]: 4.13 scrub starts 2024-03-24T12:52:28.747 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[41500]: 4.13 scrub ok 2024-03-24T12:52:28.747 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:52:28.747 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:52:28.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[29919]: 3.5 scrub starts 2024-03-24T12:52:28.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[29919]: 3.5 scrub ok 2024-03-24T12:52:28.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[29919]: 4.13 scrub starts 2024-03-24T12:52:28.748 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:28 smithi029 ceph-mon[29919]: 4.13 scrub ok 2024-03-24T12:52:28.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:28 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:52:28.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:28 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:52:28.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:28 smithi190 ceph-mon[33748]: 3.5 scrub starts 2024-03-24T12:52:28.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:28 smithi190 ceph-mon[33748]: 3.5 scrub ok 2024-03-24T12:52:28.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:28 smithi190 ceph-mon[33748]: 4.13 scrub starts 2024-03-24T12:52:28.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:28 smithi190 ceph-mon[33748]: 4.13 scrub ok 2024-03-24T12:52:29.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:52:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:52:29.666 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: 6.c scrub starts 2024-03-24T12:52:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: 6.c scrub ok 2024-03-24T12:52:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T12:52:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: purged_snaps scrub starts 2024-03-24T12:52:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: purged_snaps scrub ok 2024-03-24T12:52:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:52:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: 5.8 scrub starts 2024-03-24T12:52:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: 5.8 scrub ok 2024-03-24T12:52:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: 3.1c scrub starts 2024-03-24T12:52:29.667 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[41500]: 3.1c scrub ok 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: 6.c scrub starts 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: 6.c scrub ok 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: purged_snaps scrub starts 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: purged_snaps scrub ok 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: 5.8 scrub starts 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: 5.8 scrub ok 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: 3.1c scrub starts 2024-03-24T12:52:29.668 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:29 smithi029 ceph-mon[29919]: 3.1c scrub ok 2024-03-24T12:52:29.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: 6.c scrub starts 2024-03-24T12:52:29.948 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: 6.c scrub ok 2024-03-24T12:52:29.948 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: osdmap e484: 8 total, 8 up, 8 in 2024-03-24T12:52:29.948 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: purged_snaps scrub starts 2024-03-24T12:52:29.948 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: purged_snaps scrub ok 2024-03-24T12:52:29.948 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:29.948 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: pgmap v1175: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-24T12:52:29.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: 5.8 scrub starts 2024-03-24T12:52:29.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: 5.8 scrub ok 2024-03-24T12:52:29.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: 3.1c scrub starts 2024-03-24T12:52:29.949 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:29 smithi190 ceph-mon[33748]: 3.1c scrub ok 2024-03-24T12:52:29.950 INFO:teuthology.orchestra.run.smithi029.stdout:55834575202 2024-03-24T12:52:30.114 INFO:teuthology.orchestra.run.smithi029.stdout:1876900708380 2024-03-24T12:52:30.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:30 smithi190 ceph-mon[33748]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T12:52:30.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:30 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/4276344302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T12:52:30.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:30 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3865385002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T12:52:30.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:30 smithi029 ceph-mon[29919]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T12:52:30.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:30 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/4276344302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T12:52:30.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:30 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3865385002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T12:52:30.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:30 smithi029 ceph-mon[41500]: osdmap e485: 8 total, 8 up, 8 in 2024-03-24T12:52:30.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:30 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/4276344302' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T12:52:30.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:30 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3865385002' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T12:52:31.037 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332420 got 219043332423 for osd.6 2024-03-24T12:52:31.037 DEBUG:teuthology.parallel:result is None 2024-03-24T12:52:31.782 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[41500]: 5.12 scrub starts 2024-03-24T12:52:31.782 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[41500]: 5.12 scrub ok 2024-03-24T12:52:31.782 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[41500]: 6.1c scrub starts 2024-03-24T12:52:31.782 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[41500]: 6.1c scrub ok 2024-03-24T12:52:31.782 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[41500]: pgmap v1177: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-24T12:52:31.782 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[41500]: 4.d scrub starts 2024-03-24T12:52:31.782 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[41500]: 4.d scrub ok 2024-03-24T12:52:31.782 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/785376617' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T12:52:31.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[29919]: 5.12 scrub starts 2024-03-24T12:52:31.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[29919]: 5.12 scrub ok 2024-03-24T12:52:31.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[29919]: 6.1c scrub starts 2024-03-24T12:52:31.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[29919]: 6.1c scrub ok 2024-03-24T12:52:31.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[29919]: pgmap v1177: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-24T12:52:31.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[29919]: 4.d scrub starts 2024-03-24T12:52:31.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[29919]: 4.d scrub ok 2024-03-24T12:52:31.783 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:31 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/785376617' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T12:52:31.808 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575198 got 55834575202 for osd.1 2024-03-24T12:52:31.808 DEBUG:teuthology.parallel:result is None 2024-03-24T12:52:31.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:31 smithi190 ceph-mon[33748]: 5.12 scrub starts 2024-03-24T12:52:31.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:31 smithi190 ceph-mon[33748]: 5.12 scrub ok 2024-03-24T12:52:31.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:31 smithi190 ceph-mon[33748]: 6.1c scrub starts 2024-03-24T12:52:31.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:31 smithi190 ceph-mon[33748]: 6.1c scrub ok 2024-03-24T12:52:31.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:31 smithi190 ceph-mon[33748]: pgmap v1177: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 923 B/s rd, 0 op/s 2024-03-24T12:52:31.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:31 smithi190 ceph-mon[33748]: 4.d scrub starts 2024-03-24T12:52:31.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:31 smithi190 ceph-mon[33748]: 4.d scrub ok 2024-03-24T12:52:31.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:31 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/785376617' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T12:52:32.169 INFO:teuthology.orchestra.run.smithi029.stdout:115964117334 2024-03-24T12:52:32.797 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117328 got 115964117334 for osd.3 2024-03-24T12:52:32.797 DEBUG:teuthology.parallel:result is None 2024-03-24T12:52:33.169 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411673 got 77309411675 for osd.2 2024-03-24T12:52:33.169 DEBUG:teuthology.parallel:result is None 2024-03-24T12:52:33.663 INFO:teuthology.orchestra.run.smithi029.stdout:141733921107 2024-03-24T12:52:33.940 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1876900708375 got 1876900708380 for osd.5 2024-03-24T12:52:33.940 DEBUG:teuthology.parallel:result is None 2024-03-24T12:52:34.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:34 smithi029 ceph-mon[29919]: 3.b deep-scrub starts 2024-03-24T12:52:34.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:34 smithi029 ceph-mon[29919]: 3.b deep-scrub ok 2024-03-24T12:52:34.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:34 smithi029 ceph-mon[29919]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T12:52:34.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:34 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/1253460766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T12:52:34.167 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:34 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:52:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:34 smithi029 ceph-mon[41500]: 3.b deep-scrub starts 2024-03-24T12:52:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:34 smithi029 ceph-mon[41500]: 3.b deep-scrub ok 2024-03-24T12:52:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:34 smithi029 ceph-mon[41500]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T12:52:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:34 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/1253460766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T12:52:34.167 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:34 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:52:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:34 smithi190 ceph-mon[33748]: 3.b deep-scrub starts 2024-03-24T12:52:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:34 smithi190 ceph-mon[33748]: 3.b deep-scrub ok 2024-03-24T12:52:34.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:34 smithi190 ceph-mon[33748]: pgmap v1178: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-24T12:52:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:34 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/1253460766' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T12:52:34.383 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:34 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T12:52:34.703 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733921102 got 141733921107 for osd.4 2024-03-24T12:52:34.703 DEBUG:teuthology.parallel:result is None 2024-03-24T12:52:34.703 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-24T12:52:34.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:52:34.956 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:35.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:35 smithi029 ceph-mon[41500]: 4.17 deep-scrub starts 2024-03-24T12:52:35.356 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:35 smithi029 ceph-mon[41500]: 4.17 deep-scrub ok 2024-03-24T12:52:35.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:35 smithi029 ceph-mon[29919]: 4.17 deep-scrub starts 2024-03-24T12:52:35.356 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:35 smithi029 ceph-mon[29919]: 4.17 deep-scrub ok 2024-03-24T12:52:35.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:35 smithi190 ceph-mon[33748]: 4.17 deep-scrub starts 2024-03-24T12:52:35.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:35 smithi190 ceph-mon[33748]: 4.17 deep-scrub ok 2024-03-24T12:52:36.115 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[29919]: 4.10 scrub starts 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[29919]: 4.10 scrub ok 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[29919]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[29919]: 5.7 scrub starts 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[29919]: 5.7 scrub ok 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[29919]: 6.2 scrub starts 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[29919]: 6.2 scrub ok 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[41500]: 4.10 scrub starts 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[41500]: 4.10 scrub ok 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[41500]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[41500]: 5.7 scrub starts 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[41500]: 5.7 scrub ok 2024-03-24T12:52:36.116 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[41500]: 6.2 scrub starts 2024-03-24T12:52:36.117 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:36 smithi029 ceph-mon[41500]: 6.2 scrub ok 2024-03-24T12:52:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:36 smithi190 ceph-mon[33748]: 4.10 scrub starts 2024-03-24T12:52:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:36 smithi190 ceph-mon[33748]: 4.10 scrub ok 2024-03-24T12:52:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:36 smithi190 ceph-mon[33748]: pgmap v1179: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:36 smithi190 ceph-mon[33748]: 5.7 scrub starts 2024-03-24T12:52:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:36 smithi190 ceph-mon[33748]: 5.7 scrub ok 2024-03-24T12:52:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:36 smithi190 ceph-mon[33748]: 6.2 scrub starts 2024-03-24T12:52:36.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:36 smithi190 ceph-mon[33748]: 6.2 scrub ok 2024-03-24T12:52:36.928 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:52:36.930 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:52:37.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:37 smithi029 ceph-mon[29919]: 1.2 scrub starts 2024-03-24T12:52:37.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:37 smithi029 ceph-mon[29919]: 1.2 scrub ok 2024-03-24T12:52:37.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:37 smithi029 ceph-mon[29919]: 4.9 scrub starts 2024-03-24T12:52:37.298 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:37 smithi029 ceph-mon[29919]: 4.9 scrub ok 2024-03-24T12:52:37.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:37 smithi029 ceph-mon[41500]: 1.2 scrub starts 2024-03-24T12:52:37.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:37 smithi029 ceph-mon[41500]: 1.2 scrub ok 2024-03-24T12:52:37.298 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:37 smithi029 ceph-mon[41500]: 4.9 scrub starts 2024-03-24T12:52:37.299 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:37 smithi029 ceph-mon[41500]: 4.9 scrub ok 2024-03-24T12:52:37.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:37 smithi190 ceph-mon[33748]: 1.2 scrub starts 2024-03-24T12:52:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:37 smithi190 ceph-mon[33748]: 1.2 scrub ok 2024-03-24T12:52:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:37 smithi190 ceph-mon[33748]: 4.9 scrub starts 2024-03-24T12:52:37.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:37 smithi190 ceph-mon[33748]: 4.9 scrub ok 2024-03-24T12:52:37.736 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":1179,"stamp":"2024-03-24T12:52:34.986100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3124,"num_read_kb":2876,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":1657173,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2605,"ondisk_log_size":2605,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4981064,"kb_used_data":7844,"kb_used_omap":20,"kb_used_meta":4973099,"kb_avail":744980152,"statfs":{"total":767960285184,"available":762859675648,"internally_reserved":0,"allocated":8032256,"data_stored":4852102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20883,"internal_metadata":5092453997},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002008"},"pg_stats":[{"pgid":"4.c","version":"390'30","reported_seq":1067,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473578+0000","last_change":"2024-03-24T12:51:46.244134+0000","last_active":"2024-03-24T12:52:29.473578+0000","last_peered":"2024-03-24T12:52:29.473578+0000","last_clean":"2024-03-24T12:52:29.473578+0000","last_became_active":"2024-03-24T12:50:31.813808+0000","last_became_peered":"2024-03-24T12:50:31.813808+0000","last_unstale":"2024-03-24T12:52:29.473578+0000","last_undegraded":"2024-03-24T12:52:29.473578+0000","last_fullsized":"2024-03-24T12:52:29.473578+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'30","last_scrub_stamp":"2024-03-24T12:51:46.244011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:46.244011+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:51.518564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:32.303852+0000","last_change":"2024-03-24T12:52:32.303874+0000","last_active":"2024-03-24T12:52:32.303852+0000","last_peered":"2024-03-24T12:52:32.303852+0000","last_clean":"2024-03-24T12:52:32.303852+0000","last_became_active":"2024-03-24T12:51:04.953574+0000","last_became_peered":"2024-03-24T12:51:04.953574+0000","last_unstale":"2024-03-24T12:52:32.303852+0000","last_undegraded":"2024-03-24T12:52:32.303852+0000","last_fullsized":"2024-03-24T12:52:32.303852+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:32.303785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:32.303785+0000","last_clean_scrub_stamp":"2024-03-24T12:52:32.303785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:35.811701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"440'13","reported_seq":1277,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473613+0000","last_change":"2024-03-24T12:51:28.013037+0000","last_active":"2024-03-24T12:52:29.473613+0000","last_peered":"2024-03-24T12:52:29.473613+0000","last_clean":"2024-03-24T12:52:29.473613+0000","last_became_active":"2024-03-24T12:50:27.806738+0000","last_became_peered":"2024-03-24T12:50:27.806738+0000","last_unstale":"2024-03-24T12:52:29.473613+0000","last_undegraded":"2024-03-24T12:52:29.473613+0000","last_fullsized":"2024-03-24T12:52:29.473613+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13","last_scrub_stamp":"2024-03-24T12:51:28.012891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:28.012891+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:54.551766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":942,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473806+0000","last_change":"2024-03-24T12:52:22.286274+0000","last_active":"2024-03-24T12:52:29.473806+0000","last_peered":"2024-03-24T12:52:29.473806+0000","last_clean":"2024-03-24T12:52:29.473806+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:52:29.473806+0000","last_undegraded":"2024-03-24T12:52:29.473806+0000","last_fullsized":"2024-03-24T12:52:29.473806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:22.286131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:42:45.013715+0000","last_clean_scrub_stamp":"2024-03-24T12:52:22.286131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:47.522175+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1256,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903063+0000","last_change":"2024-03-24T12:52:11.720132+0000","last_active":"2024-03-24T12:52:29.903063+0000","last_peered":"2024-03-24T12:52:29.903063+0000","last_clean":"2024-03-24T12:52:29.903063+0000","last_became_active":"2024-03-24T12:41:52.953388+0000","last_became_peered":"2024-03-24T12:41:52.953388+0000","last_unstale":"2024-03-24T12:52:29.903063+0000","last_undegraded":"2024-03-24T12:52:29.903063+0000","last_fullsized":"2024-03-24T12:52:29.903063+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:52:11.720024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:52:11.720024+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:17.140601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"390'53","reported_seq":1028,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:31.193673+0000","last_change":"2024-03-24T12:52:31.193733+0000","last_active":"2024-03-24T12:52:31.193673+0000","last_peered":"2024-03-24T12:52:31.193673+0000","last_clean":"2024-03-24T12:52:31.193673+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:52:31.193673+0000","last_undegraded":"2024-03-24T12:52:31.193673+0000","last_fullsized":"2024-03-24T12:52:31.193673+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'53","last_scrub_stamp":"2024-03-24T12:52:31.193601+0000","last_deep_scrub":"207'39","last_deep_scrub_stamp":"2024-03-24T12:36:22.669686+0000","last_clean_scrub_stamp":"2024-03-24T12:52:31.193601+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:47.495688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":108,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835396+0000","last_change":"2024-03-24T12:51:43.316224+0000","last_active":"2024-03-24T12:52:29.835396+0000","last_peered":"2024-03-24T12:52:29.835396+0000","last_clean":"2024-03-24T12:52:29.835396+0000","last_became_active":"2024-03-24T12:50:31.810110+0000","last_became_peered":"2024-03-24T12:50:31.810110+0000","last_unstale":"2024-03-24T12:52:29.835396+0000","last_undegraded":"2024-03-24T12:52:29.835396+0000","last_fullsized":"2024-03-24T12:52:29.835396+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:43.316129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:43.316129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:44.079453+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":506,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903605+0000","last_change":"2024-03-24T12:52:17.893939+0000","last_active":"2024-03-24T12:52:29.903605+0000","last_peered":"2024-03-24T12:52:29.903605+0000","last_clean":"2024-03-24T12:52:29.903605+0000","last_became_active":"2024-03-24T12:50:53.091678+0000","last_became_peered":"2024-03-24T12:50:53.091678+0000","last_unstale":"2024-03-24T12:52:29.903605+0000","last_undegraded":"2024-03-24T12:52:29.903605+0000","last_fullsized":"2024-03-24T12:52:29.903605+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:17.893845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:52:17.893845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:26.182758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":968,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473111+0000","last_change":"2024-03-24T12:51:58.021051+0000","last_active":"2024-03-24T12:52:29.473111+0000","last_peered":"2024-03-24T12:52:29.473111+0000","last_clean":"2024-03-24T12:52:29.473111+0000","last_became_active":"2024-03-24T12:51:04.946095+0000","last_became_peered":"2024-03-24T12:51:04.946095+0000","last_unstale":"2024-03-24T12:52:29.473111+0000","last_undegraded":"2024-03-24T12:52:29.473111+0000","last_fullsized":"2024-03-24T12:52:29.473111+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:58.020949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:53.627010+0000","last_clean_scrub_stamp":"2024-03-24T12:51:58.020949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:17.580521+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":636,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902647+0000","last_change":"2024-03-24T12:51:33.193289+0000","last_active":"2024-03-24T12:52:29.902647+0000","last_peered":"2024-03-24T12:52:29.902647+0000","last_clean":"2024-03-24T12:52:29.902647+0000","last_became_active":"2024-03-24T12:51:07.935815+0000","last_became_peered":"2024-03-24T12:51:07.935815+0000","last_unstale":"2024-03-24T12:52:29.902647+0000","last_undegraded":"2024-03-24T12:52:29.902647+0000","last_fullsized":"2024-03-24T12:52:29.902647+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:33.193159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:51:33.193159+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:57.291047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1313440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"390'27","reported_seq":1093,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473718+0000","last_change":"2024-03-24T12:51:52.197139+0000","last_active":"2024-03-24T12:52:29.473718+0000","last_peered":"2024-03-24T12:52:29.473718+0000","last_clean":"2024-03-24T12:52:29.473718+0000","last_became_active":"2024-03-24T12:50:31.824877+0000","last_became_peered":"2024-03-24T12:50:31.824877+0000","last_unstale":"2024-03-24T12:52:29.473718+0000","last_undegraded":"2024-03-24T12:52:29.473718+0000","last_fullsized":"2024-03-24T12:52:29.473718+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'27","last_scrub_stamp":"2024-03-24T12:51:52.197041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:52.197041+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:16.595704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":939,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472808+0000","last_change":"2024-03-24T12:51:27.213082+0000","last_active":"2024-03-24T12:52:29.472808+0000","last_peered":"2024-03-24T12:52:29.472808+0000","last_clean":"2024-03-24T12:52:29.472808+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:52:29.472808+0000","last_undegraded":"2024-03-24T12:52:29.472808+0000","last_fullsized":"2024-03-24T12:52:29.472808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:27.212982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:27.212982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:55.955204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":215,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904385+0000","last_change":"2024-03-24T12:51:55.067409+0000","last_active":"2024-03-24T12:52:29.904385+0000","last_peered":"2024-03-24T12:52:29.904385+0000","last_clean":"2024-03-24T12:52:29.904385+0000","last_became_active":"2024-03-24T12:46:46.944569+0000","last_became_peered":"2024-03-24T12:46:46.944569+0000","last_unstale":"2024-03-24T12:52:29.904385+0000","last_undegraded":"2024-03-24T12:52:29.904385+0000","last_fullsized":"2024-03-24T12:52:29.904385+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:55.067325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:55.067325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:22.524561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904487+0000","last_change":"2024-03-24T12:52:28.184400+0000","last_active":"2024-03-24T12:52:29.904487+0000","last_peered":"2024-03-24T12:52:29.904487+0000","last_clean":"2024-03-24T12:52:29.904487+0000","last_became_active":"2024-03-24T12:51:04.943360+0000","last_became_peered":"2024-03-24T12:51:04.943360+0000","last_unstale":"2024-03-24T12:52:29.904487+0000","last_undegraded":"2024-03-24T12:52:29.904487+0000","last_fullsized":"2024-03-24T12:52:29.904487+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:28.184313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:07.890930+0000","last_clean_scrub_stamp":"2024-03-24T12:52:28.184313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:42.512115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"391'46","reported_seq":1001,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473352+0000","last_change":"2024-03-24T12:51:56.890687+0000","last_active":"2024-03-24T12:52:29.473352+0000","last_peered":"2024-03-24T12:52:29.473352+0000","last_clean":"2024-03-24T12:52:29.473352+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:52:29.473352+0000","last_undegraded":"2024-03-24T12:52:29.473352+0000","last_fullsized":"2024-03-24T12:52:29.473352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'46","last_scrub_stamp":"2024-03-24T12:51:56.890514+0000","last_deep_scrub":"391'46","last_deep_scrub_stamp":"2024-03-24T12:50:34.028632+0000","last_clean_scrub_stamp":"2024-03-24T12:51:56.890514+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:07.160033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902698+0000","last_change":"2024-03-24T12:51:53.043463+0000","last_active":"2024-03-24T12:52:29.902698+0000","last_peered":"2024-03-24T12:52:29.902698+0000","last_clean":"2024-03-24T12:52:29.902698+0000","last_became_active":"2024-03-24T12:51:04.955308+0000","last_became_peered":"2024-03-24T12:51:04.955308+0000","last_unstale":"2024-03-24T12:52:29.902698+0000","last_undegraded":"2024-03-24T12:52:29.902698+0000","last_fullsized":"2024-03-24T12:52:29.902698+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:53.043370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:53.043370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:09.727128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"440'14","reported_seq":1278,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473671+0000","last_change":"2024-03-24T12:52:08.220047+0000","last_active":"2024-03-24T12:52:29.473671+0000","last_peered":"2024-03-24T12:52:29.473671+0000","last_clean":"2024-03-24T12:52:29.473671+0000","last_became_active":"2024-03-24T12:50:27.805141+0000","last_became_peered":"2024-03-24T12:50:27.805141+0000","last_unstale":"2024-03-24T12:52:29.473671+0000","last_undegraded":"2024-03-24T12:52:29.473671+0000","last_fullsized":"2024-03-24T12:52:29.473671+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'14","last_scrub_stamp":"2024-03-24T12:52:08.219892+0000","last_deep_scrub":"440'14","last_deep_scrub_stamp":"2024-03-24T12:50:49.885051+0000","last_clean_scrub_stamp":"2024-03-24T12:52:08.219892+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:13.162516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":117,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904207+0000","last_change":"2024-03-24T12:52:12.213861+0000","last_active":"2024-03-24T12:52:29.904207+0000","last_peered":"2024-03-24T12:52:29.904207+0000","last_clean":"2024-03-24T12:52:29.904207+0000","last_became_active":"2024-03-24T12:50:27.814939+0000","last_became_peered":"2024-03-24T12:50:27.814939+0000","last_unstale":"2024-03-24T12:52:29.904207+0000","last_undegraded":"2024-03-24T12:52:29.904207+0000","last_fullsized":"2024-03-24T12:52:29.904207+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:12.213729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:00.198109+0000","last_clean_scrub_stamp":"2024-03-24T12:52:12.213729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:37.473828+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":583,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903144+0000","last_change":"2024-03-24T12:51:18.327429+0000","last_active":"2024-03-24T12:52:29.903144+0000","last_peered":"2024-03-24T12:52:29.903144+0000","last_clean":"2024-03-24T12:52:29.903144+0000","last_became_active":"2024-03-24T12:50:34.821401+0000","last_became_peered":"2024-03-24T12:50:34.821401+0000","last_unstale":"2024-03-24T12:52:29.903144+0000","last_undegraded":"2024-03-24T12:52:29.903144+0000","last_fullsized":"2024-03-24T12:52:29.903144+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:18.327295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:51:18.327295+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:40.745547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":978,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472786+0000","last_change":"2024-03-24T12:51:40.198464+0000","last_active":"2024-03-24T12:52:29.472786+0000","last_peered":"2024-03-24T12:52:29.472786+0000","last_clean":"2024-03-24T12:52:29.472786+0000","last_became_active":"2024-03-24T12:51:29.008877+0000","last_became_peered":"2024-03-24T12:51:29.008877+0000","last_unstale":"2024-03-24T12:52:29.472786+0000","last_undegraded":"2024-03-24T12:52:29.472786+0000","last_fullsized":"2024-03-24T12:52:29.472786+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:40.198374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:59.761116+0000","last_clean_scrub_stamp":"2024-03-24T12:51:40.198374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:59.717777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"450'50","reported_seq":119,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902786+0000","last_change":"2024-03-24T12:52:23.071097+0000","last_active":"2024-03-24T12:52:29.902786+0000","last_peered":"2024-03-24T12:52:29.902786+0000","last_clean":"2024-03-24T12:52:29.902786+0000","last_became_active":"2024-03-24T12:51:29.010620+0000","last_became_peered":"2024-03-24T12:51:29.010620+0000","last_unstale":"2024-03-24T12:52:29.902786+0000","last_undegraded":"2024-03-24T12:52:29.902786+0000","last_fullsized":"2024-03-24T12:52:29.902786+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'50","last_scrub_stamp":"2024-03-24T12:52:23.070958+0000","last_deep_scrub":"450'50","last_deep_scrub_stamp":"2024-03-24T12:50:49.623869+0000","last_clean_scrub_stamp":"2024-03-24T12:52:23.070958+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:43.306693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":940,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904479+0000","last_change":"2024-03-24T12:51:52.063115+0000","last_active":"2024-03-24T12:52:29.904479+0000","last_peered":"2024-03-24T12:52:29.904479+0000","last_clean":"2024-03-24T12:52:29.904479+0000","last_became_active":"2024-03-24T12:49:54.335714+0000","last_became_peered":"2024-03-24T12:49:54.335714+0000","last_unstale":"2024-03-24T12:52:29.904479+0000","last_undegraded":"2024-03-24T12:52:29.904479+0000","last_fullsized":"2024-03-24T12:52:29.904479+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:52.063033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:52.063033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:06.673419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":945,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473010+0000","last_change":"2024-03-24T12:51:13.115221+0000","last_active":"2024-03-24T12:52:29.473010+0000","last_peered":"2024-03-24T12:52:29.473010+0000","last_clean":"2024-03-24T12:52:29.473010+0000","last_became_active":"2024-03-24T12:51:04.946394+0000","last_became_peered":"2024-03-24T12:51:04.946394+0000","last_unstale":"2024-03-24T12:52:29.473010+0000","last_undegraded":"2024-03-24T12:52:29.473010+0000","last_fullsized":"2024-03-24T12:52:29.473010+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:13.115088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:51:13.115088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":755,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473494+0000","last_change":"2024-03-24T12:51:39.228740+0000","last_active":"2024-03-24T12:52:29.473494+0000","last_peered":"2024-03-24T12:52:29.473494+0000","last_clean":"2024-03-24T12:52:29.473494+0000","last_became_active":"2024-03-24T12:50:34.830594+0000","last_became_peered":"2024-03-24T12:50:34.830594+0000","last_unstale":"2024-03-24T12:52:29.473494+0000","last_undegraded":"2024-03-24T12:52:29.473494+0000","last_fullsized":"2024-03-24T12:52:29.473494+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:39.228652+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T12:51:39.228652+0000","last_clean_scrub_stamp":"2024-03-24T12:51:39.228652+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.727008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"459'13","reported_seq":98,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902585+0000","last_change":"2024-03-24T12:52:05.000817+0000","last_active":"2024-03-24T12:52:29.902585+0000","last_peered":"2024-03-24T12:52:29.902585+0000","last_clean":"2024-03-24T12:52:29.902585+0000","last_became_active":"2024-03-24T12:51:04.958698+0000","last_became_peered":"2024-03-24T12:51:04.958698+0000","last_unstale":"2024-03-24T12:52:29.902585+0000","last_undegraded":"2024-03-24T12:52:29.902585+0000","last_fullsized":"2024-03-24T12:52:29.902585+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'13","last_scrub_stamp":"2024-03-24T12:52:05.000708+0000","last_deep_scrub":"417'10","last_deep_scrub_stamp":"2024-03-24T12:48:14.628984+0000","last_clean_scrub_stamp":"2024-03-24T12:52:05.000708+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:05.026370+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"390'49","reported_seq":1115,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473000+0000","last_change":"2024-03-24T12:51:36.186049+0000","last_active":"2024-03-24T12:52:29.473000+0000","last_peered":"2024-03-24T12:52:29.473000+0000","last_clean":"2024-03-24T12:52:29.473000+0000","last_became_active":"2024-03-24T12:50:31.813242+0000","last_became_peered":"2024-03-24T12:50:31.813242+0000","last_unstale":"2024-03-24T12:52:29.473000+0000","last_undegraded":"2024-03-24T12:52:29.473000+0000","last_fullsized":"2024-03-24T12:52:29.473000+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'49","last_scrub_stamp":"2024-03-24T12:51:36.185877+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:51:36.185877+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:01.750024+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":880,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474439+0000","last_change":"2024-03-24T12:51:49.643147+0000","last_active":"2024-03-24T12:52:29.474439+0000","last_peered":"2024-03-24T12:52:29.474439+0000","last_clean":"2024-03-24T12:52:29.474439+0000","last_became_active":"2024-03-24T12:49:54.360509+0000","last_became_peered":"2024-03-24T12:49:54.360509+0000","last_unstale":"2024-03-24T12:52:29.474439+0000","last_undegraded":"2024-03-24T12:52:29.474439+0000","last_fullsized":"2024-03-24T12:52:29.474439+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:51:49.643033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:49.643033+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:05.384239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":960,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904237+0000","last_change":"2024-03-24T12:52:22.975994+0000","last_active":"2024-03-24T12:52:29.904237+0000","last_peered":"2024-03-24T12:52:29.904237+0000","last_clean":"2024-03-24T12:52:29.904237+0000","last_became_active":"2024-03-24T12:50:31.823357+0000","last_became_peered":"2024-03-24T12:50:31.823357+0000","last_unstale":"2024-03-24T12:52:29.904237+0000","last_undegraded":"2024-03-24T12:52:29.904237+0000","last_fullsized":"2024-03-24T12:52:29.904237+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:22.975898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:03.105774+0000","last_clean_scrub_stamp":"2024-03-24T12:52:22.975898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:25.616897+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"447'129","reported_seq":940,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904588+0000","last_change":"2024-03-24T12:52:17.123283+0000","last_active":"2024-03-24T12:52:29.904588+0000","last_peered":"2024-03-24T12:52:29.904588+0000","last_clean":"2024-03-24T12:52:29.904588+0000","last_became_active":"2024-03-24T12:51:07.937015+0000","last_became_peered":"2024-03-24T12:51:07.937015+0000","last_unstale":"2024-03-24T12:52:29.904588+0000","last_undegraded":"2024-03-24T12:52:29.904588+0000","last_fullsized":"2024-03-24T12:52:29.904588+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":3,"last_scrub":"447'129","last_scrub_stamp":"2024-03-24T12:52:17.123191+0000","last_deep_scrub":"447'129","last_deep_scrub_stamp":"2024-03-24T12:50:35.407598+0000","last_clean_scrub_stamp":"2024-03-24T12:52:17.123191+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:39.925267+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"445'20","reported_seq":1268,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473947+0000","last_change":"2024-03-24T12:52:20.418347+0000","last_active":"2024-03-24T12:52:29.473947+0000","last_peered":"2024-03-24T12:52:29.473947+0000","last_clean":"2024-03-24T12:52:29.473947+0000","last_became_active":"2024-03-24T12:50:31.819567+0000","last_became_peered":"2024-03-24T12:50:31.819567+0000","last_unstale":"2024-03-24T12:52:29.473947+0000","last_undegraded":"2024-03-24T12:52:29.473947+0000","last_fullsized":"2024-03-24T12:52:29.473947+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'20","last_scrub_stamp":"2024-03-24T12:52:20.418264+0000","last_deep_scrub":"318'14","last_deep_scrub_stamp":"2024-03-24T12:42:45.093908+0000","last_clean_scrub_stamp":"2024-03-24T12:52:20.418264+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:46.222323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"390'35","reported_seq":101,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903387+0000","last_change":"2024-03-24T12:51:58.862508+0000","last_active":"2024-03-24T12:52:29.903387+0000","last_peered":"2024-03-24T12:52:29.903387+0000","last_clean":"2024-03-24T12:52:29.903387+0000","last_became_active":"2024-03-24T12:50:48.871716+0000","last_became_peered":"2024-03-24T12:50:48.871716+0000","last_unstale":"2024-03-24T12:52:29.903387+0000","last_undegraded":"2024-03-24T12:52:29.903387+0000","last_fullsized":"2024-03-24T12:52:29.903387+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'35","last_scrub_stamp":"2024-03-24T12:51:58.862393+0000","last_deep_scrub":"390'35","last_deep_scrub_stamp":"2024-03-24T12:51:58.862393+0000","last_clean_scrub_stamp":"2024-03-24T12:51:58.862393+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:01.616480+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":129,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904109+0000","last_change":"2024-03-24T12:52:28.169863+0000","last_active":"2024-03-24T12:52:29.904109+0000","last_peered":"2024-03-24T12:52:29.904109+0000","last_clean":"2024-03-24T12:52:29.904109+0000","last_became_active":"2024-03-24T12:51:04.946157+0000","last_became_peered":"2024-03-24T12:51:04.946157+0000","last_unstale":"2024-03-24T12:52:29.904109+0000","last_undegraded":"2024-03-24T12:52:29.904109+0000","last_fullsized":"2024-03-24T12:52:29.904109+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:28.169714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:52:28.169714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:34.596704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":861,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472860+0000","last_change":"2024-03-24T12:51:11.671035+0000","last_active":"2024-03-24T12:52:29.472860+0000","last_peered":"2024-03-24T12:52:29.472860+0000","last_clean":"2024-03-24T12:52:29.472860+0000","last_became_active":"2024-03-24T12:51:04.944332+0000","last_became_peered":"2024-03-24T12:51:04.944332+0000","last_unstale":"2024-03-24T12:52:29.472860+0000","last_undegraded":"2024-03-24T12:52:29.472860+0000","last_fullsized":"2024-03-24T12:52:29.472860+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:11.670904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:51:11.670904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:38.001797+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":821,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835965+0000","last_change":"2024-03-24T12:52:01.103100+0000","last_active":"2024-03-24T12:52:29.835965+0000","last_peered":"2024-03-24T12:52:29.835965+0000","last_clean":"2024-03-24T12:52:29.835965+0000","last_became_active":"2024-03-24T12:51:07.936943+0000","last_became_peered":"2024-03-24T12:51:07.936943+0000","last_unstale":"2024-03-24T12:52:29.835965+0000","last_undegraded":"2024-03-24T12:52:29.835965+0000","last_fullsized":"2024-03-24T12:52:29.835965+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:52:01.103008+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T12:48:16.724235+0000","last_clean_scrub_stamp":"2024-03-24T12:52:01.103008+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:16.285328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":108,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835526+0000","last_change":"2024-03-24T12:52:03.077440+0000","last_active":"2024-03-24T12:52:29.835526+0000","last_peered":"2024-03-24T12:52:29.835526+0000","last_clean":"2024-03-24T12:52:29.835526+0000","last_became_active":"2024-03-24T12:50:31.817753+0000","last_became_peered":"2024-03-24T12:50:31.817753+0000","last_unstale":"2024-03-24T12:52:29.835526+0000","last_undegraded":"2024-03-24T12:52:29.835526+0000","last_fullsized":"2024-03-24T12:52:29.835526+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:03.077349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:03.077349+0000","last_clean_scrub_stamp":"2024-03-24T12:52:03.077349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:24.824316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"390'47","reported_seq":1001,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474315+0000","last_change":"2024-03-24T12:51:43.650531+0000","last_active":"2024-03-24T12:52:29.474315+0000","last_peered":"2024-03-24T12:52:29.474315+0000","last_clean":"2024-03-24T12:52:29.474315+0000","last_became_active":"2024-03-24T12:50:27.807819+0000","last_became_peered":"2024-03-24T12:50:27.807819+0000","last_unstale":"2024-03-24T12:52:29.474315+0000","last_undegraded":"2024-03-24T12:52:29.474315+0000","last_fullsized":"2024-03-24T12:52:29.474315+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'47","last_scrub_stamp":"2024-03-24T12:51:43.650438+0000","last_deep_scrub":"195'27","last_deep_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_clean_scrub_stamp":"2024-03-24T12:51:43.650438+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:57.775684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":840,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903341+0000","last_change":"2024-03-24T12:51:34.863763+0000","last_active":"2024-03-24T12:52:29.903341+0000","last_peered":"2024-03-24T12:52:29.903341+0000","last_clean":"2024-03-24T12:52:29.903341+0000","last_became_active":"2024-03-24T12:50:53.092560+0000","last_became_peered":"2024-03-24T12:50:53.092560+0000","last_unstale":"2024-03-24T12:52:29.903341+0000","last_undegraded":"2024-03-24T12:52:29.903341+0000","last_fullsized":"2024-03-24T12:52:29.903341+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:34.863682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:34.863682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:45.166072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":983,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903045+0000","last_change":"2024-03-24T12:52:02.915127+0000","last_active":"2024-03-24T12:52:29.903045+0000","last_peered":"2024-03-24T12:52:29.903045+0000","last_clean":"2024-03-24T12:52:29.903045+0000","last_became_active":"2024-03-24T12:50:31.810981+0000","last_became_peered":"2024-03-24T12:50:31.810981+0000","last_unstale":"2024-03-24T12:52:29.903045+0000","last_undegraded":"2024-03-24T12:52:29.903045+0000","last_fullsized":"2024-03-24T12:52:29.903045+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:02.915028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:19.627429+0000","last_clean_scrub_stamp":"2024-03-24T12:52:02.915028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:02.929174+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":563,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835828+0000","last_change":"2024-03-24T12:51:27.442953+0000","last_active":"2024-03-24T12:52:29.835828+0000","last_peered":"2024-03-24T12:52:29.835828+0000","last_clean":"2024-03-24T12:52:29.835828+0000","last_became_active":"2024-03-24T12:51:07.935124+0000","last_became_peered":"2024-03-24T12:51:07.935124+0000","last_unstale":"2024-03-24T12:52:29.835828+0000","last_undegraded":"2024-03-24T12:52:29.835828+0000","last_fullsized":"2024-03-24T12:52:29.835828+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:27.442795+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T12:51:27.442795+0000","last_clean_scrub_stamp":"2024-03-24T12:51:27.442795+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:46.528507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":109,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904786+0000","last_change":"2024-03-24T12:51:19.337021+0000","last_active":"2024-03-24T12:52:29.904786+0000","last_peered":"2024-03-24T12:52:29.904786+0000","last_clean":"2024-03-24T12:52:29.904786+0000","last_became_active":"2024-03-24T12:50:27.803276+0000","last_became_peered":"2024-03-24T12:50:27.803276+0000","last_unstale":"2024-03-24T12:52:29.904786+0000","last_undegraded":"2024-03-24T12:52:29.904786+0000","last_fullsized":"2024-03-24T12:52:29.904786+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:19.336919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:07.923592+0000","last_clean_scrub_stamp":"2024-03-24T12:51:19.336919+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"390'42","reported_seq":918,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474124+0000","last_change":"2024-03-24T12:51:17.642108+0000","last_active":"2024-03-24T12:52:29.474124+0000","last_peered":"2024-03-24T12:52:29.474124+0000","last_clean":"2024-03-24T12:52:29.474124+0000","last_became_active":"2024-03-24T12:49:54.352611+0000","last_became_peered":"2024-03-24T12:49:54.352611+0000","last_unstale":"2024-03-24T12:52:29.474124+0000","last_undegraded":"2024-03-24T12:52:29.474124+0000","last_fullsized":"2024-03-24T12:52:29.474124+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'42","last_scrub_stamp":"2024-03-24T12:51:17.642011+0000","last_deep_scrub":"260'30","last_deep_scrub_stamp":"2024-03-24T12:39:51.288276+0000","last_clean_scrub_stamp":"2024-03-24T12:51:17.642011+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:43.109003+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2555,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:31.584524+0000","last_change":"2024-03-24T12:51:27.802927+0000","last_active":"2024-03-24T12:52:31.584524+0000","last_peered":"2024-03-24T12:52:31.584524+0000","last_clean":"2024-03-24T12:52:31.584524+0000","last_became_active":"2024-03-24T12:49:54.337309+0000","last_became_peered":"2024-03-24T12:49:54.337309+0000","last_unstale":"2024-03-24T12:52:31.584524+0000","last_undegraded":"2024-03-24T12:52:31.584524+0000","last_fullsized":"2024-03-24T12:52:31.584524+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:51:27.802797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:27.802797+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:35.222016+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1574,"num_read_kb":1569,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":889,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472722+0000","last_change":"2024-03-24T12:51:57.531772+0000","last_active":"2024-03-24T12:52:29.472722+0000","last_peered":"2024-03-24T12:52:29.472722+0000","last_clean":"2024-03-24T12:52:29.472722+0000","last_became_active":"2024-03-24T12:51:04.937869+0000","last_became_peered":"2024-03-24T12:51:04.937869+0000","last_unstale":"2024-03-24T12:52:29.472722+0000","last_undegraded":"2024-03-24T12:52:29.472722+0000","last_fullsized":"2024-03-24T12:52:29.472722+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:57.531645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:13.629968+0000","last_clean_scrub_stamp":"2024-03-24T12:51:57.531645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:04.827173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":966,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902669+0000","last_change":"2024-03-24T12:52:04.806381+0000","last_active":"2024-03-24T12:52:29.902669+0000","last_peered":"2024-03-24T12:52:29.902669+0000","last_clean":"2024-03-24T12:52:29.902669+0000","last_became_active":"2024-03-24T12:51:04.944003+0000","last_became_peered":"2024-03-24T12:51:04.944003+0000","last_unstale":"2024-03-24T12:52:29.902669+0000","last_undegraded":"2024-03-24T12:52:29.902669+0000","last_fullsized":"2024-03-24T12:52:29.902669+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:04.806299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:03.186807+0000","last_clean_scrub_stamp":"2024-03-24T12:52:04.806299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:33.723170+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":804,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902669+0000","last_change":"2024-03-24T12:52:06.822906+0000","last_active":"2024-03-24T12:52:29.902669+0000","last_peered":"2024-03-24T12:52:29.902669+0000","last_clean":"2024-03-24T12:52:29.902669+0000","last_became_active":"2024-03-24T12:50:52.890546+0000","last_became_peered":"2024-03-24T12:50:52.890546+0000","last_unstale":"2024-03-24T12:52:29.902669+0000","last_undegraded":"2024-03-24T12:52:29.902669+0000","last_fullsized":"2024-03-24T12:52:29.902669+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:52:06.822827+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T12:50:36.279886+0000","last_clean_scrub_stamp":"2024-03-24T12:52:06.822827+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:25.887334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":931,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903516+0000","last_change":"2024-03-24T12:51:53.932874+0000","last_active":"2024-03-24T12:52:29.903516+0000","last_peered":"2024-03-24T12:52:29.903516+0000","last_clean":"2024-03-24T12:52:29.903516+0000","last_became_active":"2024-03-24T12:50:27.795417+0000","last_became_peered":"2024-03-24T12:50:27.795417+0000","last_unstale":"2024-03-24T12:52:29.903516+0000","last_undegraded":"2024-03-24T12:52:29.903516+0000","last_fullsized":"2024-03-24T12:52:29.903516+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:53.932765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:17.105384+0000","last_clean_scrub_stamp":"2024-03-24T12:51:53.932765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:21.746857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"445'46","reported_seq":151,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835909+0000","last_change":"2024-03-24T12:52:23.860497+0000","last_active":"2024-03-24T12:52:29.835909+0000","last_peered":"2024-03-24T12:52:29.835909+0000","last_clean":"2024-03-24T12:52:29.835909+0000","last_became_active":"2024-03-24T12:50:31.822980+0000","last_became_peered":"2024-03-24T12:50:31.822980+0000","last_unstale":"2024-03-24T12:52:29.835909+0000","last_undegraded":"2024-03-24T12:52:29.835909+0000","last_fullsized":"2024-03-24T12:52:29.835909+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'46","last_scrub_stamp":"2024-03-24T12:52:23.860377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:52:23.860377+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:36.611967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903472+0000","last_change":"2024-03-24T12:51:09.943444+0000","last_active":"2024-03-24T12:52:29.903472+0000","last_peered":"2024-03-24T12:52:29.903472+0000","last_clean":"2024-03-24T12:52:29.903472+0000","last_became_active":"2024-03-24T12:51:04.943402+0000","last_became_peered":"2024-03-24T12:51:04.943402+0000","last_unstale":"2024-03-24T12:52:29.903472+0000","last_undegraded":"2024-03-24T12:52:29.903472+0000","last_fullsized":"2024-03-24T12:52:29.903472+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:09.943351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:09.943351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:35.556673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1268,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903315+0000","last_change":"2024-03-24T12:51:26.773048+0000","last_active":"2024-03-24T12:52:29.903315+0000","last_peered":"2024-03-24T12:52:29.903315+0000","last_clean":"2024-03-24T12:52:29.903315+0000","last_became_active":"2024-03-24T12:50:34.825664+0000","last_became_peered":"2024-03-24T12:50:34.825664+0000","last_unstale":"2024-03-24T12:52:29.903315+0000","last_undegraded":"2024-03-24T12:52:29.903315+0000","last_fullsized":"2024-03-24T12:52:29.903315+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:26.772939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:51:26.772939+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:33.216983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":117,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904722+0000","last_change":"2024-03-24T12:52:14.219048+0000","last_active":"2024-03-24T12:52:29.904722+0000","last_peered":"2024-03-24T12:52:29.904722+0000","last_clean":"2024-03-24T12:52:29.904722+0000","last_became_active":"2024-03-24T12:50:27.789008+0000","last_became_peered":"2024-03-24T12:50:27.789008+0000","last_unstale":"2024-03-24T12:52:29.904722+0000","last_undegraded":"2024-03-24T12:52:29.904722+0000","last_fullsized":"2024-03-24T12:52:29.904722+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:14.218930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:37.376732+0000","last_clean_scrub_stamp":"2024-03-24T12:52:14.218930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:33.319542+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":938,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473672+0000","last_change":"2024-03-24T12:51:50.930206+0000","last_active":"2024-03-24T12:52:29.473672+0000","last_peered":"2024-03-24T12:52:29.473672+0000","last_clean":"2024-03-24T12:52:29.473672+0000","last_became_active":"2024-03-24T12:50:27.805167+0000","last_became_peered":"2024-03-24T12:50:27.805167+0000","last_unstale":"2024-03-24T12:52:29.473672+0000","last_undegraded":"2024-03-24T12:52:29.473672+0000","last_fullsized":"2024-03-24T12:52:29.473672+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:50.930107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:50.930107+0000","last_clean_scrub_stamp":"2024-03-24T12:51:50.930107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:52.705763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"390'26","reported_seq":1019,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903059+0000","last_change":"2024-03-24T12:51:35.889121+0000","last_active":"2024-03-24T12:52:29.903059+0000","last_peered":"2024-03-24T12:52:29.903059+0000","last_clean":"2024-03-24T12:52:29.903059+0000","last_became_active":"2024-03-24T12:50:27.806165+0000","last_became_peered":"2024-03-24T12:50:27.806165+0000","last_unstale":"2024-03-24T12:52:29.903059+0000","last_undegraded":"2024-03-24T12:52:29.903059+0000","last_fullsized":"2024-03-24T12:52:29.903059+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'26","last_scrub_stamp":"2024-03-24T12:51:35.889000+0000","last_deep_scrub":"207'22","last_deep_scrub_stamp":"2024-03-24T12:44:50.024489+0000","last_clean_scrub_stamp":"2024-03-24T12:51:35.889000+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:44.894998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":991,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472549+0000","last_change":"2024-03-24T12:51:41.986847+0000","last_active":"2024-03-24T12:52:29.472549+0000","last_peered":"2024-03-24T12:52:29.472549+0000","last_clean":"2024-03-24T12:52:29.472549+0000","last_became_active":"2024-03-24T12:51:04.945086+0000","last_became_peered":"2024-03-24T12:51:04.945086+0000","last_unstale":"2024-03-24T12:52:29.472549+0000","last_undegraded":"2024-03-24T12:52:29.472549+0000","last_fullsized":"2024-03-24T12:52:29.472549+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:41.986754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:41.986754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:58.940084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":947,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902751+0000","last_change":"2024-03-24T12:51:20.006047+0000","last_active":"2024-03-24T12:52:29.902751+0000","last_peered":"2024-03-24T12:52:29.902751+0000","last_clean":"2024-03-24T12:52:29.902751+0000","last_became_active":"2024-03-24T12:50:52.889431+0000","last_became_peered":"2024-03-24T12:50:52.889431+0000","last_unstale":"2024-03-24T12:52:29.902751+0000","last_undegraded":"2024-03-24T12:52:29.902751+0000","last_fullsized":"2024-03-24T12:52:29.902751+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:20.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:55.977877+0000","last_clean_scrub_stamp":"2024-03-24T12:51:20.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:39.280301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":975,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904095+0000","last_change":"2024-03-24T12:52:12.776812+0000","last_active":"2024-03-24T12:52:29.904095+0000","last_peered":"2024-03-24T12:52:29.904095+0000","last_clean":"2024-03-24T12:52:29.904095+0000","last_became_active":"2024-03-24T12:50:31.827081+0000","last_became_peered":"2024-03-24T12:50:31.827081+0000","last_unstale":"2024-03-24T12:52:29.904095+0000","last_undegraded":"2024-03-24T12:52:29.904095+0000","last_fullsized":"2024-03-24T12:52:29.904095+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:12.776703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:15.087115+0000","last_clean_scrub_stamp":"2024-03-24T12:52:12.776703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:32.373142+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":625,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903729+0000","last_change":"2024-03-24T12:51:32.333250+0000","last_active":"2024-03-24T12:52:29.903729+0000","last_peered":"2024-03-24T12:52:29.903729+0000","last_clean":"2024-03-24T12:52:29.903729+0000","last_became_active":"2024-03-24T12:51:32.332849+0000","last_became_peered":"2024-03-24T12:51:32.332849+0000","last_unstale":"2024-03-24T12:52:29.903729+0000","last_undegraded":"2024-03-24T12:52:29.903729+0000","last_fullsized":"2024-03-24T12:52:29.903729+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:19.670406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:51:19.670406+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":853,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473299+0000","last_change":"2024-03-24T12:51:35.648398+0000","last_active":"2024-03-24T12:52:29.473299+0000","last_peered":"2024-03-24T12:52:29.473299+0000","last_clean":"2024-03-24T12:52:29.473299+0000","last_became_active":"2024-03-24T12:49:54.344124+0000","last_became_peered":"2024-03-24T12:49:54.344124+0000","last_unstale":"2024-03-24T12:52:29.473299+0000","last_undegraded":"2024-03-24T12:52:29.473299+0000","last_fullsized":"2024-03-24T12:52:29.473299+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:35.648299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:35.648299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:39.110905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"459'67","reported_seq":1114,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903454+0000","last_change":"2024-03-24T12:52:01.846412+0000","last_active":"2024-03-24T12:52:29.903454+0000","last_peered":"2024-03-24T12:52:29.903454+0000","last_clean":"2024-03-24T12:52:29.903454+0000","last_became_active":"2024-03-24T12:51:04.944772+0000","last_became_peered":"2024-03-24T12:51:04.944772+0000","last_unstale":"2024-03-24T12:52:29.903454+0000","last_undegraded":"2024-03-24T12:52:29.903454+0000","last_fullsized":"2024-03-24T12:52:29.903454+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'67","last_scrub_stamp":"2024-03-24T12:52:01.846244+0000","last_deep_scrub":"248'51","last_deep_scrub_stamp":"2024-03-24T12:39:47.606538+0000","last_clean_scrub_stamp":"2024-03-24T12:52:01.846244+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:13.523680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":980,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473117+0000","last_change":"2024-03-24T12:52:18.139530+0000","last_active":"2024-03-24T12:52:29.473117+0000","last_peered":"2024-03-24T12:52:29.473117+0000","last_clean":"2024-03-24T12:52:29.473117+0000","last_became_active":"2024-03-24T12:50:31.815376+0000","last_became_peered":"2024-03-24T12:50:31.815376+0000","last_unstale":"2024-03-24T12:52:29.473117+0000","last_undegraded":"2024-03-24T12:52:29.473117+0000","last_fullsized":"2024-03-24T12:52:29.473117+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:18.139435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:52:18.139435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:31.841962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":131,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902924+0000","last_change":"2024-03-24T12:52:18.111218+0000","last_active":"2024-03-24T12:52:29.902924+0000","last_peered":"2024-03-24T12:52:29.902924+0000","last_clean":"2024-03-24T12:52:29.902924+0000","last_became_active":"2024-03-24T12:50:31.819931+0000","last_became_peered":"2024-03-24T12:50:31.819931+0000","last_unstale":"2024-03-24T12:52:29.902924+0000","last_undegraded":"2024-03-24T12:52:29.902924+0000","last_fullsized":"2024-03-24T12:52:29.902924+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:52:18.111114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:52:18.111114+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:24.490924+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":85,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903156+0000","last_change":"2024-03-24T12:52:25.148398+0000","last_active":"2024-03-24T12:52:29.903156+0000","last_peered":"2024-03-24T12:52:29.903156+0000","last_clean":"2024-03-24T12:52:29.903156+0000","last_became_active":"2024-03-24T12:51:04.947211+0000","last_became_peered":"2024-03-24T12:51:04.947211+0000","last_unstale":"2024-03-24T12:52:29.903156+0000","last_undegraded":"2024-03-24T12:52:29.903156+0000","last_fullsized":"2024-03-24T12:52:29.903156+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:25.148294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:25.148294+0000","last_clean_scrub_stamp":"2024-03-24T12:52:25.148294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:49.124103+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":667,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902821+0000","last_change":"2024-03-24T12:51:32.170253+0000","last_active":"2024-03-24T12:52:29.902821+0000","last_peered":"2024-03-24T12:52:29.902821+0000","last_clean":"2024-03-24T12:52:29.902821+0000","last_became_active":"2024-03-24T12:50:30.800045+0000","last_became_peered":"2024-03-24T12:50:30.800045+0000","last_unstale":"2024-03-24T12:52:29.902821+0000","last_undegraded":"2024-03-24T12:52:29.902821+0000","last_fullsized":"2024-03-24T12:52:29.902821+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:32.170145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:51:32.170145+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:56.010658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"390'21","reported_seq":907,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474064+0000","last_change":"2024-03-24T12:51:21.643588+0000","last_active":"2024-03-24T12:52:29.474064+0000","last_peered":"2024-03-24T12:52:29.474064+0000","last_clean":"2024-03-24T12:52:29.474064+0000","last_became_active":"2024-03-24T12:49:54.357030+0000","last_became_peered":"2024-03-24T12:49:54.357030+0000","last_unstale":"2024-03-24T12:52:29.474064+0000","last_undegraded":"2024-03-24T12:52:29.474064+0000","last_fullsized":"2024-03-24T12:52:29.474064+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:51:21.643496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:21.643496+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:40.112167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":133,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903523+0000","last_change":"2024-03-24T12:51:36.185698+0000","last_active":"2024-03-24T12:52:29.903523+0000","last_peered":"2024-03-24T12:52:29.903523+0000","last_clean":"2024-03-24T12:52:29.903523+0000","last_became_active":"2024-03-24T12:51:04.947379+0000","last_became_peered":"2024-03-24T12:51:04.947379+0000","last_unstale":"2024-03-24T12:52:29.903523+0000","last_undegraded":"2024-03-24T12:52:29.903523+0000","last_fullsized":"2024-03-24T12:52:29.903523+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:51:36.185568+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:51:36.185568+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:58.237401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":968,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472920+0000","last_change":"2024-03-24T12:51:22.009451+0000","last_active":"2024-03-24T12:52:29.472920+0000","last_peered":"2024-03-24T12:52:29.472920+0000","last_clean":"2024-03-24T12:52:29.472920+0000","last_became_active":"2024-03-24T12:51:04.945405+0000","last_became_peered":"2024-03-24T12:51:04.945405+0000","last_unstale":"2024-03-24T12:52:29.472920+0000","last_undegraded":"2024-03-24T12:52:29.472920+0000","last_fullsized":"2024-03-24T12:52:29.472920+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:22.009361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:11.890544+0000","last_clean_scrub_stamp":"2024-03-24T12:51:22.009361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:45.666922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":859,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472596+0000","last_change":"2024-03-24T12:51:54.676419+0000","last_active":"2024-03-24T12:52:29.472596+0000","last_peered":"2024-03-24T12:52:29.472596+0000","last_clean":"2024-03-24T12:52:29.472596+0000","last_became_active":"2024-03-24T12:49:54.344376+0000","last_became_peered":"2024-03-24T12:49:54.344376+0000","last_unstale":"2024-03-24T12:52:29.472596+0000","last_undegraded":"2024-03-24T12:52:29.472596+0000","last_fullsized":"2024-03-24T12:52:29.472596+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:54.676248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:31.678822+0000","last_clean_scrub_stamp":"2024-03-24T12:51:54.676248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:18.893396+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"391'51","reported_seq":120,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835480+0000","last_change":"2024-03-24T12:51:33.431445+0000","last_active":"2024-03-24T12:52:29.835480+0000","last_peered":"2024-03-24T12:52:29.835480+0000","last_clean":"2024-03-24T12:52:29.835480+0000","last_became_active":"2024-03-24T12:50:31.824015+0000","last_became_peered":"2024-03-24T12:50:31.824015+0000","last_unstale":"2024-03-24T12:52:29.835480+0000","last_undegraded":"2024-03-24T12:52:29.835480+0000","last_fullsized":"2024-03-24T12:52:29.835480+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'51","last_scrub_stamp":"2024-03-24T12:51:33.431269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:33.431269+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:46.626660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":982,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903895+0000","last_change":"2024-03-24T12:51:25.784512+0000","last_active":"2024-03-24T12:52:29.903895+0000","last_peered":"2024-03-24T12:52:29.903895+0000","last_clean":"2024-03-24T12:52:29.903895+0000","last_became_active":"2024-03-24T12:50:31.831092+0000","last_became_peered":"2024-03-24T12:50:31.831092+0000","last_unstale":"2024-03-24T12:52:29.903895+0000","last_undegraded":"2024-03-24T12:52:29.903895+0000","last_fullsized":"2024-03-24T12:52:29.903895+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:25.784411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:51:25.784411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:46.642101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":938,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473258+0000","last_change":"2024-03-24T12:51:21.043249+0000","last_active":"2024-03-24T12:52:29.473258+0000","last_peered":"2024-03-24T12:52:29.473258+0000","last_clean":"2024-03-24T12:52:29.473258+0000","last_became_active":"2024-03-24T12:50:27.803159+0000","last_became_peered":"2024-03-24T12:50:27.803159+0000","last_unstale":"2024-03-24T12:52:29.473258+0000","last_undegraded":"2024-03-24T12:52:29.473258+0000","last_fullsized":"2024-03-24T12:52:29.473258+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:21.043170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:21.043170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.936793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":943,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903059+0000","last_change":"2024-03-24T12:52:26.913509+0000","last_active":"2024-03-24T12:52:29.903059+0000","last_peered":"2024-03-24T12:52:29.903059+0000","last_clean":"2024-03-24T12:52:29.903059+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:52:29.903059+0000","last_undegraded":"2024-03-24T12:52:29.903059+0000","last_fullsized":"2024-03-24T12:52:29.903059+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:26.913402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:26.913402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:32.403316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":970,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472164+0000","last_change":"2024-03-24T12:52:29.052546+0000","last_active":"2024-03-24T12:52:29.472164+0000","last_peered":"2024-03-24T12:52:29.472164+0000","last_clean":"2024-03-24T12:52:29.472164+0000","last_became_active":"2024-03-24T12:50:49.121399+0000","last_became_peered":"2024-03-24T12:50:49.121399+0000","last_unstale":"2024-03-24T12:52:29.472164+0000","last_undegraded":"2024-03-24T12:52:29.472164+0000","last_fullsized":"2024-03-24T12:52:29.472164+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:29.052442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:52:29.052442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:48.858312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"390'29","reported_seq":1052,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473211+0000","last_change":"2024-03-24T12:51:29.219473+0000","last_active":"2024-03-24T12:52:29.473211+0000","last_peered":"2024-03-24T12:52:29.473211+0000","last_clean":"2024-03-24T12:52:29.473211+0000","last_became_active":"2024-03-24T12:51:04.951647+0000","last_became_peered":"2024-03-24T12:51:04.951647+0000","last_unstale":"2024-03-24T12:52:29.473211+0000","last_undegraded":"2024-03-24T12:52:29.473211+0000","last_fullsized":"2024-03-24T12:52:29.473211+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'29","last_scrub_stamp":"2024-03-24T12:51:29.219377+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:51:29.219377+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:31.668781+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":946,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904272+0000","last_change":"2024-03-24T12:52:25.912332+0000","last_active":"2024-03-24T12:52:29.904272+0000","last_peered":"2024-03-24T12:52:29.904272+0000","last_clean":"2024-03-24T12:52:29.904272+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:52:29.904272+0000","last_undegraded":"2024-03-24T12:52:29.904272+0000","last_fullsized":"2024-03-24T12:52:29.904272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:25.912195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:25.912195+0000","last_clean_scrub_stamp":"2024-03-24T12:52:25.912195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:51.098004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":950,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904121+0000","last_change":"2024-03-24T12:52:16.275082+0000","last_active":"2024-03-24T12:52:29.904121+0000","last_peered":"2024-03-24T12:52:29.904121+0000","last_clean":"2024-03-24T12:52:29.904121+0000","last_became_active":"2024-03-24T12:51:29.010623+0000","last_became_peered":"2024-03-24T12:51:29.010623+0000","last_unstale":"2024-03-24T12:52:29.904121+0000","last_undegraded":"2024-03-24T12:52:29.904121+0000","last_fullsized":"2024-03-24T12:52:29.904121+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:16.274967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:28.075823+0000","last_clean_scrub_stamp":"2024-03-24T12:52:16.274967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:44.153593+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"445'13","reported_seq":1317,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903748+0000","last_change":"2024-03-24T12:52:13.758958+0000","last_active":"2024-03-24T12:52:29.903748+0000","last_peered":"2024-03-24T12:52:29.903748+0000","last_clean":"2024-03-24T12:52:29.903748+0000","last_became_active":"2024-03-24T12:50:31.835895+0000","last_became_peered":"2024-03-24T12:50:31.835895+0000","last_unstale":"2024-03-24T12:52:29.903748+0000","last_undegraded":"2024-03-24T12:52:29.903748+0000","last_fullsized":"2024-03-24T12:52:29.903748+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'13","last_scrub_stamp":"2024-03-24T12:52:13.758834+0000","last_deep_scrub":"417'12","last_deep_scrub_stamp":"2024-03-24T12:48:19.860403+0000","last_clean_scrub_stamp":"2024-03-24T12:52:13.758834+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:17.767324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"390'39","reported_seq":177,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902695+0000","last_change":"2024-03-24T12:51:57.154175+0000","last_active":"2024-03-24T12:52:29.902695+0000","last_peered":"2024-03-24T12:52:29.902695+0000","last_clean":"2024-03-24T12:52:29.902695+0000","last_became_active":"2024-03-24T12:51:29.011314+0000","last_became_peered":"2024-03-24T12:51:29.011314+0000","last_unstale":"2024-03-24T12:52:29.902695+0000","last_undegraded":"2024-03-24T12:52:29.902695+0000","last_fullsized":"2024-03-24T12:52:29.902695+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'39","last_scrub_stamp":"2024-03-24T12:51:57.154010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:57.154010+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:06.965895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":959,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904502+0000","last_change":"2024-03-24T12:51:20.818276+0000","last_active":"2024-03-24T12:52:29.904502+0000","last_peered":"2024-03-24T12:52:29.904502+0000","last_clean":"2024-03-24T12:52:29.904502+0000","last_became_active":"2024-03-24T12:49:54.336098+0000","last_became_peered":"2024-03-24T12:49:54.336098+0000","last_unstale":"2024-03-24T12:52:29.904502+0000","last_undegraded":"2024-03-24T12:52:29.904502+0000","last_fullsized":"2024-03-24T12:52:29.904502+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:51:20.818126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:20.818126+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:30.115519+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":115,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902921+0000","last_change":"2024-03-24T12:51:22.350188+0000","last_active":"2024-03-24T12:52:29.902921+0000","last_peered":"2024-03-24T12:52:29.902921+0000","last_clean":"2024-03-24T12:52:29.902921+0000","last_became_active":"2024-03-24T12:50:31.819654+0000","last_became_peered":"2024-03-24T12:50:31.819654+0000","last_unstale":"2024-03-24T12:52:29.902921+0000","last_undegraded":"2024-03-24T12:52:29.902921+0000","last_fullsized":"2024-03-24T12:52:29.902921+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:22.350074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:07.478619+0000","last_clean_scrub_stamp":"2024-03-24T12:51:22.350074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:34.699867+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"390'18","reported_seq":99,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904289+0000","last_change":"2024-03-24T12:51:29.025082+0000","last_active":"2024-03-24T12:52:29.904289+0000","last_peered":"2024-03-24T12:52:29.904289+0000","last_clean":"2024-03-24T12:52:29.904289+0000","last_became_active":"2024-03-24T12:51:29.010696+0000","last_became_peered":"2024-03-24T12:51:29.010696+0000","last_unstale":"2024-03-24T12:52:29.904289+0000","last_undegraded":"2024-03-24T12:52:29.904289+0000","last_fullsized":"2024-03-24T12:52:29.904289+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'18","last_scrub_stamp":"2024-03-24T12:51:13.967634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:13.967634+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":100,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835405+0000","last_change":"2024-03-24T12:51:25.514014+0000","last_active":"2024-03-24T12:52:29.835405+0000","last_peered":"2024-03-24T12:52:29.835405+0000","last_clean":"2024-03-24T12:52:29.835405+0000","last_became_active":"2024-03-24T12:50:31.818929+0000","last_became_peered":"2024-03-24T12:50:31.818929+0000","last_unstale":"2024-03-24T12:52:29.835405+0000","last_undegraded":"2024-03-24T12:52:29.835405+0000","last_fullsized":"2024-03-24T12:52:29.835405+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:25.513910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:25.513910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:53.719018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":121,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903924+0000","last_change":"2024-03-24T12:51:47.154108+0000","last_active":"2024-03-24T12:52:29.903924+0000","last_peered":"2024-03-24T12:52:29.903924+0000","last_clean":"2024-03-24T12:52:29.903924+0000","last_became_active":"2024-03-24T12:51:04.946383+0000","last_became_peered":"2024-03-24T12:51:04.946383+0000","last_unstale":"2024-03-24T12:52:29.903924+0000","last_undegraded":"2024-03-24T12:52:29.903924+0000","last_fullsized":"2024-03-24T12:52:29.903924+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:47.153980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:47.153980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:05.489680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":962,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903181+0000","last_change":"2024-03-24T12:51:22.916093+0000","last_active":"2024-03-24T12:52:29.903181+0000","last_peered":"2024-03-24T12:52:29.903181+0000","last_clean":"2024-03-24T12:52:29.903181+0000","last_became_active":"2024-03-24T12:50:31.830543+0000","last_became_peered":"2024-03-24T12:50:31.830543+0000","last_unstale":"2024-03-24T12:52:29.903181+0000","last_undegraded":"2024-03-24T12:52:29.903181+0000","last_fullsized":"2024-03-24T12:52:29.903181+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:22.915989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:55.271679+0000","last_clean_scrub_stamp":"2024-03-24T12:51:22.915989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"390'21","reported_seq":1056,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474280+0000","last_change":"2024-03-24T12:52:00.557559+0000","last_active":"2024-03-24T12:52:29.474280+0000","last_peered":"2024-03-24T12:52:29.474280+0000","last_clean":"2024-03-24T12:52:29.474280+0000","last_became_active":"2024-03-24T12:51:29.009880+0000","last_became_peered":"2024-03-24T12:51:29.009880+0000","last_unstale":"2024-03-24T12:52:29.474280+0000","last_undegraded":"2024-03-24T12:52:29.474280+0000","last_fullsized":"2024-03-24T12:52:29.474280+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:52:00.557442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:52:00.557442+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:22.074975+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":988,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903517+0000","last_change":"2024-03-24T12:52:24.884788+0000","last_active":"2024-03-24T12:52:29.903517+0000","last_peered":"2024-03-24T12:52:29.903517+0000","last_clean":"2024-03-24T12:52:29.903517+0000","last_became_active":"2024-03-24T12:50:31.830406+0000","last_became_peered":"2024-03-24T12:50:31.830406+0000","last_unstale":"2024-03-24T12:52:29.903517+0000","last_undegraded":"2024-03-24T12:52:29.903517+0000","last_fullsized":"2024-03-24T12:52:29.903517+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:24.884685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:59.056269+0000","last_clean_scrub_stamp":"2024-03-24T12:52:24.884685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:53.188835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904632+0000","last_change":"2024-03-24T12:51:31.110791+0000","last_active":"2024-03-24T12:52:29.904632+0000","last_peered":"2024-03-24T12:52:29.904632+0000","last_clean":"2024-03-24T12:52:29.904632+0000","last_became_active":"2024-03-24T12:51:04.951872+0000","last_became_peered":"2024-03-24T12:51:04.951872+0000","last_unstale":"2024-03-24T12:52:29.904632+0000","last_undegraded":"2024-03-24T12:52:29.904632+0000","last_fullsized":"2024-03-24T12:52:29.904632+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:31.110702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:31.110702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:41.855777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":958,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473955+0000","last_change":"2024-03-24T12:52:01.809237+0000","last_active":"2024-03-24T12:52:29.473955+0000","last_peered":"2024-03-24T12:52:29.473955+0000","last_clean":"2024-03-24T12:52:29.473955+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:52:29.473955+0000","last_undegraded":"2024-03-24T12:52:29.473955+0000","last_fullsized":"2024-03-24T12:52:29.473955+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:01.809149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:52:01.809149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:04.495865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"390'25","reported_seq":119,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904371+0000","last_change":"2024-03-24T12:51:37.172266+0000","last_active":"2024-03-24T12:52:29.904371+0000","last_peered":"2024-03-24T12:52:29.904371+0000","last_clean":"2024-03-24T12:52:29.904371+0000","last_became_active":"2024-03-24T12:50:27.797323+0000","last_became_peered":"2024-03-24T12:50:27.797323+0000","last_unstale":"2024-03-24T12:52:29.904371+0000","last_undegraded":"2024-03-24T12:52:29.904371+0000","last_fullsized":"2024-03-24T12:52:29.904371+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'25","last_scrub_stamp":"2024-03-24T12:51:37.172138+0000","last_deep_scrub":"390'25","last_deep_scrub_stamp":"2024-03-24T12:50:13.115092+0000","last_clean_scrub_stamp":"2024-03-24T12:51:37.172138+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:00.476611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":923,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473355+0000","last_change":"2024-03-24T12:52:06.512987+0000","last_active":"2024-03-24T12:52:29.473355+0000","last_peered":"2024-03-24T12:52:29.473355+0000","last_clean":"2024-03-24T12:52:29.473355+0000","last_became_active":"2024-03-24T12:51:04.939149+0000","last_became_peered":"2024-03-24T12:51:04.939149+0000","last_unstale":"2024-03-24T12:52:29.473355+0000","last_undegraded":"2024-03-24T12:52:29.473355+0000","last_fullsized":"2024-03-24T12:52:29.473355+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:06.512843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:06.512843+0000","last_clean_scrub_stamp":"2024-03-24T12:52:06.512843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:08.997175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"459'11","reported_seq":98,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904493+0000","last_change":"2024-03-24T12:51:40.175588+0000","last_active":"2024-03-24T12:52:29.904493+0000","last_peered":"2024-03-24T12:52:29.904493+0000","last_clean":"2024-03-24T12:52:29.904493+0000","last_became_active":"2024-03-24T12:51:04.947498+0000","last_became_peered":"2024-03-24T12:51:04.947498+0000","last_unstale":"2024-03-24T12:52:29.904493+0000","last_undegraded":"2024-03-24T12:52:29.904493+0000","last_fullsized":"2024-03-24T12:52:29.904493+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'11","last_scrub_stamp":"2024-03-24T12:51:40.175438+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:51:40.175438+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:05.098888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":934,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473885+0000","last_change":"2024-03-24T12:52:20.325148+0000","last_active":"2024-03-24T12:52:29.473885+0000","last_peered":"2024-03-24T12:52:29.473885+0000","last_clean":"2024-03-24T12:52:29.473885+0000","last_became_active":"2024-03-24T12:50:27.790028+0000","last_became_peered":"2024-03-24T12:50:27.790028+0000","last_unstale":"2024-03-24T12:52:29.473885+0000","last_undegraded":"2024-03-24T12:52:29.473885+0000","last_fullsized":"2024-03-24T12:52:29.473885+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:20.325040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:20.325040+0000","last_clean_scrub_stamp":"2024-03-24T12:52:20.325040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:33.789908+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"480'35","reported_seq":992,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904595+0000","last_change":"2024-03-24T12:52:13.153802+0000","last_active":"2024-03-24T12:52:29.904595+0000","last_peered":"2024-03-24T12:52:29.904595+0000","last_clean":"2024-03-24T12:52:29.904595+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:52:29.904595+0000","last_undegraded":"2024-03-24T12:52:29.904595+0000","last_fullsized":"2024-03-24T12:52:29.904595+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'35","last_scrub_stamp":"2024-03-24T12:52:13.153700+0000","last_deep_scrub":"480'35","last_deep_scrub_stamp":"2024-03-24T12:52:13.153700+0000","last_clean_scrub_stamp":"2024-03-24T12:52:13.153700+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:26.584964+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":949,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904053+0000","last_change":"2024-03-24T12:52:07.808986+0000","last_active":"2024-03-24T12:52:29.904053+0000","last_peered":"2024-03-24T12:52:29.904053+0000","last_clean":"2024-03-24T12:52:29.904053+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:52:29.904053+0000","last_undegraded":"2024-03-24T12:52:29.904053+0000","last_fullsized":"2024-03-24T12:52:29.904053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:07.808884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:07.808884+0000","last_clean_scrub_stamp":"2024-03-24T12:52:07.808884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:20.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"440'14","reported_seq":147,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904602+0000","last_change":"2024-03-24T12:52:05.185347+0000","last_active":"2024-03-24T12:52:29.904602+0000","last_peered":"2024-03-24T12:52:29.904602+0000","last_clean":"2024-03-24T12:52:29.904602+0000","last_became_active":"2024-03-24T12:50:27.804040+0000","last_became_peered":"2024-03-24T12:50:27.804040+0000","last_unstale":"2024-03-24T12:52:29.904602+0000","last_undegraded":"2024-03-24T12:52:29.904602+0000","last_fullsized":"2024-03-24T12:52:29.904602+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'14","last_scrub_stamp":"2024-03-24T12:52:05.185248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:52:05.185248+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:12.355695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":873,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474350+0000","last_change":"2024-03-24T12:52:15.419360+0000","last_active":"2024-03-24T12:52:29.474350+0000","last_peered":"2024-03-24T12:52:29.474350+0000","last_clean":"2024-03-24T12:52:29.474350+0000","last_became_active":"2024-03-24T12:51:04.939419+0000","last_became_peered":"2024-03-24T12:51:04.939419+0000","last_unstale":"2024-03-24T12:52:29.474350+0000","last_undegraded":"2024-03-24T12:52:29.474350+0000","last_fullsized":"2024-03-24T12:52:29.474350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:52:15.419178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:15.419178+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:33.212605+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"390'32","reported_seq":1070,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473389+0000","last_change":"2024-03-24T12:52:28.232357+0000","last_active":"2024-03-24T12:52:29.473389+0000","last_peered":"2024-03-24T12:52:29.473389+0000","last_clean":"2024-03-24T12:52:29.473389+0000","last_became_active":"2024-03-24T12:50:31.824602+0000","last_became_peered":"2024-03-24T12:50:31.824602+0000","last_unstale":"2024-03-24T12:52:29.473389+0000","last_undegraded":"2024-03-24T12:52:29.473389+0000","last_fullsized":"2024-03-24T12:52:29.473389+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'32","last_scrub_stamp":"2024-03-24T12:52:28.232237+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:52:28.232237+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:40.554584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":970,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473428+0000","last_change":"2024-03-24T12:51:24.144859+0000","last_active":"2024-03-24T12:52:29.473428+0000","last_peered":"2024-03-24T12:52:29.473428+0000","last_clean":"2024-03-24T12:52:29.473428+0000","last_became_active":"2024-03-24T12:50:31.813180+0000","last_became_peered":"2024-03-24T12:50:31.813180+0000","last_unstale":"2024-03-24T12:52:29.473428+0000","last_undegraded":"2024-03-24T12:52:29.473428+0000","last_fullsized":"2024-03-24T12:52:29.473428+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:24.144684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:32.853388+0000","last_clean_scrub_stamp":"2024-03-24T12:51:24.144684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:52.372455+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":957,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903610+0000","last_change":"2024-03-24T12:52:29.876547+0000","last_active":"2024-03-24T12:52:29.903610+0000","last_peered":"2024-03-24T12:52:29.903610+0000","last_clean":"2024-03-24T12:52:29.903610+0000","last_became_active":"2024-03-24T12:51:04.939496+0000","last_became_peered":"2024-03-24T12:51:04.939496+0000","last_unstale":"2024-03-24T12:52:29.903610+0000","last_undegraded":"2024-03-24T12:52:29.903610+0000","last_fullsized":"2024-03-24T12:52:29.903610+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:29.876464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:52:29.876464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:44.738196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903907+0000","last_change":"2024-03-24T12:51:41.197640+0000","last_active":"2024-03-24T12:52:29.903907+0000","last_peered":"2024-03-24T12:52:29.903907+0000","last_clean":"2024-03-24T12:52:29.903907+0000","last_became_active":"2024-03-24T12:51:04.947440+0000","last_became_peered":"2024-03-24T12:51:04.947440+0000","last_unstale":"2024-03-24T12:52:29.903907+0000","last_undegraded":"2024-03-24T12:52:29.903907+0000","last_fullsized":"2024-03-24T12:52:29.903907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:41.197541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:41.197541+0000","last_clean_scrub_stamp":"2024-03-24T12:51:41.197541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:01.047569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"390'21","reported_seq":366,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902804+0000","last_change":"2024-03-24T12:51:24.020873+0000","last_active":"2024-03-24T12:52:29.902804+0000","last_peered":"2024-03-24T12:52:29.902804+0000","last_clean":"2024-03-24T12:52:29.902804+0000","last_became_active":"2024-03-24T12:50:31.811177+0000","last_became_peered":"2024-03-24T12:50:31.811177+0000","last_unstale":"2024-03-24T12:52:29.902804+0000","last_undegraded":"2024-03-24T12:52:29.902804+0000","last_fullsized":"2024-03-24T12:52:29.902804+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:51:24.020758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:24.020758+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:49.568763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":953,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903305+0000","last_change":"2024-03-24T12:52:20.809516+0000","last_active":"2024-03-24T12:52:29.903305+0000","last_peered":"2024-03-24T12:52:29.903305+0000","last_clean":"2024-03-24T12:52:29.903305+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:52:29.903305+0000","last_undegraded":"2024-03-24T12:52:29.903305+0000","last_fullsized":"2024-03-24T12:52:29.903305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:20.809395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:28.831117+0000","last_clean_scrub_stamp":"2024-03-24T12:52:20.809395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:29.857567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":939,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903976+0000","last_change":"2024-03-24T12:52:10.089536+0000","last_active":"2024-03-24T12:52:29.903976+0000","last_peered":"2024-03-24T12:52:29.903976+0000","last_clean":"2024-03-24T12:52:29.903976+0000","last_became_active":"2024-03-24T12:49:54.340307+0000","last_became_peered":"2024-03-24T12:49:54.340307+0000","last_unstale":"2024-03-24T12:52:29.903976+0000","last_undegraded":"2024-03-24T12:52:29.903976+0000","last_fullsized":"2024-03-24T12:52:29.903976+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:10.089444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:28.070148+0000","last_clean_scrub_stamp":"2024-03-24T12:52:10.089444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:27.544845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":857,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472597+0000","last_change":"2024-03-24T12:51:40.601123+0000","last_active":"2024-03-24T12:52:29.472597+0000","last_peered":"2024-03-24T12:52:29.472597+0000","last_clean":"2024-03-24T12:52:29.472597+0000","last_became_active":"2024-03-24T12:50:27.812903+0000","last_became_peered":"2024-03-24T12:50:27.812903+0000","last_unstale":"2024-03-24T12:52:29.472597+0000","last_undegraded":"2024-03-24T12:52:29.472597+0000","last_fullsized":"2024-03-24T12:52:29.472597+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:40.601022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:51:40.601022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:53.339776+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"390'27","reported_seq":1054,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904152+0000","last_change":"2024-03-24T12:51:30.121983+0000","last_active":"2024-03-24T12:52:29.904152+0000","last_peered":"2024-03-24T12:52:29.904152+0000","last_clean":"2024-03-24T12:52:29.904152+0000","last_became_active":"2024-03-24T12:50:31.810900+0000","last_became_peered":"2024-03-24T12:50:31.810900+0000","last_unstale":"2024-03-24T12:52:29.904152+0000","last_undegraded":"2024-03-24T12:52:29.904152+0000","last_fullsized":"2024-03-24T12:52:29.904152+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'27","last_scrub_stamp":"2024-03-24T12:51:30.121864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:30.121864+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.551603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"427'2","reported_seq":121,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903530+0000","last_change":"2024-03-24T12:52:25.117369+0000","last_active":"2024-03-24T12:52:29.903530+0000","last_peered":"2024-03-24T12:52:29.903530+0000","last_clean":"2024-03-24T12:52:29.903530+0000","last_became_active":"2024-03-24T12:50:27.803506+0000","last_became_peered":"2024-03-24T12:50:27.803506+0000","last_unstale":"2024-03-24T12:52:29.903530+0000","last_undegraded":"2024-03-24T12:52:29.903530+0000","last_fullsized":"2024-03-24T12:52:29.903530+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"427'2","last_scrub_stamp":"2024-03-24T12:52:25.117265+0000","last_deep_scrub":"427'2","last_deep_scrub_stamp":"2024-03-24T12:49:18.871645+0000","last_clean_scrub_stamp":"2024-03-24T12:52:25.117265+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:31.450317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":971,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903817+0000","last_change":"2024-03-24T12:51:52.980296+0000","last_active":"2024-03-24T12:52:29.903817+0000","last_peered":"2024-03-24T12:52:29.903817+0000","last_clean":"2024-03-24T12:52:29.903817+0000","last_became_active":"2024-03-24T12:50:31.827976+0000","last_became_peered":"2024-03-24T12:50:31.827976+0000","last_unstale":"2024-03-24T12:52:29.903817+0000","last_undegraded":"2024-03-24T12:52:29.903817+0000","last_fullsized":"2024-03-24T12:52:29.903817+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:52.980202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:52.980202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:09.905656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903718+0000","last_change":"2024-03-24T12:51:19.089676+0000","last_active":"2024-03-24T12:52:29.903718+0000","last_peered":"2024-03-24T12:52:29.903718+0000","last_clean":"2024-03-24T12:52:29.903718+0000","last_became_active":"2024-03-24T12:51:04.950249+0000","last_became_peered":"2024-03-24T12:51:04.950249+0000","last_unstale":"2024-03-24T12:52:29.903718+0000","last_undegraded":"2024-03-24T12:52:29.903718+0000","last_fullsized":"2024-03-24T12:52:29.903718+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:19.089546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:19.089546+0000","last_clean_scrub_stamp":"2024-03-24T12:51:19.089546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:32.391246+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"390'12","reported_seq":75,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902974+0000","last_change":"2024-03-24T12:51:04.988204+0000","last_active":"2024-03-24T12:52:29.902974+0000","last_peered":"2024-03-24T12:52:29.902974+0000","last_clean":"2024-03-24T12:52:29.902974+0000","last_became_active":"2024-03-24T12:51:04.956111+0000","last_became_peered":"2024-03-24T12:51:04.956111+0000","last_unstale":"2024-03-24T12:52:29.902974+0000","last_undegraded":"2024-03-24T12:52:29.902974+0000","last_fullsized":"2024-03-24T12:52:29.902974+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'12","last_scrub_stamp":"2024-03-24T12:50:58.967629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:50:58.967629+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":915,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473818+0000","last_change":"2024-03-24T12:51:59.523110+0000","last_active":"2024-03-24T12:52:29.473818+0000","last_peered":"2024-03-24T12:52:29.473818+0000","last_clean":"2024-03-24T12:52:29.473818+0000","last_became_active":"2024-03-24T12:51:04.938167+0000","last_became_peered":"2024-03-24T12:51:04.938167+0000","last_unstale":"2024-03-24T12:52:29.473818+0000","last_undegraded":"2024-03-24T12:52:29.473818+0000","last_fullsized":"2024-03-24T12:52:29.473818+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:59.522983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:59.522983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:27.336806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":108,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835359+0000","last_change":"2024-03-24T12:51:52.176344+0000","last_active":"2024-03-24T12:52:29.835359+0000","last_peered":"2024-03-24T12:52:29.835359+0000","last_clean":"2024-03-24T12:52:29.835359+0000","last_became_active":"2024-03-24T12:50:31.824539+0000","last_became_peered":"2024-03-24T12:50:31.824539+0000","last_unstale":"2024-03-24T12:52:29.835359+0000","last_undegraded":"2024-03-24T12:52:29.835359+0000","last_fullsized":"2024-03-24T12:52:29.835359+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:52.176209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:52.176209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:21.605172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":934,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903880+0000","last_change":"2024-03-24T12:51:19.869035+0000","last_active":"2024-03-24T12:52:29.903880+0000","last_peered":"2024-03-24T12:52:29.903880+0000","last_clean":"2024-03-24T12:52:29.903880+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:52:29.903880+0000","last_undegraded":"2024-03-24T12:52:29.903880+0000","last_fullsized":"2024-03-24T12:52:29.903880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:19.868931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:52.028896+0000","last_clean_scrub_stamp":"2024-03-24T12:51:19.868931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:33.722290+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"390'27","reported_seq":112,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835860+0000","last_change":"2024-03-24T12:51:40.429566+0000","last_active":"2024-03-24T12:52:29.835860+0000","last_peered":"2024-03-24T12:52:29.835860+0000","last_clean":"2024-03-24T12:52:29.835860+0000","last_became_active":"2024-03-24T12:50:31.816089+0000","last_became_peered":"2024-03-24T12:50:31.816089+0000","last_unstale":"2024-03-24T12:52:29.835860+0000","last_undegraded":"2024-03-24T12:52:29.835860+0000","last_fullsized":"2024-03-24T12:52:29.835860+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'27","last_scrub_stamp":"2024-03-24T12:51:40.429472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:40.429472+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:49.619158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902878+0000","last_change":"2024-03-24T12:52:21.048054+0000","last_active":"2024-03-24T12:52:29.902878+0000","last_peered":"2024-03-24T12:52:29.902878+0000","last_clean":"2024-03-24T12:52:29.902878+0000","last_became_active":"2024-03-24T12:51:04.954448+0000","last_became_peered":"2024-03-24T12:51:04.954448+0000","last_unstale":"2024-03-24T12:52:29.902878+0000","last_undegraded":"2024-03-24T12:52:29.902878+0000","last_fullsized":"2024-03-24T12:52:29.902878+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:21.047970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:21.047970+0000","last_clean_scrub_stamp":"2024-03-24T12:52:21.047970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:43.729635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":853,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474150+0000","last_change":"2024-03-24T12:51:55.619431+0000","last_active":"2024-03-24T12:52:29.474150+0000","last_peered":"2024-03-24T12:52:29.474150+0000","last_clean":"2024-03-24T12:52:29.474150+0000","last_became_active":"2024-03-24T12:49:54.353078+0000","last_became_peered":"2024-03-24T12:49:54.353078+0000","last_unstale":"2024-03-24T12:52:29.474150+0000","last_undegraded":"2024-03-24T12:52:29.474150+0000","last_fullsized":"2024-03-24T12:52:29.474150+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:55.619346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:55.619346+0000","last_clean_scrub_stamp":"2024-03-24T12:51:55.619346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:05.660350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":613,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904299+0000","last_change":"2024-03-24T12:52:15.115619+0000","last_active":"2024-03-24T12:52:29.904299+0000","last_peered":"2024-03-24T12:52:29.904299+0000","last_clean":"2024-03-24T12:52:29.904299+0000","last_became_active":"2024-03-24T12:50:53.093060+0000","last_became_peered":"2024-03-24T12:50:53.093060+0000","last_unstale":"2024-03-24T12:52:29.904299+0000","last_undegraded":"2024-03-24T12:52:29.904299+0000","last_fullsized":"2024-03-24T12:52:29.904299+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:15.115522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:15.115522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:30.161569+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"390'30","reported_seq":1002,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474282+0000","last_change":"2024-03-24T12:51:30.654950+0000","last_active":"2024-03-24T12:52:29.474282+0000","last_peered":"2024-03-24T12:52:29.474282+0000","last_clean":"2024-03-24T12:52:29.474282+0000","last_became_active":"2024-03-24T12:51:29.009891+0000","last_became_peered":"2024-03-24T12:51:29.009891+0000","last_unstale":"2024-03-24T12:52:29.474282+0000","last_undegraded":"2024-03-24T12:52:29.474282+0000","last_fullsized":"2024-03-24T12:52:29.474282+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'30","last_scrub_stamp":"2024-03-24T12:51:30.654811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:30.654811+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:54.184830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"392'2","reported_seq":971,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902930+0000","last_change":"2024-03-24T12:51:55.761073+0000","last_active":"2024-03-24T12:52:29.902930+0000","last_peered":"2024-03-24T12:52:29.902930+0000","last_clean":"2024-03-24T12:52:29.902930+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:52:29.902930+0000","last_undegraded":"2024-03-24T12:52:29.902930+0000","last_fullsized":"2024-03-24T12:52:29.902930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'2","last_scrub_stamp":"2024-03-24T12:51:55.760985+0000","last_deep_scrub":"392'2","last_deep_scrub_stamp":"2024-03-24T12:46:18.735806+0000","last_clean_scrub_stamp":"2024-03-24T12:51:55.760985+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:19.445802+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"467'14","reported_seq":147,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835957+0000","last_change":"2024-03-24T12:52:10.916938+0000","last_active":"2024-03-24T12:52:29.835957+0000","last_peered":"2024-03-24T12:52:29.835957+0000","last_clean":"2024-03-24T12:52:29.835957+0000","last_became_active":"2024-03-24T12:51:29.007730+0000","last_became_peered":"2024-03-24T12:51:29.007730+0000","last_unstale":"2024-03-24T12:52:29.835957+0000","last_undegraded":"2024-03-24T12:52:29.835957+0000","last_fullsized":"2024-03-24T12:52:29.835957+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'14","last_scrub_stamp":"2024-03-24T12:52:10.916780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:52:10.916780+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:30.192552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":943,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:30.877420+0000","last_change":"2024-03-24T12:52:30.877452+0000","last_active":"2024-03-24T12:52:30.877420+0000","last_peered":"2024-03-24T12:52:30.877420+0000","last_clean":"2024-03-24T12:52:30.877420+0000","last_became_active":"2024-03-24T12:50:27.792660+0000","last_became_peered":"2024-03-24T12:50:27.792660+0000","last_unstale":"2024-03-24T12:52:30.877420+0000","last_undegraded":"2024-03-24T12:52:30.877420+0000","last_fullsized":"2024-03-24T12:52:30.877420+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:52:30.877340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:30.877340+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:50.065177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"391'24","reported_seq":89,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904846+0000","last_change":"2024-03-24T12:52:20.035072+0000","last_active":"2024-03-24T12:52:29.904846+0000","last_peered":"2024-03-24T12:52:29.904846+0000","last_clean":"2024-03-24T12:52:29.904846+0000","last_became_active":"2024-03-24T12:51:04.954618+0000","last_became_peered":"2024-03-24T12:51:04.954618+0000","last_unstale":"2024-03-24T12:52:29.904846+0000","last_undegraded":"2024-03-24T12:52:29.904846+0000","last_fullsized":"2024-03-24T12:52:29.904846+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'24","last_scrub_stamp":"2024-03-24T12:52:20.034924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:52:20.034924+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:29.781477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":945,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473178+0000","last_change":"2024-03-24T12:52:24.210026+0000","last_active":"2024-03-24T12:52:29.473178+0000","last_peered":"2024-03-24T12:52:29.473178+0000","last_clean":"2024-03-24T12:52:29.473178+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:52:29.473178+0000","last_undegraded":"2024-03-24T12:52:29.473178+0000","last_fullsized":"2024-03-24T12:52:29.473178+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:24.209940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:24.209940+0000","last_clean_scrub_stamp":"2024-03-24T12:52:24.209940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:43.795213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":937,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473551+0000","last_change":"2024-03-24T12:52:03.225369+0000","last_active":"2024-03-24T12:52:29.473551+0000","last_peered":"2024-03-24T12:52:29.473551+0000","last_clean":"2024-03-24T12:52:29.473551+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:52:29.473551+0000","last_undegraded":"2024-03-24T12:52:29.473551+0000","last_fullsized":"2024-03-24T12:52:29.473551+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:03.225280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:52:03.225280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:09.177641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":964,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904780+0000","last_change":"2024-03-24T12:52:19.228835+0000","last_active":"2024-03-24T12:52:29.904780+0000","last_peered":"2024-03-24T12:52:29.904780+0000","last_clean":"2024-03-24T12:52:29.904780+0000","last_became_active":"2024-03-24T12:50:31.810641+0000","last_became_peered":"2024-03-24T12:50:31.810641+0000","last_unstale":"2024-03-24T12:52:29.904780+0000","last_undegraded":"2024-03-24T12:52:29.904780+0000","last_fullsized":"2024-03-24T12:52:29.904780+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:19.228735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:19.228735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:39.883765+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"390'39","reported_seq":1063,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473746+0000","last_change":"2024-03-24T12:51:46.965544+0000","last_active":"2024-03-24T12:52:29.473746+0000","last_peered":"2024-03-24T12:52:29.473746+0000","last_clean":"2024-03-24T12:52:29.473746+0000","last_became_active":"2024-03-24T12:51:04.946216+0000","last_became_peered":"2024-03-24T12:51:04.946216+0000","last_unstale":"2024-03-24T12:52:29.473746+0000","last_undegraded":"2024-03-24T12:52:29.473746+0000","last_fullsized":"2024-03-24T12:52:29.473746+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'39","last_scrub_stamp":"2024-03-24T12:51:46.965416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:46.965416+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:48.057143+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":926,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474339+0000","last_change":"2024-03-24T12:52:14.372276+0000","last_active":"2024-03-24T12:52:29.474339+0000","last_peered":"2024-03-24T12:52:29.474339+0000","last_clean":"2024-03-24T12:52:29.474339+0000","last_became_active":"2024-03-24T12:50:48.867700+0000","last_became_peered":"2024-03-24T12:50:48.867700+0000","last_unstale":"2024-03-24T12:52:29.474339+0000","last_undegraded":"2024-03-24T12:52:29.474339+0000","last_fullsized":"2024-03-24T12:52:29.474339+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:14.372001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:52:14.372001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:26.161574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":970,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904200+0000","last_change":"2024-03-24T12:51:53.819927+0000","last_active":"2024-03-24T12:52:29.904200+0000","last_peered":"2024-03-24T12:52:29.904200+0000","last_clean":"2024-03-24T12:52:29.904200+0000","last_became_active":"2024-03-24T12:49:54.354682+0000","last_became_peered":"2024-03-24T12:49:54.354682+0000","last_unstale":"2024-03-24T12:52:29.904200+0000","last_undegraded":"2024-03-24T12:52:29.904200+0000","last_fullsized":"2024-03-24T12:52:29.904200+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:53.819837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:15.638516+0000","last_clean_scrub_stamp":"2024-03-24T12:51:53.819837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:56.226242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":980,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473560+0000","last_change":"2024-03-24T12:51:41.241353+0000","last_active":"2024-03-24T12:52:29.473560+0000","last_peered":"2024-03-24T12:52:29.473560+0000","last_clean":"2024-03-24T12:52:29.473560+0000","last_became_active":"2024-03-24T12:50:31.823435+0000","last_became_peered":"2024-03-24T12:50:31.823435+0000","last_unstale":"2024-03-24T12:52:29.473560+0000","last_undegraded":"2024-03-24T12:52:29.473560+0000","last_fullsized":"2024-03-24T12:52:29.473560+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:41.241241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:41.241241+0000","last_clean_scrub_stamp":"2024-03-24T12:51:41.241241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:11.003582+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"390'9","reported_seq":969,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473294+0000","last_change":"2024-03-24T12:52:13.208344+0000","last_active":"2024-03-24T12:52:29.473294+0000","last_peered":"2024-03-24T12:52:29.473294+0000","last_clean":"2024-03-24T12:52:29.473294+0000","last_became_active":"2024-03-24T12:46:47.943482+0000","last_became_peered":"2024-03-24T12:46:47.943482+0000","last_unstale":"2024-03-24T12:52:29.473294+0000","last_undegraded":"2024-03-24T12:52:29.473294+0000","last_fullsized":"2024-03-24T12:52:29.473294+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'9","last_scrub_stamp":"2024-03-24T12:52:13.208243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:52:13.208243+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:31.850070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":117,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903505+0000","last_change":"2024-03-24T12:52:29.182813+0000","last_active":"2024-03-24T12:52:29.903505+0000","last_peered":"2024-03-24T12:52:29.903505+0000","last_clean":"2024-03-24T12:52:29.903505+0000","last_became_active":"2024-03-24T12:50:27.806126+0000","last_became_peered":"2024-03-24T12:50:27.806126+0000","last_unstale":"2024-03-24T12:52:29.903505+0000","last_undegraded":"2024-03-24T12:52:29.903505+0000","last_fullsized":"2024-03-24T12:52:29.903505+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:29.182714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:52:29.182714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:50.217959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":937,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903642+0000","last_change":"2024-03-24T12:51:54.926712+0000","last_active":"2024-03-24T12:52:29.903642+0000","last_peered":"2024-03-24T12:52:29.903642+0000","last_clean":"2024-03-24T12:52:29.903642+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:52:29.903642+0000","last_undegraded":"2024-03-24T12:52:29.903642+0000","last_fullsized":"2024-03-24T12:52:29.903642+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:54.926619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:54.926619+0000","last_clean_scrub_stamp":"2024-03-24T12:51:54.926619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:02.496301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":123,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902807+0000","last_change":"2024-03-24T12:52:24.165469+0000","last_active":"2024-03-24T12:52:29.902807+0000","last_peered":"2024-03-24T12:52:29.902807+0000","last_clean":"2024-03-24T12:52:29.902807+0000","last_became_active":"2024-03-24T12:51:04.946329+0000","last_became_peered":"2024-03-24T12:51:04.946329+0000","last_unstale":"2024-03-24T12:52:29.902807+0000","last_undegraded":"2024-03-24T12:52:29.902807+0000","last_fullsized":"2024-03-24T12:52:29.902807+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:24.165340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:48:17.733385+0000","last_clean_scrub_stamp":"2024-03-24T12:52:24.165340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:34.348927+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"390'21","reported_seq":976,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473109+0000","last_change":"2024-03-24T12:51:32.239405+0000","last_active":"2024-03-24T12:52:29.473109+0000","last_peered":"2024-03-24T12:52:29.473109+0000","last_clean":"2024-03-24T12:52:29.473109+0000","last_became_active":"2024-03-24T12:49:54.334449+0000","last_became_peered":"2024-03-24T12:49:54.334449+0000","last_unstale":"2024-03-24T12:52:29.473109+0000","last_undegraded":"2024-03-24T12:52:29.473109+0000","last_fullsized":"2024-03-24T12:52:29.473109+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:51:32.239283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:32.239283+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:50.617143+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":123,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904512+0000","last_change":"2024-03-24T12:52:16.072739+0000","last_active":"2024-03-24T12:52:29.904512+0000","last_peered":"2024-03-24T12:52:29.904512+0000","last_clean":"2024-03-24T12:52:29.904512+0000","last_became_active":"2024-03-24T12:50:31.825465+0000","last_became_peered":"2024-03-24T12:50:31.825465+0000","last_unstale":"2024-03-24T12:52:29.904512+0000","last_undegraded":"2024-03-24T12:52:29.904512+0000","last_fullsized":"2024-03-24T12:52:29.904512+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:16.072645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:23.741374+0000","last_clean_scrub_stamp":"2024-03-24T12:52:16.072645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:30.400623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":123,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903068+0000","last_change":"2024-03-24T12:51:46.136545+0000","last_active":"2024-03-24T12:52:29.903068+0000","last_peered":"2024-03-24T12:52:29.903068+0000","last_clean":"2024-03-24T12:52:29.903068+0000","last_became_active":"2024-03-24T12:51:29.010381+0000","last_became_peered":"2024-03-24T12:51:29.010381+0000","last_unstale":"2024-03-24T12:52:29.903068+0000","last_undegraded":"2024-03-24T12:52:29.903068+0000","last_fullsized":"2024-03-24T12:52:29.903068+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:46.136410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:46.136410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:03.165840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":856,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472989+0000","last_change":"2024-03-24T12:51:33.627968+0000","last_active":"2024-03-24T12:52:29.472989+0000","last_peered":"2024-03-24T12:52:29.472989+0000","last_clean":"2024-03-24T12:52:29.472989+0000","last_became_active":"2024-03-24T12:49:54.338779+0000","last_became_peered":"2024-03-24T12:49:54.338779+0000","last_unstale":"2024-03-24T12:52:29.472989+0000","last_undegraded":"2024-03-24T12:52:29.472989+0000","last_fullsized":"2024-03-24T12:52:29.472989+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:33.627822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:51:33.627822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:44.768312+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"391'46","reported_seq":93,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903774+0000","last_change":"2024-03-24T12:51:33.150692+0000","last_active":"2024-03-24T12:52:29.903774+0000","last_peered":"2024-03-24T12:52:29.903774+0000","last_clean":"2024-03-24T12:52:29.903774+0000","last_became_active":"2024-03-24T12:51:04.950754+0000","last_became_peered":"2024-03-24T12:51:04.950754+0000","last_unstale":"2024-03-24T12:52:29.903774+0000","last_undegraded":"2024-03-24T12:52:29.903774+0000","last_fullsized":"2024-03-24T12:52:29.903774+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'46","last_scrub_stamp":"2024-03-24T12:51:33.150579+0000","last_deep_scrub":"207'34","last_deep_scrub_stamp":"2024-03-24T12:45:07.975149+0000","last_clean_scrub_stamp":"2024-03-24T12:51:33.150579+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:40.324063+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903069+0000","last_change":"2024-03-24T12:51:11.924177+0000","last_active":"2024-03-24T12:52:29.903069+0000","last_peered":"2024-03-24T12:52:29.903069+0000","last_clean":"2024-03-24T12:52:29.903069+0000","last_became_active":"2024-03-24T12:51:04.950651+0000","last_became_peered":"2024-03-24T12:51:04.950651+0000","last_unstale":"2024-03-24T12:52:29.903069+0000","last_undegraded":"2024-03-24T12:52:29.903069+0000","last_fullsized":"2024-03-24T12:52:29.903069+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:11.924069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:46.052822+0000","last_clean_scrub_stamp":"2024-03-24T12:51:11.924069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:30.993659+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":961,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472769+0000","last_change":"2024-03-24T12:51:44.249303+0000","last_active":"2024-03-24T12:52:29.472769+0000","last_peered":"2024-03-24T12:52:29.472769+0000","last_clean":"2024-03-24T12:52:29.472769+0000","last_became_active":"2024-03-24T12:50:31.812672+0000","last_became_peered":"2024-03-24T12:50:31.812672+0000","last_unstale":"2024-03-24T12:52:29.472769+0000","last_undegraded":"2024-03-24T12:52:29.472769+0000","last_fullsized":"2024-03-24T12:52:29.472769+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:44.249190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:17.828354+0000","last_clean_scrub_stamp":"2024-03-24T12:51:44.249190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:08.015177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903806+0000","last_change":"2024-03-24T12:51:45.247226+0000","last_active":"2024-03-24T12:52:29.903806+0000","last_peered":"2024-03-24T12:52:29.903806+0000","last_clean":"2024-03-24T12:52:29.903806+0000","last_became_active":"2024-03-24T12:51:04.943974+0000","last_became_peered":"2024-03-24T12:51:04.943974+0000","last_unstale":"2024-03-24T12:52:29.903806+0000","last_undegraded":"2024-03-24T12:52:29.903806+0000","last_fullsized":"2024-03-24T12:52:29.903806+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:45.247095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:48:05.852493+0000","last_clean_scrub_stamp":"2024-03-24T12:51:45.247095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:00.710445+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"390'21","reported_seq":1055,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472620+0000","last_change":"2024-03-24T12:51:56.221729+0000","last_active":"2024-03-24T12:52:29.472620+0000","last_peered":"2024-03-24T12:52:29.472620+0000","last_clean":"2024-03-24T12:52:29.472620+0000","last_became_active":"2024-03-24T12:50:31.813188+0000","last_became_peered":"2024-03-24T12:50:31.813188+0000","last_unstale":"2024-03-24T12:52:29.472620+0000","last_undegraded":"2024-03-24T12:52:29.472620+0000","last_fullsized":"2024-03-24T12:52:29.472620+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:51:56.221622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:56.221622+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:04.676808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":903,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473643+0000","last_change":"2024-03-24T12:51:58.515152+0000","last_active":"2024-03-24T12:52:29.473643+0000","last_peered":"2024-03-24T12:52:29.473643+0000","last_clean":"2024-03-24T12:52:29.473643+0000","last_became_active":"2024-03-24T12:50:28.072789+0000","last_became_peered":"2024-03-24T12:50:28.072789+0000","last_unstale":"2024-03-24T12:52:29.473643+0000","last_undegraded":"2024-03-24T12:52:29.473643+0000","last_fullsized":"2024-03-24T12:52:29.473643+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:58.515066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:05.825276+0000","last_clean_scrub_stamp":"2024-03-24T12:51:58.515066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:10.418026+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":626,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902940+0000","last_change":"2024-03-24T12:52:20.949198+0000","last_active":"2024-03-24T12:52:29.902940+0000","last_peered":"2024-03-24T12:52:29.902940+0000","last_clean":"2024-03-24T12:52:29.902940+0000","last_became_active":"2024-03-24T12:50:53.092959+0000","last_became_peered":"2024-03-24T12:50:53.092959+0000","last_unstale":"2024-03-24T12:52:29.902940+0000","last_undegraded":"2024-03-24T12:52:29.902940+0000","last_fullsized":"2024-03-24T12:52:29.902940+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:20.949077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:55.905225+0000","last_clean_scrub_stamp":"2024-03-24T12:52:20.949077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:35.165828+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":924,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472901+0000","last_change":"2024-03-24T12:52:26.346000+0000","last_active":"2024-03-24T12:52:29.472901+0000","last_peered":"2024-03-24T12:52:29.472901+0000","last_clean":"2024-03-24T12:52:29.472901+0000","last_became_active":"2024-03-24T12:50:27.791835+0000","last_became_peered":"2024-03-24T12:50:27.791835+0000","last_unstale":"2024-03-24T12:52:29.472901+0000","last_undegraded":"2024-03-24T12:52:29.472901+0000","last_fullsized":"2024-03-24T12:52:29.472901+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:26.345892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:26.345892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:42.102058+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1574,"num_read_kb":1569,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1641120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1533,"num_read_kb":1290,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":10198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038083,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808336,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936816,"statfs":{"total":95995035648,"available":95167299584,"internally_reserved":0,"allocated":1118208,"data_stored":714161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":826603978},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332425,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":645556,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93099596,"statfs":{"total":95995035648,"available":95333986304,"internally_reserved":0,"allocated":761856,"data_stored":385783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":660273607},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575203,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948248,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92796904,"statfs":{"total":95995035648,"available":95024029696,"internally_reserved":0,"allocated":802816,"data_stored":380100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":970193349},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":433,"seq":1859720839202,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94672,"kb_used_data":768,"kb_used_omap":4,"kb_used_meta":93883,"kb_avail":93650480,"statfs":{"total":95995035648,"available":95898091520,"internally_reserved":0,"allocated":786432,"data_stored":385715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":96136868},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411677,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952632,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792520,"statfs":{"total":95995035648,"available":95019540480,"internally_reserved":0,"allocated":1028096,"data_stored":647931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117335,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732260,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93012892,"statfs":{"total":95995035648,"available":95245201408,"internally_reserved":0,"allocated":1396736,"data_stored":977055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":748419527},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921107,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744136,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93001016,"statfs":{"total":95995035648,"available":95233040384,"internally_reserved":0,"allocated":786432,"data_stored":385513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":437,"seq":1876900708381,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55224,"kb_used_data":1320,"kb_used_omap":6,"kb_used_meta":53881,"kb_avail":93689928,"statfs":{"total":95995035648,"available":95938486272,"internally_reserved":0,"allocated":1351680,"data_stored":975844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6878,"internal_metadata":55174434},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:52:37.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T12:52:37.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:37 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:52:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:52:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:38 smithi190 ceph-mon[33748]: 3.1e scrub starts 2024-03-24T12:52:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:38 smithi190 ceph-mon[33748]: 3.1e scrub ok 2024-03-24T12:52:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:38 smithi190 ceph-mon[33748]: from='client.25712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:52:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:38 smithi190 ceph-mon[33748]: pgmap v1180: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-24T12:52:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:38 smithi190 ceph-mon[33748]: 6.a scrub starts 2024-03-24T12:52:38.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:38 smithi190 ceph-mon[33748]: 6.a scrub ok 2024-03-24T12:52:38.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[41500]: 3.1e scrub starts 2024-03-24T12:52:38.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[41500]: 3.1e scrub ok 2024-03-24T12:52:38.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[41500]: from='client.25712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:52:38.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[41500]: pgmap v1180: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-24T12:52:38.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[41500]: 6.a scrub starts 2024-03-24T12:52:38.408 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[41500]: 6.a scrub ok 2024-03-24T12:52:38.408 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[29919]: 3.1e scrub starts 2024-03-24T12:52:38.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[29919]: 3.1e scrub ok 2024-03-24T12:52:38.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[29919]: from='client.25712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:52:38.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[29919]: pgmap v1180: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 600 B/s rd, 0 op/s 2024-03-24T12:52:38.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[29919]: 6.a scrub starts 2024-03-24T12:52:38.409 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:38 smithi029 ceph-mon[29919]: 6.a scrub ok 2024-03-24T12:52:39.166 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:39 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:52:39.166 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:39 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:52:39.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:39 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:52:39.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:52:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug there is no tcmu-runner data available 2024-03-24T12:52:39.943 INFO:teuthology.orchestra.run.smithi029.stdout: 2024-03-24T12:52:39.945 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2024-03-24T12:52:40.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[29919]: 6.0 scrub starts 2024-03-24T12:52:40.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[29919]: 6.0 scrub ok 2024-03-24T12:52:40.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[29919]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:40.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[29919]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:40.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:52:40.325 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:52:40.326 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:52:40.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[41500]: 6.0 scrub starts 2024-03-24T12:52:40.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[41500]: 6.0 scrub ok 2024-03-24T12:52:40.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[41500]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:40.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[41500]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:40.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:52:40.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:52:40.326 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:40 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:52:40.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:40 smithi190 ceph-mon[33748]: 6.0 scrub starts 2024-03-24T12:52:40.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:40 smithi190 ceph-mon[33748]: 6.0 scrub ok 2024-03-24T12:52:40.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:40 smithi190 ceph-mon[33748]: from='client.24430 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-24T12:52:40.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:40 smithi190 ceph-mon[33748]: pgmap v1181: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-24T12:52:40.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:40 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:52:40.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:40 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:52:40.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:40 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:52:40.824 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":1181,"stamp":"2024-03-24T12:52:38.987008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3129,"num_read_kb":2881,"num_write":1008,"num_write_kb":45,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":1657173,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2608,"ondisk_log_size":2608,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4981052,"kb_used_data":7856,"kb_used_omap":20,"kb_used_meta":4973099,"kb_avail":744980164,"statfs":{"total":767960285184,"available":762859687936,"internally_reserved":0,"allocated":8044544,"data_stored":4858763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":20883,"internal_metadata":5092453997},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002475"},"pg_stats":[{"pgid":"4.c","version":"390'30","reported_seq":1067,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473578+0000","last_change":"2024-03-24T12:51:46.244134+0000","last_active":"2024-03-24T12:52:29.473578+0000","last_peered":"2024-03-24T12:52:29.473578+0000","last_clean":"2024-03-24T12:52:29.473578+0000","last_became_active":"2024-03-24T12:50:31.813808+0000","last_became_peered":"2024-03-24T12:50:31.813808+0000","last_unstale":"2024-03-24T12:52:29.473578+0000","last_undegraded":"2024-03-24T12:52:29.473578+0000","last_fullsized":"2024-03-24T12:52:29.473578+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'30","last_scrub_stamp":"2024-03-24T12:51:46.244011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:46.244011+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:51.518564+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:32.303852+0000","last_change":"2024-03-24T12:52:32.303874+0000","last_active":"2024-03-24T12:52:32.303852+0000","last_peered":"2024-03-24T12:52:32.303852+0000","last_clean":"2024-03-24T12:52:32.303852+0000","last_became_active":"2024-03-24T12:51:04.953574+0000","last_became_peered":"2024-03-24T12:51:04.953574+0000","last_unstale":"2024-03-24T12:52:32.303852+0000","last_undegraded":"2024-03-24T12:52:32.303852+0000","last_fullsized":"2024-03-24T12:52:32.303852+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:32.303785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:32.303785+0000","last_clean_scrub_stamp":"2024-03-24T12:52:32.303785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:35.811701+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"440'13","reported_seq":1278,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473664+0000","last_change":"2024-03-24T12:51:28.013037+0000","last_active":"2024-03-24T12:52:29.473664+0000","last_peered":"2024-03-24T12:52:29.473664+0000","last_clean":"2024-03-24T12:52:29.473664+0000","last_became_active":"2024-03-24T12:50:27.806738+0000","last_became_peered":"2024-03-24T12:50:27.806738+0000","last_unstale":"2024-03-24T12:52:29.473664+0000","last_undegraded":"2024-03-24T12:52:29.473664+0000","last_fullsized":"2024-03-24T12:52:29.473664+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'13","last_scrub_stamp":"2024-03-24T12:51:28.012891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:28.012891+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:54.551766+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":942,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473806+0000","last_change":"2024-03-24T12:52:22.286274+0000","last_active":"2024-03-24T12:52:29.473806+0000","last_peered":"2024-03-24T12:52:29.473806+0000","last_clean":"2024-03-24T12:52:29.473806+0000","last_became_active":"2024-03-24T12:26:09.350440+0000","last_became_peered":"2024-03-24T12:26:09.350440+0000","last_unstale":"2024-03-24T12:52:29.473806+0000","last_undegraded":"2024-03-24T12:52:29.473806+0000","last_fullsized":"2024-03-24T12:52:29.473806+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:22.286131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:42:45.013715+0000","last_clean_scrub_stamp":"2024-03-24T12:52:22.286131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:47.522175+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"83'128","reported_seq":1257,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903135+0000","last_change":"2024-03-24T12:52:11.720132+0000","last_active":"2024-03-24T12:52:29.903135+0000","last_peered":"2024-03-24T12:52:29.903135+0000","last_clean":"2024-03-24T12:52:29.903135+0000","last_became_active":"2024-03-24T12:41:52.953388+0000","last_became_peered":"2024-03-24T12:41:52.953388+0000","last_unstale":"2024-03-24T12:52:29.903135+0000","last_undegraded":"2024-03-24T12:52:29.903135+0000","last_fullsized":"2024-03-24T12:52:29.903135+0000","mapping_epoch":298,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:52:11.720024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:52:11.720024+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:17.140601+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"390'53","reported_seq":1028,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:31.193673+0000","last_change":"2024-03-24T12:52:31.193733+0000","last_active":"2024-03-24T12:52:31.193673+0000","last_peered":"2024-03-24T12:52:31.193673+0000","last_clean":"2024-03-24T12:52:31.193673+0000","last_became_active":"2024-03-24T12:26:05.316282+0000","last_became_peered":"2024-03-24T12:26:05.316282+0000","last_unstale":"2024-03-24T12:52:31.193673+0000","last_undegraded":"2024-03-24T12:52:31.193673+0000","last_fullsized":"2024-03-24T12:52:31.193673+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'53","last_scrub_stamp":"2024-03-24T12:52:31.193601+0000","last_deep_scrub":"207'39","last_deep_scrub_stamp":"2024-03-24T12:36:22.669686+0000","last_clean_scrub_stamp":"2024-03-24T12:52:31.193601+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:47.495688+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":108,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835396+0000","last_change":"2024-03-24T12:51:43.316224+0000","last_active":"2024-03-24T12:52:29.835396+0000","last_peered":"2024-03-24T12:52:29.835396+0000","last_clean":"2024-03-24T12:52:29.835396+0000","last_became_active":"2024-03-24T12:50:31.810110+0000","last_became_peered":"2024-03-24T12:50:31.810110+0000","last_unstale":"2024-03-24T12:52:29.835396+0000","last_undegraded":"2024-03-24T12:52:29.835396+0000","last_fullsized":"2024-03-24T12:52:29.835396+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:43.316129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:43.316129+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:44.079453+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":506,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903605+0000","last_change":"2024-03-24T12:52:17.893939+0000","last_active":"2024-03-24T12:52:29.903605+0000","last_peered":"2024-03-24T12:52:29.903605+0000","last_clean":"2024-03-24T12:52:29.903605+0000","last_became_active":"2024-03-24T12:50:53.091678+0000","last_became_peered":"2024-03-24T12:50:53.091678+0000","last_unstale":"2024-03-24T12:52:29.903605+0000","last_undegraded":"2024-03-24T12:52:29.903605+0000","last_fullsized":"2024-03-24T12:52:29.903605+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:17.893845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:52:17.893845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:26.182758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":968,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473111+0000","last_change":"2024-03-24T12:51:58.021051+0000","last_active":"2024-03-24T12:52:29.473111+0000","last_peered":"2024-03-24T12:52:29.473111+0000","last_clean":"2024-03-24T12:52:29.473111+0000","last_became_active":"2024-03-24T12:51:04.946095+0000","last_became_peered":"2024-03-24T12:51:04.946095+0000","last_unstale":"2024-03-24T12:52:29.473111+0000","last_undegraded":"2024-03-24T12:52:29.473111+0000","last_fullsized":"2024-03-24T12:52:29.473111+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:58.020949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:53.627010+0000","last_clean_scrub_stamp":"2024-03-24T12:51:58.020949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:17.580521+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.8","version":"83'128","reported_seq":636,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902647+0000","last_change":"2024-03-24T12:51:33.193289+0000","last_active":"2024-03-24T12:52:29.902647+0000","last_peered":"2024-03-24T12:52:29.902647+0000","last_clean":"2024-03-24T12:52:29.902647+0000","last_became_active":"2024-03-24T12:51:07.935815+0000","last_became_peered":"2024-03-24T12:51:07.935815+0000","last_unstale":"2024-03-24T12:52:29.902647+0000","last_undegraded":"2024-03-24T12:52:29.902647+0000","last_fullsized":"2024-03-24T12:52:29.902647+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:33.193159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:51:33.193159+0000","objects_scrubbed":1,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:57.291047+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1313440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"390'27","reported_seq":1093,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473718+0000","last_change":"2024-03-24T12:51:52.197139+0000","last_active":"2024-03-24T12:52:29.473718+0000","last_peered":"2024-03-24T12:52:29.473718+0000","last_clean":"2024-03-24T12:52:29.473718+0000","last_became_active":"2024-03-24T12:50:31.824877+0000","last_became_peered":"2024-03-24T12:50:31.824877+0000","last_unstale":"2024-03-24T12:52:29.473718+0000","last_undegraded":"2024-03-24T12:52:29.473718+0000","last_fullsized":"2024-03-24T12:52:29.473718+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'27","last_scrub_stamp":"2024-03-24T12:51:52.197041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:52.197041+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:16.595704+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":939,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472808+0000","last_change":"2024-03-24T12:51:27.213082+0000","last_active":"2024-03-24T12:52:29.472808+0000","last_peered":"2024-03-24T12:52:29.472808+0000","last_clean":"2024-03-24T12:52:29.472808+0000","last_became_active":"2024-03-24T12:26:03.159335+0000","last_became_peered":"2024-03-24T12:26:03.159335+0000","last_unstale":"2024-03-24T12:52:29.472808+0000","last_undegraded":"2024-03-24T12:52:29.472808+0000","last_fullsized":"2024-03-24T12:52:29.472808+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:27.212982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:27.212982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:55.955204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":215,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904385+0000","last_change":"2024-03-24T12:51:55.067409+0000","last_active":"2024-03-24T12:52:29.904385+0000","last_peered":"2024-03-24T12:52:29.904385+0000","last_clean":"2024-03-24T12:52:29.904385+0000","last_became_active":"2024-03-24T12:46:46.944569+0000","last_became_peered":"2024-03-24T12:46:46.944569+0000","last_unstale":"2024-03-24T12:52:29.904385+0000","last_undegraded":"2024-03-24T12:52:29.904385+0000","last_fullsized":"2024-03-24T12:52:29.904385+0000","mapping_epoch":399,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":400,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:55.067325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:55.067325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:22.524561+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904487+0000","last_change":"2024-03-24T12:52:28.184400+0000","last_active":"2024-03-24T12:52:29.904487+0000","last_peered":"2024-03-24T12:52:29.904487+0000","last_clean":"2024-03-24T12:52:29.904487+0000","last_became_active":"2024-03-24T12:51:04.943360+0000","last_became_peered":"2024-03-24T12:51:04.943360+0000","last_unstale":"2024-03-24T12:52:29.904487+0000","last_undegraded":"2024-03-24T12:52:29.904487+0000","last_fullsized":"2024-03-24T12:52:29.904487+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:28.184313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:07.890930+0000","last_clean_scrub_stamp":"2024-03-24T12:52:28.184313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:42.512115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"391'46","reported_seq":1001,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473352+0000","last_change":"2024-03-24T12:51:56.890687+0000","last_active":"2024-03-24T12:52:29.473352+0000","last_peered":"2024-03-24T12:52:29.473352+0000","last_clean":"2024-03-24T12:52:29.473352+0000","last_became_active":"2024-03-24T12:26:16.873435+0000","last_became_peered":"2024-03-24T12:26:16.873435+0000","last_unstale":"2024-03-24T12:52:29.473352+0000","last_undegraded":"2024-03-24T12:52:29.473352+0000","last_fullsized":"2024-03-24T12:52:29.473352+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'46","last_scrub_stamp":"2024-03-24T12:51:56.890514+0000","last_deep_scrub":"391'46","last_deep_scrub_stamp":"2024-03-24T12:50:34.028632+0000","last_clean_scrub_stamp":"2024-03-24T12:51:56.890514+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:07.160033+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902698+0000","last_change":"2024-03-24T12:51:53.043463+0000","last_active":"2024-03-24T12:52:29.902698+0000","last_peered":"2024-03-24T12:52:29.902698+0000","last_clean":"2024-03-24T12:52:29.902698+0000","last_became_active":"2024-03-24T12:51:04.955308+0000","last_became_peered":"2024-03-24T12:51:04.955308+0000","last_unstale":"2024-03-24T12:52:29.902698+0000","last_undegraded":"2024-03-24T12:52:29.902698+0000","last_fullsized":"2024-03-24T12:52:29.902698+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:53.043370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:53.043370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:09.727128+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"440'14","reported_seq":1279,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473753+0000","last_change":"2024-03-24T12:52:08.220047+0000","last_active":"2024-03-24T12:52:29.473753+0000","last_peered":"2024-03-24T12:52:29.473753+0000","last_clean":"2024-03-24T12:52:29.473753+0000","last_became_active":"2024-03-24T12:50:27.805141+0000","last_became_peered":"2024-03-24T12:50:27.805141+0000","last_unstale":"2024-03-24T12:52:29.473753+0000","last_undegraded":"2024-03-24T12:52:29.473753+0000","last_fullsized":"2024-03-24T12:52:29.473753+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'14","last_scrub_stamp":"2024-03-24T12:52:08.219892+0000","last_deep_scrub":"440'14","last_deep_scrub_stamp":"2024-03-24T12:50:49.885051+0000","last_clean_scrub_stamp":"2024-03-24T12:52:08.219892+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:13.162516+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":117,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904207+0000","last_change":"2024-03-24T12:52:12.213861+0000","last_active":"2024-03-24T12:52:29.904207+0000","last_peered":"2024-03-24T12:52:29.904207+0000","last_clean":"2024-03-24T12:52:29.904207+0000","last_became_active":"2024-03-24T12:50:27.814939+0000","last_became_peered":"2024-03-24T12:50:27.814939+0000","last_unstale":"2024-03-24T12:52:29.904207+0000","last_undegraded":"2024-03-24T12:52:29.904207+0000","last_fullsized":"2024-03-24T12:52:29.904207+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:12.213729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:00.198109+0000","last_clean_scrub_stamp":"2024-03-24T12:52:12.213729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:37.473828+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.a","version":"83'128","reported_seq":583,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903144+0000","last_change":"2024-03-24T12:51:18.327429+0000","last_active":"2024-03-24T12:52:29.903144+0000","last_peered":"2024-03-24T12:52:29.903144+0000","last_clean":"2024-03-24T12:52:29.903144+0000","last_became_active":"2024-03-24T12:50:34.821401+0000","last_became_peered":"2024-03-24T12:50:34.821401+0000","last_unstale":"2024-03-24T12:52:29.903144+0000","last_undegraded":"2024-03-24T12:52:29.903144+0000","last_fullsized":"2024-03-24T12:52:29.903144+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:18.327295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:51:18.327295+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:40.745547+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":978,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472786+0000","last_change":"2024-03-24T12:51:40.198464+0000","last_active":"2024-03-24T12:52:29.472786+0000","last_peered":"2024-03-24T12:52:29.472786+0000","last_clean":"2024-03-24T12:52:29.472786+0000","last_became_active":"2024-03-24T12:51:29.008877+0000","last_became_peered":"2024-03-24T12:51:29.008877+0000","last_unstale":"2024-03-24T12:52:29.472786+0000","last_undegraded":"2024-03-24T12:52:29.472786+0000","last_fullsized":"2024-03-24T12:52:29.472786+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:40.198374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:59.761116+0000","last_clean_scrub_stamp":"2024-03-24T12:51:40.198374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:59.717777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"450'50","reported_seq":119,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902786+0000","last_change":"2024-03-24T12:52:23.071097+0000","last_active":"2024-03-24T12:52:29.902786+0000","last_peered":"2024-03-24T12:52:29.902786+0000","last_clean":"2024-03-24T12:52:29.902786+0000","last_became_active":"2024-03-24T12:51:29.010620+0000","last_became_peered":"2024-03-24T12:51:29.010620+0000","last_unstale":"2024-03-24T12:52:29.902786+0000","last_undegraded":"2024-03-24T12:52:29.902786+0000","last_fullsized":"2024-03-24T12:52:29.902786+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"450'50","last_scrub_stamp":"2024-03-24T12:52:23.070958+0000","last_deep_scrub":"450'50","last_deep_scrub_stamp":"2024-03-24T12:50:49.623869+0000","last_clean_scrub_stamp":"2024-03-24T12:52:23.070958+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:43.306693+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":56,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":940,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904479+0000","last_change":"2024-03-24T12:51:52.063115+0000","last_active":"2024-03-24T12:52:29.904479+0000","last_peered":"2024-03-24T12:52:29.904479+0000","last_clean":"2024-03-24T12:52:29.904479+0000","last_became_active":"2024-03-24T12:49:54.335714+0000","last_became_peered":"2024-03-24T12:49:54.335714+0000","last_unstale":"2024-03-24T12:52:29.904479+0000","last_undegraded":"2024-03-24T12:52:29.904479+0000","last_fullsized":"2024-03-24T12:52:29.904479+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:52.063033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:52.063033+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:06.673419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":953,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:35.118231+0000","last_change":"2024-03-24T12:52:35.118256+0000","last_active":"2024-03-24T12:52:35.118231+0000","last_peered":"2024-03-24T12:52:35.118231+0000","last_clean":"2024-03-24T12:52:35.118231+0000","last_became_active":"2024-03-24T12:51:04.946394+0000","last_became_peered":"2024-03-24T12:51:04.946394+0000","last_unstale":"2024-03-24T12:52:35.118231+0000","last_undegraded":"2024-03-24T12:52:35.118231+0000","last_fullsized":"2024-03-24T12:52:35.118231+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:35.118167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:35.118167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:53.670670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"83'128","reported_seq":755,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473494+0000","last_change":"2024-03-24T12:51:39.228740+0000","last_active":"2024-03-24T12:52:29.473494+0000","last_peered":"2024-03-24T12:52:29.473494+0000","last_clean":"2024-03-24T12:52:29.473494+0000","last_became_active":"2024-03-24T12:50:34.830594+0000","last_became_peered":"2024-03-24T12:50:34.830594+0000","last_unstale":"2024-03-24T12:52:29.473494+0000","last_undegraded":"2024-03-24T12:52:29.473494+0000","last_fullsized":"2024-03-24T12:52:29.473494+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:39.228652+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T12:51:39.228652+0000","last_clean_scrub_stamp":"2024-03-24T12:51:39.228652+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.727008+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"459'13","reported_seq":99,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902642+0000","last_change":"2024-03-24T12:52:05.000817+0000","last_active":"2024-03-24T12:52:29.902642+0000","last_peered":"2024-03-24T12:52:29.902642+0000","last_clean":"2024-03-24T12:52:29.902642+0000","last_became_active":"2024-03-24T12:51:04.958698+0000","last_became_peered":"2024-03-24T12:51:04.958698+0000","last_unstale":"2024-03-24T12:52:29.902642+0000","last_undegraded":"2024-03-24T12:52:29.902642+0000","last_fullsized":"2024-03-24T12:52:29.902642+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'13","last_scrub_stamp":"2024-03-24T12:52:05.000708+0000","last_deep_scrub":"417'10","last_deep_scrub_stamp":"2024-03-24T12:48:14.628984+0000","last_clean_scrub_stamp":"2024-03-24T12:52:05.000708+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:05.026370+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"390'49","reported_seq":1115,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473000+0000","last_change":"2024-03-24T12:51:36.186049+0000","last_active":"2024-03-24T12:52:29.473000+0000","last_peered":"2024-03-24T12:52:29.473000+0000","last_clean":"2024-03-24T12:52:29.473000+0000","last_became_active":"2024-03-24T12:50:31.813242+0000","last_became_peered":"2024-03-24T12:50:31.813242+0000","last_unstale":"2024-03-24T12:52:29.473000+0000","last_undegraded":"2024-03-24T12:52:29.473000+0000","last_fullsized":"2024-03-24T12:52:29.473000+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'49","last_scrub_stamp":"2024-03-24T12:51:36.185877+0000","last_deep_scrub":"93'21","last_deep_scrub_stamp":"2024-03-24T12:27:59.393687+0000","last_clean_scrub_stamp":"2024-03-24T12:51:36.185877+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:01.750024+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"67'1","reported_seq":880,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474439+0000","last_change":"2024-03-24T12:51:49.643147+0000","last_active":"2024-03-24T12:52:29.474439+0000","last_peered":"2024-03-24T12:52:29.474439+0000","last_clean":"2024-03-24T12:52:29.474439+0000","last_became_active":"2024-03-24T12:49:54.360509+0000","last_became_peered":"2024-03-24T12:49:54.360509+0000","last_unstale":"2024-03-24T12:52:29.474439+0000","last_undegraded":"2024-03-24T12:52:29.474439+0000","last_fullsized":"2024-03-24T12:52:29.474439+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-24T12:51:49.643033+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:49.643033+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:05.384239+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":960,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904237+0000","last_change":"2024-03-24T12:52:22.975994+0000","last_active":"2024-03-24T12:52:29.904237+0000","last_peered":"2024-03-24T12:52:29.904237+0000","last_clean":"2024-03-24T12:52:29.904237+0000","last_became_active":"2024-03-24T12:50:31.823357+0000","last_became_peered":"2024-03-24T12:50:31.823357+0000","last_unstale":"2024-03-24T12:52:29.904237+0000","last_undegraded":"2024-03-24T12:52:29.904237+0000","last_fullsized":"2024-03-24T12:52:29.904237+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:22.975898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:03.105774+0000","last_clean_scrub_stamp":"2024-03-24T12:52:22.975898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:25.616897+0000","scrub_duration":123,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"447'129","reported_seq":940,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904588+0000","last_change":"2024-03-24T12:52:17.123283+0000","last_active":"2024-03-24T12:52:29.904588+0000","last_peered":"2024-03-24T12:52:29.904588+0000","last_clean":"2024-03-24T12:52:29.904588+0000","last_became_active":"2024-03-24T12:51:07.937015+0000","last_became_peered":"2024-03-24T12:51:07.937015+0000","last_unstale":"2024-03-24T12:52:29.904588+0000","last_undegraded":"2024-03-24T12:52:29.904588+0000","last_fullsized":"2024-03-24T12:52:29.904588+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":3,"last_scrub":"447'129","last_scrub_stamp":"2024-03-24T12:52:17.123191+0000","last_deep_scrub":"447'129","last_deep_scrub_stamp":"2024-03-24T12:50:35.407598+0000","last_clean_scrub_stamp":"2024-03-24T12:52:17.123191+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:39.925267+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":327680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3","version":"445'20","reported_seq":1269,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473997+0000","last_change":"2024-03-24T12:52:20.418347+0000","last_active":"2024-03-24T12:52:29.473997+0000","last_peered":"2024-03-24T12:52:29.473997+0000","last_clean":"2024-03-24T12:52:29.473997+0000","last_became_active":"2024-03-24T12:50:31.819567+0000","last_became_peered":"2024-03-24T12:50:31.819567+0000","last_unstale":"2024-03-24T12:52:29.473997+0000","last_undegraded":"2024-03-24T12:52:29.473997+0000","last_fullsized":"2024-03-24T12:52:29.473997+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'20","last_scrub_stamp":"2024-03-24T12:52:20.418264+0000","last_deep_scrub":"318'14","last_deep_scrub_stamp":"2024-03-24T12:42:45.093908+0000","last_clean_scrub_stamp":"2024-03-24T12:52:20.418264+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:46.222323+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"390'35","reported_seq":101,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903387+0000","last_change":"2024-03-24T12:51:58.862508+0000","last_active":"2024-03-24T12:52:29.903387+0000","last_peered":"2024-03-24T12:52:29.903387+0000","last_clean":"2024-03-24T12:52:29.903387+0000","last_became_active":"2024-03-24T12:50:48.871716+0000","last_became_peered":"2024-03-24T12:50:48.871716+0000","last_unstale":"2024-03-24T12:52:29.903387+0000","last_undegraded":"2024-03-24T12:52:29.903387+0000","last_fullsized":"2024-03-24T12:52:29.903387+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'35","last_scrub_stamp":"2024-03-24T12:51:58.862393+0000","last_deep_scrub":"390'35","last_deep_scrub_stamp":"2024-03-24T12:51:58.862393+0000","last_clean_scrub_stamp":"2024-03-24T12:51:58.862393+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:01.616480+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":130,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904152+0000","last_change":"2024-03-24T12:52:28.169863+0000","last_active":"2024-03-24T12:52:29.904152+0000","last_peered":"2024-03-24T12:52:29.904152+0000","last_clean":"2024-03-24T12:52:29.904152+0000","last_became_active":"2024-03-24T12:51:04.946157+0000","last_became_peered":"2024-03-24T12:51:04.946157+0000","last_unstale":"2024-03-24T12:52:29.904152+0000","last_undegraded":"2024-03-24T12:52:29.904152+0000","last_fullsized":"2024-03-24T12:52:29.904152+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:28.169714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:52:28.169714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:34.596704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":861,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472860+0000","last_change":"2024-03-24T12:51:11.671035+0000","last_active":"2024-03-24T12:52:29.472860+0000","last_peered":"2024-03-24T12:52:29.472860+0000","last_clean":"2024-03-24T12:52:29.472860+0000","last_became_active":"2024-03-24T12:51:04.944332+0000","last_became_peered":"2024-03-24T12:51:04.944332+0000","last_unstale":"2024-03-24T12:52:29.472860+0000","last_undegraded":"2024-03-24T12:52:29.472860+0000","last_fullsized":"2024-03-24T12:52:29.472860+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:11.670904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:51:11.670904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:38.001797+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.7","version":"83'128","reported_seq":821,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835965+0000","last_change":"2024-03-24T12:52:01.103100+0000","last_active":"2024-03-24T12:52:29.835965+0000","last_peered":"2024-03-24T12:52:29.835965+0000","last_clean":"2024-03-24T12:52:29.835965+0000","last_became_active":"2024-03-24T12:51:07.936943+0000","last_became_peered":"2024-03-24T12:51:07.936943+0000","last_unstale":"2024-03-24T12:52:29.835965+0000","last_undegraded":"2024-03-24T12:52:29.835965+0000","last_fullsized":"2024-03-24T12:52:29.835965+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:52:01.103008+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T12:48:16.724235+0000","last_clean_scrub_stamp":"2024-03-24T12:52:01.103008+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:16.285328+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":109,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835570+0000","last_change":"2024-03-24T12:52:03.077440+0000","last_active":"2024-03-24T12:52:29.835570+0000","last_peered":"2024-03-24T12:52:29.835570+0000","last_clean":"2024-03-24T12:52:29.835570+0000","last_became_active":"2024-03-24T12:50:31.817753+0000","last_became_peered":"2024-03-24T12:50:31.817753+0000","last_unstale":"2024-03-24T12:52:29.835570+0000","last_undegraded":"2024-03-24T12:52:29.835570+0000","last_fullsized":"2024-03-24T12:52:29.835570+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:03.077349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:03.077349+0000","last_clean_scrub_stamp":"2024-03-24T12:52:03.077349+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:24.824316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"390'47","reported_seq":1001,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474315+0000","last_change":"2024-03-24T12:51:43.650531+0000","last_active":"2024-03-24T12:52:29.474315+0000","last_peered":"2024-03-24T12:52:29.474315+0000","last_clean":"2024-03-24T12:52:29.474315+0000","last_became_active":"2024-03-24T12:50:27.807819+0000","last_became_peered":"2024-03-24T12:50:27.807819+0000","last_unstale":"2024-03-24T12:52:29.474315+0000","last_undegraded":"2024-03-24T12:52:29.474315+0000","last_fullsized":"2024-03-24T12:52:29.474315+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'47","last_scrub_stamp":"2024-03-24T12:51:43.650438+0000","last_deep_scrub":"195'27","last_deep_scrub_stamp":"2024-03-24T12:35:16.981148+0000","last_clean_scrub_stamp":"2024-03-24T12:51:43.650438+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:57.775684+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":840,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903341+0000","last_change":"2024-03-24T12:51:34.863763+0000","last_active":"2024-03-24T12:52:29.903341+0000","last_peered":"2024-03-24T12:52:29.903341+0000","last_clean":"2024-03-24T12:52:29.903341+0000","last_became_active":"2024-03-24T12:50:53.092560+0000","last_became_peered":"2024-03-24T12:50:53.092560+0000","last_unstale":"2024-03-24T12:52:29.903341+0000","last_undegraded":"2024-03-24T12:52:29.903341+0000","last_fullsized":"2024-03-24T12:52:29.903341+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:34.863682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:34.863682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:45.166072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":983,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903045+0000","last_change":"2024-03-24T12:52:02.915127+0000","last_active":"2024-03-24T12:52:29.903045+0000","last_peered":"2024-03-24T12:52:29.903045+0000","last_clean":"2024-03-24T12:52:29.903045+0000","last_became_active":"2024-03-24T12:50:31.810981+0000","last_became_peered":"2024-03-24T12:50:31.810981+0000","last_unstale":"2024-03-24T12:52:29.903045+0000","last_undegraded":"2024-03-24T12:52:29.903045+0000","last_fullsized":"2024-03-24T12:52:29.903045+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:02.915028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:19.627429+0000","last_clean_scrub_stamp":"2024-03-24T12:52:02.915028+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:02.929174+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"83'128","reported_seq":563,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835828+0000","last_change":"2024-03-24T12:51:27.442953+0000","last_active":"2024-03-24T12:52:29.835828+0000","last_peered":"2024-03-24T12:52:29.835828+0000","last_clean":"2024-03-24T12:52:29.835828+0000","last_became_active":"2024-03-24T12:51:07.935124+0000","last_became_peered":"2024-03-24T12:51:07.935124+0000","last_unstale":"2024-03-24T12:52:29.835828+0000","last_undegraded":"2024-03-24T12:52:29.835828+0000","last_fullsized":"2024-03-24T12:52:29.835828+0000","mapping_epoch":458,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":459,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:27.442795+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T12:51:27.442795+0000","last_clean_scrub_stamp":"2024-03-24T12:51:27.442795+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:46.528507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":117,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:35.113266+0000","last_change":"2024-03-24T12:52:35.113291+0000","last_active":"2024-03-24T12:52:35.113266+0000","last_peered":"2024-03-24T12:52:35.113266+0000","last_clean":"2024-03-24T12:52:35.113266+0000","last_became_active":"2024-03-24T12:50:27.803276+0000","last_became_peered":"2024-03-24T12:50:27.803276+0000","last_unstale":"2024-03-24T12:52:35.113266+0000","last_undegraded":"2024-03-24T12:52:35.113266+0000","last_fullsized":"2024-03-24T12:52:35.113266+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:35.113171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:07.923592+0000","last_clean_scrub_stamp":"2024-03-24T12:52:35.113171+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:54:03.240221+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"390'42","reported_seq":918,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474124+0000","last_change":"2024-03-24T12:51:17.642108+0000","last_active":"2024-03-24T12:52:29.474124+0000","last_peered":"2024-03-24T12:52:29.474124+0000","last_clean":"2024-03-24T12:52:29.474124+0000","last_became_active":"2024-03-24T12:49:54.352611+0000","last_became_peered":"2024-03-24T12:49:54.352611+0000","last_unstale":"2024-03-24T12:52:29.474124+0000","last_undegraded":"2024-03-24T12:52:29.474124+0000","last_fullsized":"2024-03-24T12:52:29.474124+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'42","last_scrub_stamp":"2024-03-24T12:51:17.642011+0000","last_deep_scrub":"260'30","last_deep_scrub_stamp":"2024-03-24T12:39:51.288276+0000","last_clean_scrub_stamp":"2024-03-24T12:51:17.642011+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:43.109003+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"75'5","reported_seq":2560,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:36.592422+0000","last_change":"2024-03-24T12:51:27.802927+0000","last_active":"2024-03-24T12:52:36.592422+0000","last_peered":"2024-03-24T12:52:36.592422+0000","last_clean":"2024-03-24T12:52:36.592422+0000","last_became_active":"2024-03-24T12:49:54.337309+0000","last_became_peered":"2024-03-24T12:49:54.337309+0000","last_unstale":"2024-03-24T12:52:36.592422+0000","last_undegraded":"2024-03-24T12:52:36.592422+0000","last_fullsized":"2024-03-24T12:52:36.592422+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"75'5","last_scrub_stamp":"2024-03-24T12:51:27.802797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:27.802797+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060460099999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1579,"num_read_kb":1574,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":889,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472722+0000","last_change":"2024-03-24T12:51:57.531772+0000","last_active":"2024-03-24T12:52:29.472722+0000","last_peered":"2024-03-24T12:52:29.472722+0000","last_clean":"2024-03-24T12:52:29.472722+0000","last_became_active":"2024-03-24T12:51:04.937869+0000","last_became_peered":"2024-03-24T12:51:04.937869+0000","last_unstale":"2024-03-24T12:52:29.472722+0000","last_undegraded":"2024-03-24T12:52:29.472722+0000","last_fullsized":"2024-03-24T12:52:29.472722+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:57.531645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:36:13.629968+0000","last_clean_scrub_stamp":"2024-03-24T12:51:57.531645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:04.827173+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":966,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902669+0000","last_change":"2024-03-24T12:52:04.806381+0000","last_active":"2024-03-24T12:52:29.902669+0000","last_peered":"2024-03-24T12:52:29.902669+0000","last_clean":"2024-03-24T12:52:29.902669+0000","last_became_active":"2024-03-24T12:51:04.944003+0000","last_became_peered":"2024-03-24T12:51:04.944003+0000","last_unstale":"2024-03-24T12:52:29.902669+0000","last_undegraded":"2024-03-24T12:52:29.902669+0000","last_fullsized":"2024-03-24T12:52:29.902669+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:04.806299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:03.186807+0000","last_clean_scrub_stamp":"2024-03-24T12:52:04.806299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:33.723170+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"83'128","reported_seq":804,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902669+0000","last_change":"2024-03-24T12:52:06.822906+0000","last_active":"2024-03-24T12:52:29.902669+0000","last_peered":"2024-03-24T12:52:29.902669+0000","last_clean":"2024-03-24T12:52:29.902669+0000","last_became_active":"2024-03-24T12:50:52.890546+0000","last_became_peered":"2024-03-24T12:50:52.890546+0000","last_unstale":"2024-03-24T12:52:29.902669+0000","last_undegraded":"2024-03-24T12:52:29.902669+0000","last_fullsized":"2024-03-24T12:52:29.902669+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":3,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:52:06.822827+0000","last_deep_scrub":"83'128","last_deep_scrub_stamp":"2024-03-24T12:50:36.279886+0000","last_clean_scrub_stamp":"2024-03-24T12:52:06.822827+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:25.887334+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":931,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903516+0000","last_change":"2024-03-24T12:51:53.932874+0000","last_active":"2024-03-24T12:52:29.903516+0000","last_peered":"2024-03-24T12:52:29.903516+0000","last_clean":"2024-03-24T12:52:29.903516+0000","last_became_active":"2024-03-24T12:50:27.795417+0000","last_became_peered":"2024-03-24T12:50:27.795417+0000","last_unstale":"2024-03-24T12:52:29.903516+0000","last_undegraded":"2024-03-24T12:52:29.903516+0000","last_fullsized":"2024-03-24T12:52:29.903516+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:53.932765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:17.105384+0000","last_clean_scrub_stamp":"2024-03-24T12:51:53.932765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:21.746857+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"445'46","reported_seq":153,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:38.521861+0000","last_change":"2024-03-24T12:52:23.860497+0000","last_active":"2024-03-24T12:52:38.521861+0000","last_peered":"2024-03-24T12:52:38.521861+0000","last_clean":"2024-03-24T12:52:38.521861+0000","last_became_active":"2024-03-24T12:50:31.822980+0000","last_became_peered":"2024-03-24T12:50:31.822980+0000","last_unstale":"2024-03-24T12:52:38.521861+0000","last_undegraded":"2024-03-24T12:52:38.521861+0000","last_fullsized":"2024-03-24T12:52:38.521861+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'46","last_scrub_stamp":"2024-03-24T12:52:23.860377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:52:23.860377+0000","objects_scrubbed":8,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:36.611967+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903472+0000","last_change":"2024-03-24T12:51:09.943444+0000","last_active":"2024-03-24T12:52:29.903472+0000","last_peered":"2024-03-24T12:52:29.903472+0000","last_clean":"2024-03-24T12:52:29.903472+0000","last_became_active":"2024-03-24T12:51:04.943402+0000","last_became_peered":"2024-03-24T12:51:04.943402+0000","last_unstale":"2024-03-24T12:52:29.903472+0000","last_undegraded":"2024-03-24T12:52:29.903472+0000","last_fullsized":"2024-03-24T12:52:29.903472+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:09.943351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:09.943351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:35.556673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"83'128","reported_seq":1268,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903315+0000","last_change":"2024-03-24T12:51:26.773048+0000","last_active":"2024-03-24T12:52:29.903315+0000","last_peered":"2024-03-24T12:52:29.903315+0000","last_clean":"2024-03-24T12:52:29.903315+0000","last_became_active":"2024-03-24T12:50:34.825664+0000","last_became_peered":"2024-03-24T12:50:34.825664+0000","last_unstale":"2024-03-24T12:52:29.903315+0000","last_undegraded":"2024-03-24T12:52:29.903315+0000","last_fullsized":"2024-03-24T12:52:29.903315+0000","mapping_epoch":446,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:26.772939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:51:26.772939+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:33.216983+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":117,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904722+0000","last_change":"2024-03-24T12:52:14.219048+0000","last_active":"2024-03-24T12:52:29.904722+0000","last_peered":"2024-03-24T12:52:29.904722+0000","last_clean":"2024-03-24T12:52:29.904722+0000","last_became_active":"2024-03-24T12:50:27.789008+0000","last_became_peered":"2024-03-24T12:50:27.789008+0000","last_unstale":"2024-03-24T12:52:29.904722+0000","last_undegraded":"2024-03-24T12:52:29.904722+0000","last_fullsized":"2024-03-24T12:52:29.904722+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:14.218930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:37.376732+0000","last_clean_scrub_stamp":"2024-03-24T12:52:14.218930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:33.319542+0000","scrub_duration":74,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":938,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473672+0000","last_change":"2024-03-24T12:51:50.930206+0000","last_active":"2024-03-24T12:52:29.473672+0000","last_peered":"2024-03-24T12:52:29.473672+0000","last_clean":"2024-03-24T12:52:29.473672+0000","last_became_active":"2024-03-24T12:50:27.805167+0000","last_became_peered":"2024-03-24T12:50:27.805167+0000","last_unstale":"2024-03-24T12:52:29.473672+0000","last_undegraded":"2024-03-24T12:52:29.473672+0000","last_fullsized":"2024-03-24T12:52:29.473672+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:50.930107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:50.930107+0000","last_clean_scrub_stamp":"2024-03-24T12:51:50.930107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:52.705763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"390'26","reported_seq":1019,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903059+0000","last_change":"2024-03-24T12:51:35.889121+0000","last_active":"2024-03-24T12:52:29.903059+0000","last_peered":"2024-03-24T12:52:29.903059+0000","last_clean":"2024-03-24T12:52:29.903059+0000","last_became_active":"2024-03-24T12:50:27.806165+0000","last_became_peered":"2024-03-24T12:50:27.806165+0000","last_unstale":"2024-03-24T12:52:29.903059+0000","last_undegraded":"2024-03-24T12:52:29.903059+0000","last_fullsized":"2024-03-24T12:52:29.903059+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'26","last_scrub_stamp":"2024-03-24T12:51:35.889000+0000","last_deep_scrub":"207'22","last_deep_scrub_stamp":"2024-03-24T12:44:50.024489+0000","last_clean_scrub_stamp":"2024-03-24T12:51:35.889000+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:44.894998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":991,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472549+0000","last_change":"2024-03-24T12:51:41.986847+0000","last_active":"2024-03-24T12:52:29.472549+0000","last_peered":"2024-03-24T12:52:29.472549+0000","last_clean":"2024-03-24T12:52:29.472549+0000","last_became_active":"2024-03-24T12:51:04.945086+0000","last_became_peered":"2024-03-24T12:51:04.945086+0000","last_unstale":"2024-03-24T12:52:29.472549+0000","last_undegraded":"2024-03-24T12:52:29.472549+0000","last_fullsized":"2024-03-24T12:52:29.472549+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:41.986754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:41.986754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:58.940084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0019260880000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":947,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902751+0000","last_change":"2024-03-24T12:51:20.006047+0000","last_active":"2024-03-24T12:52:29.902751+0000","last_peered":"2024-03-24T12:52:29.902751+0000","last_clean":"2024-03-24T12:52:29.902751+0000","last_became_active":"2024-03-24T12:50:52.889431+0000","last_became_peered":"2024-03-24T12:50:52.889431+0000","last_unstale":"2024-03-24T12:52:29.902751+0000","last_undegraded":"2024-03-24T12:52:29.902751+0000","last_fullsized":"2024-03-24T12:52:29.902751+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:20.005944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:55.977877+0000","last_clean_scrub_stamp":"2024-03-24T12:51:20.005944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:39.280301+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":975,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904095+0000","last_change":"2024-03-24T12:52:12.776812+0000","last_active":"2024-03-24T12:52:29.904095+0000","last_peered":"2024-03-24T12:52:29.904095+0000","last_clean":"2024-03-24T12:52:29.904095+0000","last_became_active":"2024-03-24T12:50:31.827081+0000","last_became_peered":"2024-03-24T12:50:31.827081+0000","last_unstale":"2024-03-24T12:52:29.904095+0000","last_undegraded":"2024-03-24T12:52:29.904095+0000","last_fullsized":"2024-03-24T12:52:29.904095+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:12.776703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:15.087115+0000","last_clean_scrub_stamp":"2024-03-24T12:52:12.776703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:32.373142+0000","scrub_duration":63,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.2","version":"83'128","reported_seq":633,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:35.839738+0000","last_change":"2024-03-24T12:52:35.839767+0000","last_active":"2024-03-24T12:52:35.839738+0000","last_peered":"2024-03-24T12:52:35.839738+0000","last_clean":"2024-03-24T12:52:35.839738+0000","last_became_active":"2024-03-24T12:51:32.332849+0000","last_became_peered":"2024-03-24T12:51:32.332849+0000","last_unstale":"2024-03-24T12:52:35.839738+0000","last_undegraded":"2024-03-24T12:52:35.839738+0000","last_fullsized":"2024-03-24T12:52:35.839738+0000","mapping_epoch":469,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":470,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:52:35.839658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:52:35.839658+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:37.239937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":853,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473299+0000","last_change":"2024-03-24T12:51:35.648398+0000","last_active":"2024-03-24T12:52:29.473299+0000","last_peered":"2024-03-24T12:52:29.473299+0000","last_clean":"2024-03-24T12:52:29.473299+0000","last_became_active":"2024-03-24T12:49:54.344124+0000","last_became_peered":"2024-03-24T12:49:54.344124+0000","last_unstale":"2024-03-24T12:52:29.473299+0000","last_undegraded":"2024-03-24T12:52:29.473299+0000","last_fullsized":"2024-03-24T12:52:29.473299+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:35.648299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:35.648299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:39.110905+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"459'67","reported_seq":1115,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903498+0000","last_change":"2024-03-24T12:52:01.846412+0000","last_active":"2024-03-24T12:52:29.903498+0000","last_peered":"2024-03-24T12:52:29.903498+0000","last_clean":"2024-03-24T12:52:29.903498+0000","last_became_active":"2024-03-24T12:51:04.944772+0000","last_became_peered":"2024-03-24T12:51:04.944772+0000","last_unstale":"2024-03-24T12:52:29.903498+0000","last_undegraded":"2024-03-24T12:52:29.903498+0000","last_fullsized":"2024-03-24T12:52:29.903498+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'67","last_scrub_stamp":"2024-03-24T12:52:01.846244+0000","last_deep_scrub":"248'51","last_deep_scrub_stamp":"2024-03-24T12:39:47.606538+0000","last_clean_scrub_stamp":"2024-03-24T12:52:01.846244+0000","objects_scrubbed":10,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:13.523680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":82,"num_read_kb":67,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":980,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473117+0000","last_change":"2024-03-24T12:52:18.139530+0000","last_active":"2024-03-24T12:52:29.473117+0000","last_peered":"2024-03-24T12:52:29.473117+0000","last_clean":"2024-03-24T12:52:29.473117+0000","last_became_active":"2024-03-24T12:50:31.815376+0000","last_became_peered":"2024-03-24T12:50:31.815376+0000","last_unstale":"2024-03-24T12:52:29.473117+0000","last_undegraded":"2024-03-24T12:52:29.473117+0000","last_fullsized":"2024-03-24T12:52:29.473117+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:18.139435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:52:18.139435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:31.841962+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"71'2","reported_seq":132,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903441+0000","last_change":"2024-03-24T12:52:18.111218+0000","last_active":"2024-03-24T12:52:29.903441+0000","last_peered":"2024-03-24T12:52:29.903441+0000","last_clean":"2024-03-24T12:52:29.903441+0000","last_became_active":"2024-03-24T12:50:31.819931+0000","last_became_peered":"2024-03-24T12:50:31.819931+0000","last_unstale":"2024-03-24T12:52:29.903441+0000","last_undegraded":"2024-03-24T12:52:29.903441+0000","last_fullsized":"2024-03-24T12:52:29.903441+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-24T12:52:18.111114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:52:18.111114+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:24.490924+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00072700900000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":85,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903156+0000","last_change":"2024-03-24T12:52:25.148398+0000","last_active":"2024-03-24T12:52:29.903156+0000","last_peered":"2024-03-24T12:52:29.903156+0000","last_clean":"2024-03-24T12:52:29.903156+0000","last_became_active":"2024-03-24T12:51:04.947211+0000","last_became_peered":"2024-03-24T12:51:04.947211+0000","last_unstale":"2024-03-24T12:52:29.903156+0000","last_undegraded":"2024-03-24T12:52:29.903156+0000","last_fullsized":"2024-03-24T12:52:29.903156+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:25.148294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:25.148294+0000","last_clean_scrub_stamp":"2024-03-24T12:52:25.148294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:49.124103+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"83'128","reported_seq":668,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902929+0000","last_change":"2024-03-24T12:51:32.170253+0000","last_active":"2024-03-24T12:52:29.902929+0000","last_peered":"2024-03-24T12:52:29.902929+0000","last_clean":"2024-03-24T12:52:29.902929+0000","last_became_active":"2024-03-24T12:50:30.800045+0000","last_became_peered":"2024-03-24T12:50:30.800045+0000","last_unstale":"2024-03-24T12:52:29.902929+0000","last_undegraded":"2024-03-24T12:52:29.902929+0000","last_fullsized":"2024-03-24T12:52:29.902929+0000","mapping_epoch":442,"log_start":"0'0","ondisk_log_start":"0'0","created":293,"last_epoch_clean":443,"parent":"0.0","parent_split_bits":4,"last_scrub":"83'128","last_scrub_stamp":"2024-03-24T12:51:32.170145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:23:42.154712+0000","last_clean_scrub_stamp":"2024-03-24T12:51:32.170145+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:56.010658+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"390'21","reported_seq":907,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474064+0000","last_change":"2024-03-24T12:51:21.643588+0000","last_active":"2024-03-24T12:52:29.474064+0000","last_peered":"2024-03-24T12:52:29.474064+0000","last_clean":"2024-03-24T12:52:29.474064+0000","last_became_active":"2024-03-24T12:49:54.357030+0000","last_became_peered":"2024-03-24T12:49:54.357030+0000","last_unstale":"2024-03-24T12:52:29.474064+0000","last_undegraded":"2024-03-24T12:52:29.474064+0000","last_fullsized":"2024-03-24T12:52:29.474064+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:51:21.643496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:21.643496+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:40.112167+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"86'2","reported_seq":133,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903523+0000","last_change":"2024-03-24T12:51:36.185698+0000","last_active":"2024-03-24T12:52:29.903523+0000","last_peered":"2024-03-24T12:52:29.903523+0000","last_clean":"2024-03-24T12:52:29.903523+0000","last_became_active":"2024-03-24T12:51:04.947379+0000","last_became_peered":"2024-03-24T12:51:04.947379+0000","last_unstale":"2024-03-24T12:52:29.903523+0000","last_undegraded":"2024-03-24T12:52:29.903523+0000","last_fullsized":"2024-03-24T12:52:29.903523+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"86'2","last_scrub_stamp":"2024-03-24T12:51:36.185568+0000","last_deep_scrub":"86'2","last_deep_scrub_stamp":"2024-03-24T12:27:09.234299+0000","last_clean_scrub_stamp":"2024-03-24T12:51:36.185568+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:58.237401+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3500,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":968,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472920+0000","last_change":"2024-03-24T12:51:22.009451+0000","last_active":"2024-03-24T12:52:29.472920+0000","last_peered":"2024-03-24T12:52:29.472920+0000","last_clean":"2024-03-24T12:52:29.472920+0000","last_became_active":"2024-03-24T12:51:04.945405+0000","last_became_peered":"2024-03-24T12:51:04.945405+0000","last_unstale":"2024-03-24T12:52:29.472920+0000","last_undegraded":"2024-03-24T12:52:29.472920+0000","last_fullsized":"2024-03-24T12:52:29.472920+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:22.009361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:11.890544+0000","last_clean_scrub_stamp":"2024-03-24T12:51:22.009361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:45.666922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":859,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472596+0000","last_change":"2024-03-24T12:51:54.676419+0000","last_active":"2024-03-24T12:52:29.472596+0000","last_peered":"2024-03-24T12:52:29.472596+0000","last_clean":"2024-03-24T12:52:29.472596+0000","last_became_active":"2024-03-24T12:49:54.344376+0000","last_became_peered":"2024-03-24T12:49:54.344376+0000","last_unstale":"2024-03-24T12:52:29.472596+0000","last_undegraded":"2024-03-24T12:52:29.472596+0000","last_fullsized":"2024-03-24T12:52:29.472596+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:54.676248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:31.678822+0000","last_clean_scrub_stamp":"2024-03-24T12:51:54.676248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:18.893396+0000","scrub_duration":67,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"391'51","reported_seq":120,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835480+0000","last_change":"2024-03-24T12:51:33.431445+0000","last_active":"2024-03-24T12:52:29.835480+0000","last_peered":"2024-03-24T12:52:29.835480+0000","last_clean":"2024-03-24T12:52:29.835480+0000","last_became_active":"2024-03-24T12:50:31.824015+0000","last_became_peered":"2024-03-24T12:50:31.824015+0000","last_unstale":"2024-03-24T12:52:29.835480+0000","last_undegraded":"2024-03-24T12:52:29.835480+0000","last_fullsized":"2024-03-24T12:52:29.835480+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'51","last_scrub_stamp":"2024-03-24T12:51:33.431269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:33.431269+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:46.626660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":982,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903895+0000","last_change":"2024-03-24T12:51:25.784512+0000","last_active":"2024-03-24T12:52:29.903895+0000","last_peered":"2024-03-24T12:52:29.903895+0000","last_clean":"2024-03-24T12:52:29.903895+0000","last_became_active":"2024-03-24T12:50:31.831092+0000","last_became_peered":"2024-03-24T12:50:31.831092+0000","last_unstale":"2024-03-24T12:52:29.903895+0000","last_undegraded":"2024-03-24T12:52:29.903895+0000","last_fullsized":"2024-03-24T12:52:29.903895+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:25.784411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:33:34.204829+0000","last_clean_scrub_stamp":"2024-03-24T12:51:25.784411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:46.642101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":938,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473258+0000","last_change":"2024-03-24T12:51:21.043249+0000","last_active":"2024-03-24T12:52:29.473258+0000","last_peered":"2024-03-24T12:52:29.473258+0000","last_clean":"2024-03-24T12:52:29.473258+0000","last_became_active":"2024-03-24T12:50:27.803159+0000","last_became_peered":"2024-03-24T12:50:27.803159+0000","last_unstale":"2024-03-24T12:52:29.473258+0000","last_undegraded":"2024-03-24T12:52:29.473258+0000","last_fullsized":"2024-03-24T12:52:29.473258+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:21.043170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:21.043170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.936793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":943,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903059+0000","last_change":"2024-03-24T12:52:26.913509+0000","last_active":"2024-03-24T12:52:29.903059+0000","last_peered":"2024-03-24T12:52:29.903059+0000","last_clean":"2024-03-24T12:52:29.903059+0000","last_became_active":"2024-03-24T12:26:09.339696+0000","last_became_peered":"2024-03-24T12:26:09.339696+0000","last_unstale":"2024-03-24T12:52:29.903059+0000","last_undegraded":"2024-03-24T12:52:29.903059+0000","last_fullsized":"2024-03-24T12:52:29.903059+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:26.913402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:26.913402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:32.403316+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":970,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472164+0000","last_change":"2024-03-24T12:52:29.052546+0000","last_active":"2024-03-24T12:52:29.472164+0000","last_peered":"2024-03-24T12:52:29.472164+0000","last_clean":"2024-03-24T12:52:29.472164+0000","last_became_active":"2024-03-24T12:50:49.121399+0000","last_became_peered":"2024-03-24T12:50:49.121399+0000","last_unstale":"2024-03-24T12:52:29.472164+0000","last_undegraded":"2024-03-24T12:52:29.472164+0000","last_fullsized":"2024-03-24T12:52:29.472164+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:29.052442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:52:29.052442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:48.858312+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"390'29","reported_seq":1060,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:36.094577+0000","last_change":"2024-03-24T12:52:36.094613+0000","last_active":"2024-03-24T12:52:36.094577+0000","last_peered":"2024-03-24T12:52:36.094577+0000","last_clean":"2024-03-24T12:52:36.094577+0000","last_became_active":"2024-03-24T12:51:04.951647+0000","last_became_peered":"2024-03-24T12:51:04.951647+0000","last_unstale":"2024-03-24T12:52:36.094577+0000","last_undegraded":"2024-03-24T12:52:36.094577+0000","last_fullsized":"2024-03-24T12:52:36.094577+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'29","last_scrub_stamp":"2024-03-24T12:52:36.094499+0000","last_deep_scrub":"140'17","last_deep_scrub_stamp":"2024-03-24T12:31:25.769706+0000","last_clean_scrub_stamp":"2024-03-24T12:52:36.094499+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:59.346821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":946,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904272+0000","last_change":"2024-03-24T12:52:25.912332+0000","last_active":"2024-03-24T12:52:29.904272+0000","last_peered":"2024-03-24T12:52:29.904272+0000","last_clean":"2024-03-24T12:52:29.904272+0000","last_became_active":"2024-03-24T12:26:03.162288+0000","last_became_peered":"2024-03-24T12:26:03.162288+0000","last_unstale":"2024-03-24T12:52:29.904272+0000","last_undegraded":"2024-03-24T12:52:29.904272+0000","last_fullsized":"2024-03-24T12:52:29.904272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:25.912195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:25.912195+0000","last_clean_scrub_stamp":"2024-03-24T12:52:25.912195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:51.098004+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":950,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904121+0000","last_change":"2024-03-24T12:52:16.275082+0000","last_active":"2024-03-24T12:52:29.904121+0000","last_peered":"2024-03-24T12:52:29.904121+0000","last_clean":"2024-03-24T12:52:29.904121+0000","last_became_active":"2024-03-24T12:51:29.010623+0000","last_became_peered":"2024-03-24T12:51:29.010623+0000","last_unstale":"2024-03-24T12:52:29.904121+0000","last_undegraded":"2024-03-24T12:52:29.904121+0000","last_fullsized":"2024-03-24T12:52:29.904121+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:16.274967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:28.075823+0000","last_clean_scrub_stamp":"2024-03-24T12:52:16.274967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:44.153593+0000","scrub_duration":152,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"445'13","reported_seq":1318,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903847+0000","last_change":"2024-03-24T12:52:13.758958+0000","last_active":"2024-03-24T12:52:29.903847+0000","last_peered":"2024-03-24T12:52:29.903847+0000","last_clean":"2024-03-24T12:52:29.903847+0000","last_became_active":"2024-03-24T12:50:31.835895+0000","last_became_peered":"2024-03-24T12:50:31.835895+0000","last_unstale":"2024-03-24T12:52:29.903847+0000","last_undegraded":"2024-03-24T12:52:29.903847+0000","last_fullsized":"2024-03-24T12:52:29.903847+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'13","last_scrub_stamp":"2024-03-24T12:52:13.758834+0000","last_deep_scrub":"417'12","last_deep_scrub_stamp":"2024-03-24T12:48:19.860403+0000","last_clean_scrub_stamp":"2024-03-24T12:52:13.758834+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:17.767324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"390'39","reported_seq":177,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902695+0000","last_change":"2024-03-24T12:51:57.154175+0000","last_active":"2024-03-24T12:52:29.902695+0000","last_peered":"2024-03-24T12:52:29.902695+0000","last_clean":"2024-03-24T12:52:29.902695+0000","last_became_active":"2024-03-24T12:51:29.011314+0000","last_became_peered":"2024-03-24T12:51:29.011314+0000","last_unstale":"2024-03-24T12:52:29.902695+0000","last_undegraded":"2024-03-24T12:52:29.902695+0000","last_fullsized":"2024-03-24T12:52:29.902695+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'39","last_scrub_stamp":"2024-03-24T12:51:57.154010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:57.154010+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:06.965895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"67'2","reported_seq":959,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904502+0000","last_change":"2024-03-24T12:51:20.818276+0000","last_active":"2024-03-24T12:52:29.904502+0000","last_peered":"2024-03-24T12:52:29.904502+0000","last_clean":"2024-03-24T12:52:29.904502+0000","last_became_active":"2024-03-24T12:49:54.336098+0000","last_became_peered":"2024-03-24T12:49:54.336098+0000","last_unstale":"2024-03-24T12:52:29.904502+0000","last_undegraded":"2024-03-24T12:52:29.904502+0000","last_fullsized":"2024-03-24T12:52:29.904502+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-24T12:51:20.818126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:20.818126+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:30.115519+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":115,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902921+0000","last_change":"2024-03-24T12:51:22.350188+0000","last_active":"2024-03-24T12:52:29.902921+0000","last_peered":"2024-03-24T12:52:29.902921+0000","last_clean":"2024-03-24T12:52:29.902921+0000","last_became_active":"2024-03-24T12:50:31.819654+0000","last_became_peered":"2024-03-24T12:50:31.819654+0000","last_unstale":"2024-03-24T12:52:29.902921+0000","last_undegraded":"2024-03-24T12:52:29.902921+0000","last_fullsized":"2024-03-24T12:52:29.902921+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:22.350074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:07.478619+0000","last_clean_scrub_stamp":"2024-03-24T12:51:22.350074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:34.699867+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"485'21","reported_seq":110,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:33.334614+0000","last_change":"2024-03-24T12:52:33.334648+0000","last_active":"2024-03-24T12:52:33.334614+0000","last_peered":"2024-03-24T12:52:33.334614+0000","last_clean":"2024-03-24T12:52:33.334614+0000","last_became_active":"2024-03-24T12:51:29.010696+0000","last_became_peered":"2024-03-24T12:51:29.010696+0000","last_unstale":"2024-03-24T12:52:33.334614+0000","last_undegraded":"2024-03-24T12:52:33.334614+0000","last_fullsized":"2024-03-24T12:52:33.334614+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'21","last_scrub_stamp":"2024-03-24T12:52:33.334546+0000","last_deep_scrub":"485'21","last_deep_scrub_stamp":"2024-03-24T12:52:33.334546+0000","last_clean_scrub_stamp":"2024-03-24T12:52:33.334546+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:43.314698+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":23,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":100,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835405+0000","last_change":"2024-03-24T12:51:25.514014+0000","last_active":"2024-03-24T12:52:29.835405+0000","last_peered":"2024-03-24T12:52:29.835405+0000","last_clean":"2024-03-24T12:52:29.835405+0000","last_became_active":"2024-03-24T12:50:31.818929+0000","last_became_peered":"2024-03-24T12:50:31.818929+0000","last_unstale":"2024-03-24T12:52:29.835405+0000","last_undegraded":"2024-03-24T12:52:29.835405+0000","last_fullsized":"2024-03-24T12:52:29.835405+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:25.513910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:25.513910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:53.719018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":121,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903924+0000","last_change":"2024-03-24T12:51:47.154108+0000","last_active":"2024-03-24T12:52:29.903924+0000","last_peered":"2024-03-24T12:52:29.903924+0000","last_clean":"2024-03-24T12:52:29.903924+0000","last_became_active":"2024-03-24T12:51:04.946383+0000","last_became_peered":"2024-03-24T12:51:04.946383+0000","last_unstale":"2024-03-24T12:52:29.903924+0000","last_undegraded":"2024-03-24T12:52:29.903924+0000","last_fullsized":"2024-03-24T12:52:29.903924+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:47.153980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:47.153980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:05.489680+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":962,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903181+0000","last_change":"2024-03-24T12:51:22.916093+0000","last_active":"2024-03-24T12:52:29.903181+0000","last_peered":"2024-03-24T12:52:29.903181+0000","last_clean":"2024-03-24T12:52:29.903181+0000","last_became_active":"2024-03-24T12:50:31.830543+0000","last_became_peered":"2024-03-24T12:50:31.830543+0000","last_unstale":"2024-03-24T12:52:29.903181+0000","last_undegraded":"2024-03-24T12:52:29.903181+0000","last_fullsized":"2024-03-24T12:52:29.903181+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:22.915989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:39:55.271679+0000","last_clean_scrub_stamp":"2024-03-24T12:51:22.915989+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"390'21","reported_seq":1056,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474280+0000","last_change":"2024-03-24T12:52:00.557559+0000","last_active":"2024-03-24T12:52:29.474280+0000","last_peered":"2024-03-24T12:52:29.474280+0000","last_clean":"2024-03-24T12:52:29.474280+0000","last_became_active":"2024-03-24T12:51:29.009880+0000","last_became_peered":"2024-03-24T12:51:29.009880+0000","last_unstale":"2024-03-24T12:52:29.474280+0000","last_undegraded":"2024-03-24T12:52:29.474280+0000","last_fullsized":"2024-03-24T12:52:29.474280+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:52:00.557442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:52:00.557442+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:22.074975+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":988,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903517+0000","last_change":"2024-03-24T12:52:24.884788+0000","last_active":"2024-03-24T12:52:29.903517+0000","last_peered":"2024-03-24T12:52:29.903517+0000","last_clean":"2024-03-24T12:52:29.903517+0000","last_became_active":"2024-03-24T12:50:31.830406+0000","last_became_peered":"2024-03-24T12:50:31.830406+0000","last_unstale":"2024-03-24T12:52:29.903517+0000","last_undegraded":"2024-03-24T12:52:29.903517+0000","last_fullsized":"2024-03-24T12:52:29.903517+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:24.884685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:59.056269+0000","last_clean_scrub_stamp":"2024-03-24T12:52:24.884685+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:53.188835+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904632+0000","last_change":"2024-03-24T12:51:31.110791+0000","last_active":"2024-03-24T12:52:29.904632+0000","last_peered":"2024-03-24T12:52:29.904632+0000","last_clean":"2024-03-24T12:52:29.904632+0000","last_became_active":"2024-03-24T12:51:04.951872+0000","last_became_peered":"2024-03-24T12:51:04.951872+0000","last_unstale":"2024-03-24T12:52:29.904632+0000","last_undegraded":"2024-03-24T12:52:29.904632+0000","last_fullsized":"2024-03-24T12:52:29.904632+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:31.110702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:31.110702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:41.855777+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":958,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473955+0000","last_change":"2024-03-24T12:52:01.809237+0000","last_active":"2024-03-24T12:52:29.473955+0000","last_peered":"2024-03-24T12:52:29.473955+0000","last_clean":"2024-03-24T12:52:29.473955+0000","last_became_active":"2024-03-24T12:26:09.342893+0000","last_became_peered":"2024-03-24T12:26:09.342893+0000","last_unstale":"2024-03-24T12:52:29.473955+0000","last_undegraded":"2024-03-24T12:52:29.473955+0000","last_fullsized":"2024-03-24T12:52:29.473955+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:01.809149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:27.660932+0000","last_clean_scrub_stamp":"2024-03-24T12:52:01.809149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:04.495865+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"390'25","reported_seq":119,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904371+0000","last_change":"2024-03-24T12:51:37.172266+0000","last_active":"2024-03-24T12:52:29.904371+0000","last_peered":"2024-03-24T12:52:29.904371+0000","last_clean":"2024-03-24T12:52:29.904371+0000","last_became_active":"2024-03-24T12:50:27.797323+0000","last_became_peered":"2024-03-24T12:50:27.797323+0000","last_unstale":"2024-03-24T12:52:29.904371+0000","last_undegraded":"2024-03-24T12:52:29.904371+0000","last_fullsized":"2024-03-24T12:52:29.904371+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'25","last_scrub_stamp":"2024-03-24T12:51:37.172138+0000","last_deep_scrub":"390'25","last_deep_scrub_stamp":"2024-03-24T12:50:13.115092+0000","last_clean_scrub_stamp":"2024-03-24T12:51:37.172138+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:00.476611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":923,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473355+0000","last_change":"2024-03-24T12:52:06.512987+0000","last_active":"2024-03-24T12:52:29.473355+0000","last_peered":"2024-03-24T12:52:29.473355+0000","last_clean":"2024-03-24T12:52:29.473355+0000","last_became_active":"2024-03-24T12:51:04.939149+0000","last_became_peered":"2024-03-24T12:51:04.939149+0000","last_unstale":"2024-03-24T12:52:29.473355+0000","last_undegraded":"2024-03-24T12:52:29.473355+0000","last_fullsized":"2024-03-24T12:52:29.473355+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:06.512843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:06.512843+0000","last_clean_scrub_stamp":"2024-03-24T12:52:06.512843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:08.997175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"459'11","reported_seq":99,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904540+0000","last_change":"2024-03-24T12:51:40.175588+0000","last_active":"2024-03-24T12:52:29.904540+0000","last_peered":"2024-03-24T12:52:29.904540+0000","last_clean":"2024-03-24T12:52:29.904540+0000","last_became_active":"2024-03-24T12:51:04.947498+0000","last_became_peered":"2024-03-24T12:51:04.947498+0000","last_unstale":"2024-03-24T12:52:29.904540+0000","last_undegraded":"2024-03-24T12:52:29.904540+0000","last_fullsized":"2024-03-24T12:52:29.904540+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"459'11","last_scrub_stamp":"2024-03-24T12:51:40.175438+0000","last_deep_scrub":"93'9","last_deep_scrub_stamp":"2024-03-24T12:27:51.378939+0000","last_clean_scrub_stamp":"2024-03-24T12:51:40.175438+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:05.098888+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":934,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473885+0000","last_change":"2024-03-24T12:52:20.325148+0000","last_active":"2024-03-24T12:52:29.473885+0000","last_peered":"2024-03-24T12:52:29.473885+0000","last_clean":"2024-03-24T12:52:29.473885+0000","last_became_active":"2024-03-24T12:50:27.790028+0000","last_became_peered":"2024-03-24T12:50:27.790028+0000","last_unstale":"2024-03-24T12:52:29.473885+0000","last_undegraded":"2024-03-24T12:52:29.473885+0000","last_fullsized":"2024-03-24T12:52:29.473885+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:20.325040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:20.325040+0000","last_clean_scrub_stamp":"2024-03-24T12:52:20.325040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:33.789908+0000","scrub_duration":179,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"480'35","reported_seq":992,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904595+0000","last_change":"2024-03-24T12:52:13.153802+0000","last_active":"2024-03-24T12:52:29.904595+0000","last_peered":"2024-03-24T12:52:29.904595+0000","last_clean":"2024-03-24T12:52:29.904595+0000","last_became_active":"2024-03-24T12:26:05.357989+0000","last_became_peered":"2024-03-24T12:26:05.357989+0000","last_unstale":"2024-03-24T12:52:29.904595+0000","last_undegraded":"2024-03-24T12:52:29.904595+0000","last_fullsized":"2024-03-24T12:52:29.904595+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'35","last_scrub_stamp":"2024-03-24T12:52:13.153700+0000","last_deep_scrub":"480'35","last_deep_scrub_stamp":"2024-03-24T12:52:13.153700+0000","last_clean_scrub_stamp":"2024-03-24T12:52:13.153700+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:26.584964+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":949,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904053+0000","last_change":"2024-03-24T12:52:07.808986+0000","last_active":"2024-03-24T12:52:29.904053+0000","last_peered":"2024-03-24T12:52:29.904053+0000","last_clean":"2024-03-24T12:52:29.904053+0000","last_became_active":"2024-03-24T12:26:03.163259+0000","last_became_peered":"2024-03-24T12:26:03.163259+0000","last_unstale":"2024-03-24T12:52:29.904053+0000","last_undegraded":"2024-03-24T12:52:29.904053+0000","last_fullsized":"2024-03-24T12:52:29.904053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:07.808884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:07.808884+0000","last_clean_scrub_stamp":"2024-03-24T12:52:07.808884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:20.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"440'14","reported_seq":148,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904635+0000","last_change":"2024-03-24T12:52:05.185347+0000","last_active":"2024-03-24T12:52:29.904635+0000","last_peered":"2024-03-24T12:52:29.904635+0000","last_clean":"2024-03-24T12:52:29.904635+0000","last_became_active":"2024-03-24T12:50:27.804040+0000","last_became_peered":"2024-03-24T12:50:27.804040+0000","last_unstale":"2024-03-24T12:52:29.904635+0000","last_undegraded":"2024-03-24T12:52:29.904635+0000","last_fullsized":"2024-03-24T12:52:29.904635+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'14","last_scrub_stamp":"2024-03-24T12:52:05.185248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:52:05.185248+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:12.355695+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"74'1","reported_seq":873,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474350+0000","last_change":"2024-03-24T12:52:15.419360+0000","last_active":"2024-03-24T12:52:29.474350+0000","last_peered":"2024-03-24T12:52:29.474350+0000","last_clean":"2024-03-24T12:52:29.474350+0000","last_became_active":"2024-03-24T12:51:04.939419+0000","last_became_peered":"2024-03-24T12:51:04.939419+0000","last_unstale":"2024-03-24T12:52:29.474350+0000","last_undegraded":"2024-03-24T12:52:29.474350+0000","last_fullsized":"2024-03-24T12:52:29.474350+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:52:15.419178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:15.419178+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:33.212605+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"390'32","reported_seq":1070,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473389+0000","last_change":"2024-03-24T12:52:28.232357+0000","last_active":"2024-03-24T12:52:29.473389+0000","last_peered":"2024-03-24T12:52:29.473389+0000","last_clean":"2024-03-24T12:52:29.473389+0000","last_became_active":"2024-03-24T12:50:31.824602+0000","last_became_peered":"2024-03-24T12:50:31.824602+0000","last_unstale":"2024-03-24T12:52:29.473389+0000","last_undegraded":"2024-03-24T12:52:29.473389+0000","last_fullsized":"2024-03-24T12:52:29.473389+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'32","last_scrub_stamp":"2024-03-24T12:52:28.232237+0000","last_deep_scrub":"88'16","last_deep_scrub_stamp":"2024-03-24T12:27:17.084148+0000","last_clean_scrub_stamp":"2024-03-24T12:52:28.232237+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:40.554584+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":970,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473428+0000","last_change":"2024-03-24T12:51:24.144859+0000","last_active":"2024-03-24T12:52:29.473428+0000","last_peered":"2024-03-24T12:52:29.473428+0000","last_clean":"2024-03-24T12:52:29.473428+0000","last_became_active":"2024-03-24T12:50:31.813180+0000","last_became_peered":"2024-03-24T12:50:31.813180+0000","last_unstale":"2024-03-24T12:52:29.473428+0000","last_undegraded":"2024-03-24T12:52:29.473428+0000","last_fullsized":"2024-03-24T12:52:29.473428+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:24.144684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:32.853388+0000","last_clean_scrub_stamp":"2024-03-24T12:51:24.144684+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:52.372455+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":957,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903610+0000","last_change":"2024-03-24T12:52:29.876547+0000","last_active":"2024-03-24T12:52:29.903610+0000","last_peered":"2024-03-24T12:52:29.903610+0000","last_clean":"2024-03-24T12:52:29.903610+0000","last_became_active":"2024-03-24T12:51:04.939496+0000","last_became_peered":"2024-03-24T12:51:04.939496+0000","last_unstale":"2024-03-24T12:52:29.903610+0000","last_undegraded":"2024-03-24T12:52:29.903610+0000","last_fullsized":"2024-03-24T12:52:29.903610+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:29.876464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:31:10.144101+0000","last_clean_scrub_stamp":"2024-03-24T12:52:29.876464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:44.738196+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903907+0000","last_change":"2024-03-24T12:51:41.197640+0000","last_active":"2024-03-24T12:52:29.903907+0000","last_peered":"2024-03-24T12:52:29.903907+0000","last_clean":"2024-03-24T12:52:29.903907+0000","last_became_active":"2024-03-24T12:51:04.947440+0000","last_became_peered":"2024-03-24T12:51:04.947440+0000","last_unstale":"2024-03-24T12:52:29.903907+0000","last_undegraded":"2024-03-24T12:52:29.903907+0000","last_fullsized":"2024-03-24T12:52:29.903907+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:41.197541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:41.197541+0000","last_clean_scrub_stamp":"2024-03-24T12:51:41.197541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:01.047569+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"390'21","reported_seq":366,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902804+0000","last_change":"2024-03-24T12:51:24.020873+0000","last_active":"2024-03-24T12:52:29.902804+0000","last_peered":"2024-03-24T12:52:29.902804+0000","last_clean":"2024-03-24T12:52:29.902804+0000","last_became_active":"2024-03-24T12:50:31.811177+0000","last_became_peered":"2024-03-24T12:50:31.811177+0000","last_unstale":"2024-03-24T12:52:29.902804+0000","last_undegraded":"2024-03-24T12:52:29.902804+0000","last_fullsized":"2024-03-24T12:52:29.902804+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:51:24.020758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:24.020758+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:49.568763+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":954,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903357+0000","last_change":"2024-03-24T12:52:20.809516+0000","last_active":"2024-03-24T12:52:29.903357+0000","last_peered":"2024-03-24T12:52:29.903357+0000","last_clean":"2024-03-24T12:52:29.903357+0000","last_became_active":"2024-03-24T12:26:03.161061+0000","last_became_peered":"2024-03-24T12:26:03.161061+0000","last_unstale":"2024-03-24T12:52:29.903357+0000","last_undegraded":"2024-03-24T12:52:29.903357+0000","last_fullsized":"2024-03-24T12:52:29.903357+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:20.809395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:28.831117+0000","last_clean_scrub_stamp":"2024-03-24T12:52:20.809395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:29.857567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":939,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903976+0000","last_change":"2024-03-24T12:52:10.089536+0000","last_active":"2024-03-24T12:52:29.903976+0000","last_peered":"2024-03-24T12:52:29.903976+0000","last_clean":"2024-03-24T12:52:29.903976+0000","last_became_active":"2024-03-24T12:49:54.340307+0000","last_became_peered":"2024-03-24T12:49:54.340307+0000","last_unstale":"2024-03-24T12:52:29.903976+0000","last_undegraded":"2024-03-24T12:52:29.903976+0000","last_fullsized":"2024-03-24T12:52:29.903976+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:10.089444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:28.070148+0000","last_clean_scrub_stamp":"2024-03-24T12:52:10.089444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:27.544845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":857,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472597+0000","last_change":"2024-03-24T12:51:40.601123+0000","last_active":"2024-03-24T12:52:29.472597+0000","last_peered":"2024-03-24T12:52:29.472597+0000","last_clean":"2024-03-24T12:52:29.472597+0000","last_became_active":"2024-03-24T12:50:27.812903+0000","last_became_peered":"2024-03-24T12:50:27.812903+0000","last_unstale":"2024-03-24T12:52:29.472597+0000","last_undegraded":"2024-03-24T12:52:29.472597+0000","last_fullsized":"2024-03-24T12:52:29.472597+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:40.601022+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:53.058946+0000","last_clean_scrub_stamp":"2024-03-24T12:51:40.601022+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:53.339776+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"390'27","reported_seq":1054,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904152+0000","last_change":"2024-03-24T12:51:30.121983+0000","last_active":"2024-03-24T12:52:29.904152+0000","last_peered":"2024-03-24T12:52:29.904152+0000","last_clean":"2024-03-24T12:52:29.904152+0000","last_became_active":"2024-03-24T12:50:31.810900+0000","last_became_peered":"2024-03-24T12:50:31.810900+0000","last_unstale":"2024-03-24T12:52:29.904152+0000","last_undegraded":"2024-03-24T12:52:29.904152+0000","last_fullsized":"2024-03-24T12:52:29.904152+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'27","last_scrub_stamp":"2024-03-24T12:51:30.121864+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:30.121864+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:47.551603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"427'2","reported_seq":121,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903530+0000","last_change":"2024-03-24T12:52:25.117369+0000","last_active":"2024-03-24T12:52:29.903530+0000","last_peered":"2024-03-24T12:52:29.903530+0000","last_clean":"2024-03-24T12:52:29.903530+0000","last_became_active":"2024-03-24T12:50:27.803506+0000","last_became_peered":"2024-03-24T12:50:27.803506+0000","last_unstale":"2024-03-24T12:52:29.903530+0000","last_undegraded":"2024-03-24T12:52:29.903530+0000","last_fullsized":"2024-03-24T12:52:29.903530+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"427'2","last_scrub_stamp":"2024-03-24T12:52:25.117265+0000","last_deep_scrub":"427'2","last_deep_scrub_stamp":"2024-03-24T12:49:18.871645+0000","last_clean_scrub_stamp":"2024-03-24T12:52:25.117265+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:31.450317+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1194,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":971,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903817+0000","last_change":"2024-03-24T12:51:52.980296+0000","last_active":"2024-03-24T12:52:29.903817+0000","last_peered":"2024-03-24T12:52:29.903817+0000","last_clean":"2024-03-24T12:52:29.903817+0000","last_became_active":"2024-03-24T12:50:31.827976+0000","last_became_peered":"2024-03-24T12:50:31.827976+0000","last_unstale":"2024-03-24T12:52:29.903817+0000","last_undegraded":"2024-03-24T12:52:29.903817+0000","last_fullsized":"2024-03-24T12:52:29.903817+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:52.980202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:52.980202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:09.905656+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903718+0000","last_change":"2024-03-24T12:51:19.089676+0000","last_active":"2024-03-24T12:52:29.903718+0000","last_peered":"2024-03-24T12:52:29.903718+0000","last_clean":"2024-03-24T12:52:29.903718+0000","last_became_active":"2024-03-24T12:51:04.950249+0000","last_became_peered":"2024-03-24T12:51:04.950249+0000","last_unstale":"2024-03-24T12:52:29.903718+0000","last_undegraded":"2024-03-24T12:52:29.903718+0000","last_fullsized":"2024-03-24T12:52:29.903718+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:19.089546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:19.089546+0000","last_clean_scrub_stamp":"2024-03-24T12:51:19.089546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:32.391246+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"390'12","reported_seq":83,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:34.316074+0000","last_change":"2024-03-24T12:52:34.316102+0000","last_active":"2024-03-24T12:52:34.316074+0000","last_peered":"2024-03-24T12:52:34.316074+0000","last_clean":"2024-03-24T12:52:34.316074+0000","last_became_active":"2024-03-24T12:51:04.956111+0000","last_became_peered":"2024-03-24T12:51:04.956111+0000","last_unstale":"2024-03-24T12:52:34.316074+0000","last_undegraded":"2024-03-24T12:52:34.316074+0000","last_fullsized":"2024-03-24T12:52:34.316074+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'12","last_scrub_stamp":"2024-03-24T12:52:34.316008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:52:34.316008+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:59.051725+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":915,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473818+0000","last_change":"2024-03-24T12:51:59.523110+0000","last_active":"2024-03-24T12:52:29.473818+0000","last_peered":"2024-03-24T12:52:29.473818+0000","last_clean":"2024-03-24T12:52:29.473818+0000","last_became_active":"2024-03-24T12:51:04.938167+0000","last_became_peered":"2024-03-24T12:51:04.938167+0000","last_unstale":"2024-03-24T12:52:29.473818+0000","last_undegraded":"2024-03-24T12:52:29.473818+0000","last_fullsized":"2024-03-24T12:52:29.473818+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:59.522983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:02.143860+0000","last_clean_scrub_stamp":"2024-03-24T12:51:59.522983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:27.336806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":108,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835359+0000","last_change":"2024-03-24T12:51:52.176344+0000","last_active":"2024-03-24T12:52:29.835359+0000","last_peered":"2024-03-24T12:52:29.835359+0000","last_clean":"2024-03-24T12:52:29.835359+0000","last_became_active":"2024-03-24T12:50:31.824539+0000","last_became_peered":"2024-03-24T12:50:31.824539+0000","last_unstale":"2024-03-24T12:52:29.835359+0000","last_undegraded":"2024-03-24T12:52:29.835359+0000","last_fullsized":"2024-03-24T12:52:29.835359+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:52.176209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:52.176209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:21.605172+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":934,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903880+0000","last_change":"2024-03-24T12:51:19.869035+0000","last_active":"2024-03-24T12:52:29.903880+0000","last_peered":"2024-03-24T12:52:29.903880+0000","last_clean":"2024-03-24T12:52:29.903880+0000","last_became_active":"2024-03-24T12:26:09.344911+0000","last_became_peered":"2024-03-24T12:26:09.344911+0000","last_unstale":"2024-03-24T12:52:29.903880+0000","last_undegraded":"2024-03-24T12:52:29.903880+0000","last_fullsized":"2024-03-24T12:52:29.903880+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:19.868931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:52.028896+0000","last_clean_scrub_stamp":"2024-03-24T12:51:19.868931+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:33.722290+0000","scrub_duration":76,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1f","version":"390'27","reported_seq":113,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.835913+0000","last_change":"2024-03-24T12:51:40.429566+0000","last_active":"2024-03-24T12:52:29.835913+0000","last_peered":"2024-03-24T12:52:29.835913+0000","last_clean":"2024-03-24T12:52:29.835913+0000","last_became_active":"2024-03-24T12:50:31.816089+0000","last_became_peered":"2024-03-24T12:50:31.816089+0000","last_unstale":"2024-03-24T12:52:29.835913+0000","last_undegraded":"2024-03-24T12:52:29.835913+0000","last_fullsized":"2024-03-24T12:52:29.835913+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'27","last_scrub_stamp":"2024-03-24T12:51:40.429472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:40.429472+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:49.619158+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902878+0000","last_change":"2024-03-24T12:52:21.048054+0000","last_active":"2024-03-24T12:52:29.902878+0000","last_peered":"2024-03-24T12:52:29.902878+0000","last_clean":"2024-03-24T12:52:29.902878+0000","last_became_active":"2024-03-24T12:51:04.954448+0000","last_became_peered":"2024-03-24T12:51:04.954448+0000","last_unstale":"2024-03-24T12:52:29.902878+0000","last_undegraded":"2024-03-24T12:52:29.902878+0000","last_fullsized":"2024-03-24T12:52:29.902878+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:21.047970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:21.047970+0000","last_clean_scrub_stamp":"2024-03-24T12:52:21.047970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:43.729635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":853,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474150+0000","last_change":"2024-03-24T12:51:55.619431+0000","last_active":"2024-03-24T12:52:29.474150+0000","last_peered":"2024-03-24T12:52:29.474150+0000","last_clean":"2024-03-24T12:52:29.474150+0000","last_became_active":"2024-03-24T12:49:54.353078+0000","last_became_peered":"2024-03-24T12:49:54.353078+0000","last_unstale":"2024-03-24T12:52:29.474150+0000","last_undegraded":"2024-03-24T12:52:29.474150+0000","last_fullsized":"2024-03-24T12:52:29.474150+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:55.619346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:55.619346+0000","last_clean_scrub_stamp":"2024-03-24T12:51:55.619346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:05.660350+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":613,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904299+0000","last_change":"2024-03-24T12:52:15.115619+0000","last_active":"2024-03-24T12:52:29.904299+0000","last_peered":"2024-03-24T12:52:29.904299+0000","last_clean":"2024-03-24T12:52:29.904299+0000","last_became_active":"2024-03-24T12:50:53.093060+0000","last_became_peered":"2024-03-24T12:50:53.093060+0000","last_unstale":"2024-03-24T12:52:29.904299+0000","last_undegraded":"2024-03-24T12:52:29.904299+0000","last_fullsized":"2024-03-24T12:52:29.904299+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:15.115522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:15.115522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:30.161569+0000","scrub_duration":219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"390'30","reported_seq":1002,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474282+0000","last_change":"2024-03-24T12:51:30.654950+0000","last_active":"2024-03-24T12:52:29.474282+0000","last_peered":"2024-03-24T12:52:29.474282+0000","last_clean":"2024-03-24T12:52:29.474282+0000","last_became_active":"2024-03-24T12:51:29.009891+0000","last_became_peered":"2024-03-24T12:51:29.009891+0000","last_unstale":"2024-03-24T12:52:29.474282+0000","last_undegraded":"2024-03-24T12:52:29.474282+0000","last_fullsized":"2024-03-24T12:52:29.474282+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'30","last_scrub_stamp":"2024-03-24T12:51:30.654811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:30.654811+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:54.184830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"392'2","reported_seq":971,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902930+0000","last_change":"2024-03-24T12:51:55.761073+0000","last_active":"2024-03-24T12:52:29.902930+0000","last_peered":"2024-03-24T12:52:29.902930+0000","last_clean":"2024-03-24T12:52:29.902930+0000","last_became_active":"2024-03-24T12:26:03.164451+0000","last_became_peered":"2024-03-24T12:26:03.164451+0000","last_unstale":"2024-03-24T12:52:29.902930+0000","last_undegraded":"2024-03-24T12:52:29.902930+0000","last_fullsized":"2024-03-24T12:52:29.902930+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"392'2","last_scrub_stamp":"2024-03-24T12:51:55.760985+0000","last_deep_scrub":"392'2","last_deep_scrub_stamp":"2024-03-24T12:46:18.735806+0000","last_clean_scrub_stamp":"2024-03-24T12:51:55.760985+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:19.445802+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"467'14","reported_seq":149,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:38.521750+0000","last_change":"2024-03-24T12:52:10.916938+0000","last_active":"2024-03-24T12:52:38.521750+0000","last_peered":"2024-03-24T12:52:38.521750+0000","last_clean":"2024-03-24T12:52:38.521750+0000","last_became_active":"2024-03-24T12:51:29.007730+0000","last_became_peered":"2024-03-24T12:51:29.007730+0000","last_unstale":"2024-03-24T12:52:38.521750+0000","last_undegraded":"2024-03-24T12:52:38.521750+0000","last_fullsized":"2024-03-24T12:52:38.521750+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'14","last_scrub_stamp":"2024-03-24T12:52:10.916780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:52:10.916780+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:30.192552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":943,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:30.877420+0000","last_change":"2024-03-24T12:52:30.877452+0000","last_active":"2024-03-24T12:52:30.877420+0000","last_peered":"2024-03-24T12:52:30.877420+0000","last_clean":"2024-03-24T12:52:30.877420+0000","last_became_active":"2024-03-24T12:50:27.792660+0000","last_became_peered":"2024-03-24T12:50:27.792660+0000","last_unstale":"2024-03-24T12:52:30.877420+0000","last_undegraded":"2024-03-24T12:52:30.877420+0000","last_fullsized":"2024-03-24T12:52:30.877420+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-24T12:52:30.877340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:30.877340+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:50.065177+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"391'24","reported_seq":89,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904846+0000","last_change":"2024-03-24T12:52:20.035072+0000","last_active":"2024-03-24T12:52:29.904846+0000","last_peered":"2024-03-24T12:52:29.904846+0000","last_clean":"2024-03-24T12:52:29.904846+0000","last_became_active":"2024-03-24T12:51:04.954618+0000","last_became_peered":"2024-03-24T12:51:04.954618+0000","last_unstale":"2024-03-24T12:52:29.904846+0000","last_undegraded":"2024-03-24T12:52:29.904846+0000","last_fullsized":"2024-03-24T12:52:29.904846+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'24","last_scrub_stamp":"2024-03-24T12:52:20.034924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:52:20.034924+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:29.781477+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":945,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473178+0000","last_change":"2024-03-24T12:52:24.210026+0000","last_active":"2024-03-24T12:52:29.473178+0000","last_peered":"2024-03-24T12:52:29.473178+0000","last_clean":"2024-03-24T12:52:29.473178+0000","last_became_active":"2024-03-24T12:26:03.160347+0000","last_became_peered":"2024-03-24T12:26:03.160347+0000","last_unstale":"2024-03-24T12:52:29.473178+0000","last_undegraded":"2024-03-24T12:52:29.473178+0000","last_fullsized":"2024-03-24T12:52:29.473178+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:24.209940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:52:24.209940+0000","last_clean_scrub_stamp":"2024-03-24T12:52:24.209940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:43.795213+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":937,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473551+0000","last_change":"2024-03-24T12:52:03.225369+0000","last_active":"2024-03-24T12:52:29.473551+0000","last_peered":"2024-03-24T12:52:29.473551+0000","last_clean":"2024-03-24T12:52:29.473551+0000","last_became_active":"2024-03-24T12:26:07.333538+0000","last_became_peered":"2024-03-24T12:26:07.333538+0000","last_unstale":"2024-03-24T12:52:29.473551+0000","last_undegraded":"2024-03-24T12:52:29.473551+0000","last_fullsized":"2024-03-24T12:52:29.473551+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:03.225280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:39.295864+0000","last_clean_scrub_stamp":"2024-03-24T12:52:03.225280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:09.177641+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":964,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904780+0000","last_change":"2024-03-24T12:52:19.228835+0000","last_active":"2024-03-24T12:52:29.904780+0000","last_peered":"2024-03-24T12:52:29.904780+0000","last_clean":"2024-03-24T12:52:29.904780+0000","last_became_active":"2024-03-24T12:50:31.810641+0000","last_became_peered":"2024-03-24T12:50:31.810641+0000","last_unstale":"2024-03-24T12:52:29.904780+0000","last_undegraded":"2024-03-24T12:52:29.904780+0000","last_fullsized":"2024-03-24T12:52:29.904780+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:19.228735+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:19.228735+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:39.883765+0000","scrub_duration":153,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"390'39","reported_seq":1063,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473746+0000","last_change":"2024-03-24T12:51:46.965544+0000","last_active":"2024-03-24T12:52:29.473746+0000","last_peered":"2024-03-24T12:52:29.473746+0000","last_clean":"2024-03-24T12:52:29.473746+0000","last_became_active":"2024-03-24T12:51:04.946216+0000","last_became_peered":"2024-03-24T12:51:04.946216+0000","last_unstale":"2024-03-24T12:52:29.473746+0000","last_undegraded":"2024-03-24T12:52:29.473746+0000","last_fullsized":"2024-03-24T12:52:29.473746+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'39","last_scrub_stamp":"2024-03-24T12:51:46.965416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:46.965416+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:48.057143+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":926,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.474339+0000","last_change":"2024-03-24T12:52:14.372276+0000","last_active":"2024-03-24T12:52:29.474339+0000","last_peered":"2024-03-24T12:52:29.474339+0000","last_clean":"2024-03-24T12:52:29.474339+0000","last_became_active":"2024-03-24T12:50:48.867700+0000","last_became_peered":"2024-03-24T12:50:48.867700+0000","last_unstale":"2024-03-24T12:52:29.474339+0000","last_undegraded":"2024-03-24T12:52:29.474339+0000","last_fullsized":"2024-03-24T12:52:29.474339+0000","mapping_epoch":449,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":450,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:14.372001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:28.360926+0000","last_clean_scrub_stamp":"2024-03-24T12:52:14.372001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:26.161574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":970,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904200+0000","last_change":"2024-03-24T12:51:53.819927+0000","last_active":"2024-03-24T12:52:29.904200+0000","last_peered":"2024-03-24T12:52:29.904200+0000","last_clean":"2024-03-24T12:52:29.904200+0000","last_became_active":"2024-03-24T12:49:54.354682+0000","last_became_peered":"2024-03-24T12:49:54.354682+0000","last_unstale":"2024-03-24T12:52:29.904200+0000","last_undegraded":"2024-03-24T12:52:29.904200+0000","last_fullsized":"2024-03-24T12:52:29.904200+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:53.819837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:46:15.638516+0000","last_clean_scrub_stamp":"2024-03-24T12:51:53.819837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:56.226242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":980,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473560+0000","last_change":"2024-03-24T12:51:41.241353+0000","last_active":"2024-03-24T12:52:29.473560+0000","last_peered":"2024-03-24T12:52:29.473560+0000","last_clean":"2024-03-24T12:52:29.473560+0000","last_became_active":"2024-03-24T12:50:31.823435+0000","last_became_peered":"2024-03-24T12:50:31.823435+0000","last_unstale":"2024-03-24T12:52:29.473560+0000","last_undegraded":"2024-03-24T12:52:29.473560+0000","last_fullsized":"2024-03-24T12:52:29.473560+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:41.241241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:41.241241+0000","last_clean_scrub_stamp":"2024-03-24T12:51:41.241241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:11.003582+0000","scrub_duration":58,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1b","version":"390'9","reported_seq":969,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473294+0000","last_change":"2024-03-24T12:52:13.208344+0000","last_active":"2024-03-24T12:52:29.473294+0000","last_peered":"2024-03-24T12:52:29.473294+0000","last_clean":"2024-03-24T12:52:29.473294+0000","last_became_active":"2024-03-24T12:46:47.943482+0000","last_became_peered":"2024-03-24T12:46:47.943482+0000","last_unstale":"2024-03-24T12:52:29.473294+0000","last_undegraded":"2024-03-24T12:52:29.473294+0000","last_fullsized":"2024-03-24T12:52:29.473294+0000","mapping_epoch":400,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":401,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'9","last_scrub_stamp":"2024-03-24T12:52:13.208243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:52:13.208243+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:31.850070+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":117,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903505+0000","last_change":"2024-03-24T12:52:29.182813+0000","last_active":"2024-03-24T12:52:29.903505+0000","last_peered":"2024-03-24T12:52:29.903505+0000","last_clean":"2024-03-24T12:52:29.903505+0000","last_became_active":"2024-03-24T12:50:27.806126+0000","last_became_peered":"2024-03-24T12:50:27.806126+0000","last_unstale":"2024-03-24T12:52:29.903505+0000","last_undegraded":"2024-03-24T12:52:29.903505+0000","last_fullsized":"2024-03-24T12:52:29.903505+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:29.182714+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:10.251686+0000","last_clean_scrub_stamp":"2024-03-24T12:52:29.182714+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:50.217959+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":937,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903642+0000","last_change":"2024-03-24T12:51:54.926712+0000","last_active":"2024-03-24T12:52:29.903642+0000","last_peered":"2024-03-24T12:52:29.903642+0000","last_clean":"2024-03-24T12:52:29.903642+0000","last_became_active":"2024-03-24T12:26:07.339409+0000","last_became_peered":"2024-03-24T12:26:07.339409+0000","last_unstale":"2024-03-24T12:52:29.903642+0000","last_undegraded":"2024-03-24T12:52:29.903642+0000","last_fullsized":"2024-03-24T12:52:29.903642+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:54.926619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:51:54.926619+0000","last_clean_scrub_stamp":"2024-03-24T12:51:54.926619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:02.496301+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":123,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902807+0000","last_change":"2024-03-24T12:52:24.165469+0000","last_active":"2024-03-24T12:52:29.902807+0000","last_peered":"2024-03-24T12:52:29.902807+0000","last_clean":"2024-03-24T12:52:29.902807+0000","last_became_active":"2024-03-24T12:51:04.946329+0000","last_became_peered":"2024-03-24T12:51:04.946329+0000","last_unstale":"2024-03-24T12:52:29.902807+0000","last_undegraded":"2024-03-24T12:52:29.902807+0000","last_fullsized":"2024-03-24T12:52:29.902807+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:24.165340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:48:17.733385+0000","last_clean_scrub_stamp":"2024-03-24T12:52:24.165340+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:34.348927+0000","scrub_duration":86,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"390'21","reported_seq":976,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473109+0000","last_change":"2024-03-24T12:51:32.239405+0000","last_active":"2024-03-24T12:52:29.473109+0000","last_peered":"2024-03-24T12:52:29.473109+0000","last_clean":"2024-03-24T12:52:29.473109+0000","last_became_active":"2024-03-24T12:49:54.334449+0000","last_became_peered":"2024-03-24T12:49:54.334449+0000","last_unstale":"2024-03-24T12:52:29.473109+0000","last_undegraded":"2024-03-24T12:52:29.473109+0000","last_fullsized":"2024-03-24T12:52:29.473109+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:51:32.239283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:32.239283+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:50.617143+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":123,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.904512+0000","last_change":"2024-03-24T12:52:16.072739+0000","last_active":"2024-03-24T12:52:29.904512+0000","last_peered":"2024-03-24T12:52:29.904512+0000","last_clean":"2024-03-24T12:52:29.904512+0000","last_became_active":"2024-03-24T12:50:31.825465+0000","last_became_peered":"2024-03-24T12:50:31.825465+0000","last_unstale":"2024-03-24T12:52:29.904512+0000","last_undegraded":"2024-03-24T12:52:29.904512+0000","last_fullsized":"2024-03-24T12:52:29.904512+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:16.072645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:23.741374+0000","last_clean_scrub_stamp":"2024-03-24T12:52:16.072645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:30.400623+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":123,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903068+0000","last_change":"2024-03-24T12:51:46.136545+0000","last_active":"2024-03-24T12:52:29.903068+0000","last_peered":"2024-03-24T12:52:29.903068+0000","last_clean":"2024-03-24T12:52:29.903068+0000","last_became_active":"2024-03-24T12:51:29.010381+0000","last_became_peered":"2024-03-24T12:51:29.010381+0000","last_unstale":"2024-03-24T12:52:29.903068+0000","last_undegraded":"2024-03-24T12:52:29.903068+0000","last_fullsized":"2024-03-24T12:52:29.903068+0000","mapping_epoch":466,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":467,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:46.136410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:06.312270+0000","last_clean_scrub_stamp":"2024-03-24T12:51:46.136410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:03.165840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":856,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472989+0000","last_change":"2024-03-24T12:51:33.627968+0000","last_active":"2024-03-24T12:52:29.472989+0000","last_peered":"2024-03-24T12:52:29.472989+0000","last_clean":"2024-03-24T12:52:29.472989+0000","last_became_active":"2024-03-24T12:49:54.338779+0000","last_became_peered":"2024-03-24T12:49:54.338779+0000","last_unstale":"2024-03-24T12:52:29.472989+0000","last_undegraded":"2024-03-24T12:52:29.472989+0000","last_fullsized":"2024-03-24T12:52:29.472989+0000","mapping_epoch":433,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":434,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:33.627822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:27:23.289666+0000","last_clean_scrub_stamp":"2024-03-24T12:51:33.627822+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:44.768312+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"391'46","reported_seq":93,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903774+0000","last_change":"2024-03-24T12:51:33.150692+0000","last_active":"2024-03-24T12:52:29.903774+0000","last_peered":"2024-03-24T12:52:29.903774+0000","last_clean":"2024-03-24T12:52:29.903774+0000","last_became_active":"2024-03-24T12:51:04.950754+0000","last_became_peered":"2024-03-24T12:51:04.950754+0000","last_unstale":"2024-03-24T12:52:29.903774+0000","last_undegraded":"2024-03-24T12:52:29.903774+0000","last_fullsized":"2024-03-24T12:52:29.903774+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'46","last_scrub_stamp":"2024-03-24T12:51:33.150579+0000","last_deep_scrub":"207'34","last_deep_scrub_stamp":"2024-03-24T12:45:07.975149+0000","last_clean_scrub_stamp":"2024-03-24T12:51:33.150579+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:52:40.324063+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":85,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:36.295216+0000","last_change":"2024-03-24T12:52:36.295236+0000","last_active":"2024-03-24T12:52:36.295216+0000","last_peered":"2024-03-24T12:52:36.295216+0000","last_clean":"2024-03-24T12:52:36.295216+0000","last_became_active":"2024-03-24T12:51:04.950651+0000","last_became_peered":"2024-03-24T12:51:04.950651+0000","last_unstale":"2024-03-24T12:52:36.295216+0000","last_undegraded":"2024-03-24T12:52:36.295216+0000","last_fullsized":"2024-03-24T12:52:36.295216+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:36.295157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:49:46.052822+0000","last_clean_scrub_stamp":"2024-03-24T12:52:36.295157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:44.462138+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":961,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472769+0000","last_change":"2024-03-24T12:51:44.249303+0000","last_active":"2024-03-24T12:52:29.472769+0000","last_peered":"2024-03-24T12:52:29.472769+0000","last_clean":"2024-03-24T12:52:29.472769+0000","last_became_active":"2024-03-24T12:50:31.812672+0000","last_became_peered":"2024-03-24T12:50:31.812672+0000","last_unstale":"2024-03-24T12:52:29.472769+0000","last_undegraded":"2024-03-24T12:52:29.472769+0000","last_fullsized":"2024-03-24T12:52:29.472769+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:44.249190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:40:17.828354+0000","last_clean_scrub_stamp":"2024-03-24T12:51:44.249190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:08.015177+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":77,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.903806+0000","last_change":"2024-03-24T12:51:45.247226+0000","last_active":"2024-03-24T12:52:29.903806+0000","last_peered":"2024-03-24T12:52:29.903806+0000","last_clean":"2024-03-24T12:52:29.903806+0000","last_became_active":"2024-03-24T12:51:04.943974+0000","last_became_peered":"2024-03-24T12:51:04.943974+0000","last_unstale":"2024-03-24T12:52:29.903806+0000","last_undegraded":"2024-03-24T12:52:29.903806+0000","last_fullsized":"2024-03-24T12:52:29.903806+0000","mapping_epoch":455,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":456,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:45.247095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:48:05.852493+0000","last_clean_scrub_stamp":"2024-03-24T12:51:45.247095+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:00.710445+0000","scrub_duration":143,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"390'21","reported_seq":1055,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472620+0000","last_change":"2024-03-24T12:51:56.221729+0000","last_active":"2024-03-24T12:52:29.472620+0000","last_peered":"2024-03-24T12:52:29.472620+0000","last_clean":"2024-03-24T12:52:29.472620+0000","last_became_active":"2024-03-24T12:50:31.813188+0000","last_became_peered":"2024-03-24T12:50:31.813188+0000","last_unstale":"2024-03-24T12:52:29.472620+0000","last_undegraded":"2024-03-24T12:52:29.472620+0000","last_fullsized":"2024-03-24T12:52:29.472620+0000","mapping_epoch":443,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":444,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'21","last_scrub_stamp":"2024-03-24T12:51:56.221622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:04.297367+0000","last_clean_scrub_stamp":"2024-03-24T12:51:56.221622+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:04.676808+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":903,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.473643+0000","last_change":"2024-03-24T12:51:58.515152+0000","last_active":"2024-03-24T12:52:29.473643+0000","last_peered":"2024-03-24T12:52:29.473643+0000","last_clean":"2024-03-24T12:52:29.473643+0000","last_became_active":"2024-03-24T12:50:28.072789+0000","last_became_peered":"2024-03-24T12:50:28.072789+0000","last_unstale":"2024-03-24T12:52:29.473643+0000","last_undegraded":"2024-03-24T12:52:29.473643+0000","last_fullsized":"2024-03-24T12:52:29.473643+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:51:58.515066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:45:05.825276+0000","last_clean_scrub_stamp":"2024-03-24T12:51:58.515066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:10.418026+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":626,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.902940+0000","last_change":"2024-03-24T12:52:20.949198+0000","last_active":"2024-03-24T12:52:29.902940+0000","last_peered":"2024-03-24T12:52:29.902940+0000","last_clean":"2024-03-24T12:52:29.902940+0000","last_became_active":"2024-03-24T12:50:53.092959+0000","last_became_peered":"2024-03-24T12:50:53.092959+0000","last_unstale":"2024-03-24T12:52:29.902940+0000","last_undegraded":"2024-03-24T12:52:29.902940+0000","last_fullsized":"2024-03-24T12:52:29.902940+0000","mapping_epoch":451,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":452,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:20.949077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:50:55.905225+0000","last_clean_scrub_stamp":"2024-03-24T12:52:20.949077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:35.165828+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":924,"reported_epoch":485,"state":"active+clean","last_fresh":"2024-03-24T12:52:29.472901+0000","last_change":"2024-03-24T12:52:26.346000+0000","last_active":"2024-03-24T12:52:29.472901+0000","last_peered":"2024-03-24T12:52:29.472901+0000","last_clean":"2024-03-24T12:52:29.472901+0000","last_became_active":"2024-03-24T12:50:27.791835+0000","last_became_peered":"2024-03-24T12:50:27.791835+0000","last_unstale":"2024-03-24T12:52:29.472901+0000","last_undegraded":"2024-03-24T12:52:29.472901+0000","last_fullsized":"2024-03-24T12:52:29.472901+0000","mapping_epoch":439,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":440,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T12:52:26.345892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T12:26:08.325526+0000","last_clean_scrub_stamp":"2024-03-24T12:52:26.345892+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T12:53:42.102058+0000","scrub_duration":64,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1579,"num_read_kb":1574,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1641120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1409,"ondisk_log_size":1409,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4694,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1533,"num_read_kb":1290,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":493,"num_bytes_recovered":10198,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},"log_size":1069,"ondisk_log_size":1069,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":112,"ondisk_log_size":112,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038084,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":808332,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":92936820,"statfs":{"total":95995035648,"available":95167303680,"internally_reserved":0,"allocated":1118208,"data_stored":714161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":826603978},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":51,"seq":219043332426,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":645552,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":644798,"kb_avail":93099600,"statfs":{"total":95995035648,"available":95333990400,"internally_reserved":0,"allocated":761856,"data_stored":385783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":660273607},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575204,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":948260,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":947454,"kb_avail":92796892,"statfs":{"total":95995035648,"available":95024017408,"internally_reserved":0,"allocated":815104,"data_stored":386761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":970193349},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":433,"seq":1859720839203,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94664,"kb_used_data":768,"kb_used_omap":4,"kb_used_meta":93883,"kb_avail":93650488,"statfs":{"total":95995035648,"available":95898099712,"internally_reserved":0,"allocated":786432,"data_stored":385715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4444,"internal_metadata":96136868},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411678,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":952632,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":951614,"kb_avail":92792520,"statfs":{"total":95995035648,"available":95019540480,"internally_reserved":0,"allocated":1028096,"data_stored":647931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":974453189},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":27,"seq":115964117336,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":732252,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":93012900,"statfs":{"total":95995035648,"available":95245209600,"internally_reserved":0,"allocated":1396736,"data_stored":977055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":748419527},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733921108,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744136,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":743358,"kb_avail":93001016,"statfs":{"total":95995035648,"available":95233040384,"internally_reserved":0,"allocated":786432,"data_stored":385513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":761199045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":437,"seq":1876900708382,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":55224,"kb_used_data":1320,"kb_used_omap":6,"kb_used_meta":53881,"kb_avail":93689928,"statfs":{"total":95995035648,"available":95938486272,"internally_reserved":0,"allocated":1351680,"data_stored":975844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6878,"internal_metadata":55174434},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":921,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2432,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T12:52:40.829 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-24T12:52:40.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 shell --fsid bc017092-e9d8-11ee-95cd-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-24T12:52:41.080 INFO:teuthology.orchestra.run.smithi029.stderr:Inferring config /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T12:52:41.340 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:41 smithi029 ceph-mon[41500]: from='client.25718 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:52:41.341 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:41 smithi029 ceph-mon[29919]: from='client.25718 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:52:41.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:41 smithi190 ceph-mon[33748]: from='client.25718 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T12:52:42.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:42 smithi029 ceph-mon[29919]: 6.13 scrub starts 2024-03-24T12:52:42.248 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:42 smithi029 ceph-mon[29919]: 6.13 scrub ok 2024-03-24T12:52:42.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:42 smithi029 ceph-mon[29919]: 6.15 scrub starts 2024-03-24T12:52:42.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:42 smithi029 ceph-mon[29919]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-24T12:52:42.249 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:42 smithi029 ceph-mon[29919]: 6.15 scrub ok 2024-03-24T12:52:42.249 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:42 smithi029 ceph-mon[41500]: 6.13 scrub starts 2024-03-24T12:52:42.249 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:42 smithi029 ceph-mon[41500]: 6.13 scrub ok 2024-03-24T12:52:42.249 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:42 smithi029 ceph-mon[41500]: 6.15 scrub starts 2024-03-24T12:52:42.249 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:42 smithi029 ceph-mon[41500]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-24T12:52:42.249 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:42 smithi029 ceph-mon[41500]: 6.15 scrub ok 2024-03-24T12:52:42.381 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:42 smithi190 ceph-mon[33748]: 6.13 scrub starts 2024-03-24T12:52:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:42 smithi190 ceph-mon[33748]: 6.13 scrub ok 2024-03-24T12:52:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:42 smithi190 ceph-mon[33748]: 6.15 scrub starts 2024-03-24T12:52:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:42 smithi190 ceph-mon[33748]: pgmap v1182: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 889 B/s rd, 0 op/s 2024-03-24T12:52:42.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:42 smithi190 ceph-mon[33748]: 6.15 scrub ok 2024-03-24T12:52:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:43 smithi190 ceph-mon[33748]: 5.5 scrub starts 2024-03-24T12:52:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:43 smithi190 ceph-mon[33748]: 5.5 scrub ok 2024-03-24T12:52:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:43 smithi190 ceph-mon[33748]: 3.0 scrub starts 2024-03-24T12:52:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:43 smithi190 ceph-mon[33748]: 3.0 scrub ok 2024-03-24T12:52:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:43 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3414804578' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-24T12:52:43.382 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:43 smithi190 ceph-mon[33748]: from='client.? 172.21.15.29:0/3414804578' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-24T12:52:43.398 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[41500]: 5.5 scrub starts 2024-03-24T12:52:43.398 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[41500]: 5.5 scrub ok 2024-03-24T12:52:43.398 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[41500]: 3.0 scrub starts 2024-03-24T12:52:43.398 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[41500]: 3.0 scrub ok 2024-03-24T12:52:43.399 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3414804578' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-24T12:52:43.399 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[41500]: from='client.? 172.21.15.29:0/3414804578' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-24T12:52:43.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[29919]: 5.5 scrub starts 2024-03-24T12:52:43.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[29919]: 5.5 scrub ok 2024-03-24T12:52:43.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[29919]: 3.0 scrub starts 2024-03-24T12:52:43.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[29919]: 3.0 scrub ok 2024-03-24T12:52:43.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3414804578' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-24T12:52:43.399 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-mon[29919]: from='client.? 172.21.15.29:0/3414804578' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-24T12:52:43.399 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:43] ENGINE Bus STOPPING 2024-03-24T12:52:43.400 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:43] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:52:43.400 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:43] ENGINE Bus STOPPED 2024-03-24T12:52:43.400 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:43] ENGINE Bus STARTING 2024-03-24T12:52:43.400 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:43] ENGINE Serving on http://:::9283 2024-03-24T12:52:43.400 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:43 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:43] ENGINE Bus STARTED 2024-03-24T12:52:43.728 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T12:52:43.738 INFO:tasks.cephadm:Teardown begin 2024-03-24T12:52:43.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:52:43.788 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:52:43.824 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T12:52:43.824 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T12:52:43.867 DEBUG:teuthology.orchestra.run.smithi190:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T12:52:43.892 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T12:52:43.892 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T12:52:43.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.a 2024-03-24T12:52:44.060 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:43 smithi029 systemd[1]: Stopping Ceph mon.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:52:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:44 smithi190 ceph-mon[33748]: 3.f scrub starts 2024-03-24T12:52:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:44 smithi190 ceph-mon[33748]: 3.f scrub ok 2024-03-24T12:52:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:44 smithi190 ceph-mon[33748]: pgmap v1183: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:52:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:52:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:52:44.132 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:44 smithi190 ceph-mon[33748]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:52:44.316 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[41500]: 3.f scrub starts 2024-03-24T12:52:44.316 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[41500]: 3.f scrub ok 2024-03-24T12:52:44.316 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[41500]: pgmap v1183: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:44.316 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:52:44.316 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:52:44.316 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:52:44.316 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[41500]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:44] ENGINE Bus STOPPING 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:44] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:44] ENGINE Bus STOPPED 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:44] ENGINE Bus STARTING 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[29919]: 3.f scrub starts 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[29919]: 3.f scrub ok 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[29919]: pgmap v1183: 142 pgs: 142 active+clean; 582 KiB data, 4.8 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T12:52:44.317 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-mon[29919]: from='mgr.14580 172.21.15.29:0/1113471362' entity='mgr.y' 2024-03-24T12:52:44.318 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a[29896]: 2024-03-24T12:52:44.249+0000 7f0ef0b43700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:52:44.318 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a[29896]: 2024-03-24T12:52:44.249+0000 7f0ef0b43700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T12:52:44.318 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 podman[227951]: 2024-03-24 12:52:44.321406162 +0000 UTC m=+0.218375804 container died b6e668dccf1e914c2f306eb4b072c2251b20a99e15ab23df7c74e9eed397f327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T12:52:44.639 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:44] ENGINE Serving on http://:::9283 2024-03-24T12:52:44.639 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:44 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:44] ENGINE Bus STARTED 2024-03-24T12:52:44.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 podman[227951]: 2024-03-24 12:52:44.77439997 +0000 UTC m=+0.671369613 container cleanup b6e668dccf1e914c2f306eb4b072c2251b20a99e15ab23df7c74e9eed397f327 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T12:52:44.916 INFO:journalctl@ceph.mon.a.smithi029.stdout:Mar 24 12:52:44 smithi029 bash[227951]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-a 2024-03-24T12:52:45.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T12:52:45.257 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:52:45.258 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T12:52:45.258 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-24T12:52:45.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.c 2024-03-24T12:52:45.416 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:45 smithi029 systemd[1]: Stopping Ceph mon.c for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:52:45.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:45 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:45] ENGINE Bus STOPPING 2024-03-24T12:52:45.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:45 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:45] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-24T12:52:45.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:45 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:45] ENGINE Bus STOPPED 2024-03-24T12:52:45.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:45 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:45] ENGINE Bus STARTING 2024-03-24T12:52:45.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:45 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:45] ENGINE Serving on http://:::9283 2024-03-24T12:52:45.917 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:45 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: [24/Mar/2024:12:52:45] ENGINE Bus STARTED 2024-03-24T12:52:45.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:45 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-c[41477]: 2024-03-24T12:52:45.618+0000 7faa3365f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:52:45.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:45 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-c[41477]: 2024-03-24T12:52:45.618+0000 7faa3365f700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T12:52:45.917 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:45 smithi029 podman[228095]: 2024-03-24 12:52:45.854853198 +0000 UTC m=+0.374105504 container died e0d35b8f460ba5d45f01a1daa188e8226a0f02594d1f400699e1d2a720caa30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:52:46.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:46 smithi029 podman[228095]: 2024-03-24 12:52:46.268562284 +0000 UTC m=+0.787814590 container cleanup e0d35b8f460ba5d45f01a1daa188e8226a0f02594d1f400699e1d2a720caa30d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:52:46.500 INFO:journalctl@ceph.mon.c.smithi029.stdout:Mar 24 12:52:46 smithi029 bash[228095]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-c 2024-03-24T12:52:46.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T12:52:46.685 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:52:46.685 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-24T12:52:46.685 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-24T12:52:46.685 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.b 2024-03-24T12:52:47.015 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:46 smithi190 systemd[1]: Stopping Ceph mon.b for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:52:47.270 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-b[33725]: 2024-03-24T12:52:47.019+0000 7fcafcb21700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:52:47.271 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-b[33725]: 2024-03-24T12:52:47.019+0000 7fcafcb21700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T12:52:47.588 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:47 smithi190 podman[159881]: 2024-03-24 12:52:47.276043044 +0000 UTC m=+0.365263590 container died 23685832c25178b2296b2b643f0c117d36e6b71d971f5615cf76af52cdd6b1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-db0330b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:52:47.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:47 smithi190 podman[159881]: 2024-03-24 12:52:47.701673276 +0000 UTC m=+0.790893833 container cleanup 23685832c25178b2296b2b643f0c117d36e6b71d971f5615cf76af52cdd6b1e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:52:47.882 INFO:journalctl@ceph.mon.b.smithi190.stdout:Mar 24 12:52:47 smithi190 bash[159881]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mon-b 2024-03-24T12:52:47.916 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:47 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: ::ffff:172.21.15.190 - - [24/Mar/2024:12:52:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-24T12:52:48.183 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T12:52:48.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:52:48.228 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-24T12:52:48.228 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T12:52:48.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mgr.y 2024-03-24T12:52:48.549 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:48 smithi029 systemd[1]: Stopping Ceph mgr.y for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:52:48.891 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:48 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:52:48.553+0000 7f4025fed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:52:48.891 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:48 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y[30177]: 2024-03-24T12:52:48.553+0000 7f4025fed700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-24T12:52:48.891 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:48 smithi029 podman[228237]: 2024-03-24 12:52:48.585499026 +0000 UTC m=+0.150401693 container died aa0b22dc45d5467f6fee9d8fef69cd9f37752f2f6088002ec615ded241947914 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T12:52:49.166 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:48 smithi029 podman[228237]: 2024-03-24 12:52:48.972251267 +0000 UTC m=+0.537153935 container cleanup aa0b22dc45d5467f6fee9d8fef69cd9f37752f2f6088002ec615ded241947914 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T12:52:49.166 INFO:journalctl@ceph.mgr.y.smithi029.stdout:Mar 24 12:52:48 smithi029 bash[228237]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-y 2024-03-24T12:52:49.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T12:52:49.598 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:52:49.598 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T12:52:49.598 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T12:52:49.598 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mgr.x 2024-03-24T12:52:49.882 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:52:49 smithi190 systemd[1]: Stopping Ceph mgr.x for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:52:50.252 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:52:49 smithi190 podman[160012]: 2024-03-24 12:52:49.989628824 +0000 UTC m=+0.193689800 container died f6342263b530aba756661fb81e337ec2c0533b984c5cd8875757d54f70a6a3d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T12:52:50.597 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:52:50 smithi190 podman[160012]: 2024-03-24 12:52:50.330722295 +0000 UTC m=+0.534783322 container cleanup f6342263b530aba756661fb81e337ec2c0533b984c5cd8875757d54f70a6a3d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:52:50.597 INFO:journalctl@ceph.mgr.x.smithi190.stdout:Mar 24 12:52:50 smithi190 bash[160012]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-mgr-x 2024-03-24T12:52:50.819 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T12:52:50.856 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:52:50.856 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T12:52:50.856 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T12:52:50.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.0 2024-03-24T12:52:51.132 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:52:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:52:51.075Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-03-24T12:52:51.132 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:52:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:52:51.077Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-03-24T12:52:51.132 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:52:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:52:51.078Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-03-24T12:52:51.132 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:52:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:52:51.078Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-03-24T12:52:51.161 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:50 smithi029 systemd[1]: Stopping Ceph osd.0 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:52:51.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[216717]: 2024-03-24T12:52:51.165+0000 7f49849c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:52:51.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[216717]: 2024-03-24T12:52:51.165+0000 7f49849c5700 -1 osd.0 485 *** Got signal Terminated *** 2024-03-24T12:52:51.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:51 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0[216717]: 2024-03-24T12:52:51.165+0000 7f49849c5700 -1 osd.0 485 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:52:57.031 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:56 smithi029 podman[228367]: 2024-03-24 12:52:56.721239218 +0000 UTC m=+5.663260846 container died 79eb38f7cef0645ed2598e616098f61b77daff9d97771f634d5d682679d22b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:52:57.416 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:57 smithi029 podman[228367]: 2024-03-24 12:52:57.218896017 +0000 UTC m=+6.160917676 container cleanup 79eb38f7cef0645ed2598e616098f61b77daff9d97771f634d5d682679d22b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T12:52:57.417 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:57 smithi029 bash[228367]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0 2024-03-24T12:52:57.786 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:57 smithi029 podman[228390]: 2024-03-24 12:52:57.480690507 +0000 UTC m=+0.763095512 container remove 79eb38f7cef0645ed2598e616098f61b77daff9d97771f634d5d682679d22b08 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T12:52:58.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:57 smithi029 podman[228484]: 2024-03-24 12:52:57.792167108 +0000 UTC m=+0.109555337 container create 92273ac1eff8196ea8e66f67bec8a79d7d5525877cab6f14c2fe4744bc3d2f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-24T12:52:58.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:57 smithi029 podman[228484]: 2024-03-24 12:52:57.700546604 +0000 UTC m=+0.017934784 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:52:58.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:58 smithi029 podman[228484]: 2024-03-24 12:52:58.012706632 +0000 UTC m=+0.330094825 container init 92273ac1eff8196ea8e66f67bec8a79d7d5525877cab6f14c2fe4744bc3d2f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:52:58.099 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:58 smithi029 podman[228484]: 2024-03-24 12:52:58.018060152 +0000 UTC m=+0.335448332 container start 92273ac1eff8196ea8e66f67bec8a79d7d5525877cab6f14c2fe4744bc3d2f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:52:58.100 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:58 smithi029 podman[228484]: 2024-03-24 12:52:58.104543575 +0000 UTC m=+0.421931771 container attach 92273ac1eff8196ea8e66f67bec8a79d7d5525877cab6f14c2fe4744bc3d2f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS) 2024-03-24T12:52:58.666 INFO:journalctl@ceph.osd.0.smithi029.stdout:Mar 24 12:52:58 smithi029 podman[228484]: 2024-03-24 12:52:58.37132588 +0000 UTC m=+0.688714073 container died 92273ac1eff8196ea8e66f67bec8a79d7d5525877cab6f14c2fe4744bc3d2f13 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, RELEASE=squid-db0330b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-24T12:52:59.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T12:52:59.173 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:52:59.173 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T12:52:59.173 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T12:52:59.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.1 2024-03-24T12:52:59.416 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:52:59 smithi029 systemd[1]: Stopping Ceph osd.1 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:52:59.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:52:59 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1[52368]: 2024-03-24T12:52:59.512+0000 7f80942ec700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:52:59.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:52:59 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1[52368]: 2024-03-24T12:52:59.512+0000 7f80942ec700 -1 osd.1 485 *** Got signal Terminated *** 2024-03-24T12:52:59.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:52:59 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1[52368]: 2024-03-24T12:52:59.512+0000 7f80942ec700 -1 osd.1 485 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:53:05.468 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:53:05 smithi029 podman[228616]: 2024-03-24 12:53:05.180808892 +0000 UTC m=+5.812293520 container died f3f2f69361b132e3f31c9779448bc5cd3dcec0b18e27af6cdfde3642ede1a819 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T12:53:05.916 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:53:05 smithi029 podman[228616]: 2024-03-24 12:53:05.655896824 +0000 UTC m=+6.287381461 container cleanup f3f2f69361b132e3f31c9779448bc5cd3dcec0b18e27af6cdfde3642ede1a819 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T12:53:05.917 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:53:05 smithi029 bash[228616]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1 2024-03-24T12:53:06.270 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:53:05 smithi029 podman[228639]: 2024-03-24 12:53:05.925271935 +0000 UTC m=+0.740136426 container remove f3f2f69361b132e3f31c9779448bc5cd3dcec0b18e27af6cdfde3642ede1a819 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T12:53:06.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:53:06 smithi029 podman[228736]: 2024-03-24 12:53:06.179968377 +0000 UTC m=+0.018733890 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:53:06.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:53:06 smithi029 podman[228736]: 2024-03-24 12:53:06.286883776 +0000 UTC m=+0.125649307 container create e3214efc573c36b3bdc3be5fee11928aefb74bee8695cf1ced864bcba36d7325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T12:53:06.524 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:53:06 smithi029 podman[228736]: 2024-03-24 12:53:06.52441607 +0000 UTC m=+0.363181569 container init e3214efc573c36b3bdc3be5fee11928aefb74bee8695cf1ced864bcba36d7325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T12:53:06.817 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:53:06 smithi029 podman[228736]: 2024-03-24 12:53:06.529787179 +0000 UTC m=+0.368552674 container start e3214efc573c36b3bdc3be5fee11928aefb74bee8695cf1ced864bcba36d7325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=) 2024-03-24T12:53:06.818 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:53:06 smithi029 podman[228736]: 2024-03-24 12:53:06.641933287 +0000 UTC m=+0.480698798 container attach e3214efc573c36b3bdc3be5fee11928aefb74bee8695cf1ced864bcba36d7325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1-deactivate, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T12:53:07.135 INFO:journalctl@ceph.osd.1.smithi029.stdout:Mar 24 12:53:06 smithi029 podman[228736]: 2024-03-24 12:53:06.823310113 +0000 UTC m=+0.662075603 container died e3214efc573c36b3bdc3be5fee11928aefb74bee8695cf1ced864bcba36d7325 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T12:53:07.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T12:53:07.470 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:53:07.471 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T12:53:07.471 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T12:53:07.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.2 2024-03-24T12:53:07.666 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:07 smithi029 systemd[1]: Stopping Ceph osd.2 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:53:08.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2[58799]: 2024-03-24T12:53:07.831+0000 7f2ff779a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:53:08.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2[58799]: 2024-03-24T12:53:07.831+0000 7f2ff779a700 -1 osd.2 485 *** Got signal Terminated *** 2024-03-24T12:53:08.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:07 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2[58799]: 2024-03-24T12:53:07.831+0000 7f2ff779a700 -1 osd.2 485 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:53:13.656 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:13 smithi029 podman[228867]: 2024-03-24 12:53:13.337392502 +0000 UTC m=+5.637680484 container died a5537e729da8a3c5f0e3333a5420591c297e9e9d6e6478c4868895c201777737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-db0330b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T12:53:13.915 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:13 smithi029 podman[228867]: 2024-03-24 12:53:13.90051604 +0000 UTC m=+6.200804023 container cleanup a5537e729da8a3c5f0e3333a5420591c297e9e9d6e6478c4868895c201777737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T12:53:13.916 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:13 smithi029 bash[228867]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2 2024-03-24T12:53:14.416 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:14 smithi029 podman[228892]: 2024-03-24 12:53:14.169929353 +0000 UTC m=+0.826650246 container remove a5537e729da8a3c5f0e3333a5420591c297e9e9d6e6478c4868895c201777737 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:53:14.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:14 smithi029 podman[228988]: 2024-03-24 12:53:14.436124918 +0000 UTC m=+0.017876843 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:53:14.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:14 smithi029 podman[228988]: 2024-03-24 12:53:14.546294357 +0000 UTC m=+0.128046283 container create 18d01ca4a502149b37d4a5906177fd4d1ac4ac0aa3f3e2de6e107482ac76c7de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T12:53:14.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:14 smithi029 podman[228988]: 2024-03-24 12:53:14.751703591 +0000 UTC m=+0.333455525 container init 18d01ca4a502149b37d4a5906177fd4d1ac4ac0aa3f3e2de6e107482ac76c7de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-24T12:53:14.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:14 smithi029 podman[228988]: 2024-03-24 12:53:14.756943953 +0000 UTC m=+0.338695891 container start 18d01ca4a502149b37d4a5906177fd4d1ac4ac0aa3f3e2de6e107482ac76c7de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:53:14.897 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:14 smithi029 podman[228988]: 2024-03-24 12:53:14.901919442 +0000 UTC m=+0.483671384 container attach 18d01ca4a502149b37d4a5906177fd4d1ac4ac0aa3f3e2de6e107482ac76c7de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-24T12:53:15.166 INFO:journalctl@ceph.osd.2.smithi029.stdout:Mar 24 12:53:15 smithi029 podman[228988]: 2024-03-24 12:53:15.049761817 +0000 UTC m=+0.631513742 container died 18d01ca4a502149b37d4a5906177fd4d1ac4ac0aa3f3e2de6e107482ac76c7de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-24T12:53:15.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T12:53:15.733 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:53:15.733 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T12:53:15.733 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T12:53:15.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.3 2024-03-24T12:53:16.078 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:15 smithi029 systemd[1]: Stopping Ceph osd.3 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:53:16.416 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:16 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3[65432]: 2024-03-24T12:53:16.082+0000 7fc900c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:53:16.416 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:16 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3[65432]: 2024-03-24T12:53:16.082+0000 7fc900c00700 -1 osd.3 485 *** Got signal Terminated *** 2024-03-24T12:53:16.416 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:16 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3[65432]: 2024-03-24T12:53:16.082+0000 7fc900c00700 -1 osd.3 485 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:53:17.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:17.204+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:17.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:17.349+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:18.416 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:18 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3[65432]: 2024-03-24T12:53:18.125+0000 7fc8fd1f4700 -1 osd.3 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.187724+0000 front 2024-03-24T12:52:55.187841+0000 (oldest deadline 2024-03-24T12:53:17.487803+0000) 2024-03-24T12:53:18.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:18.347+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:18.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:18.222+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:19.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:18.729+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:19.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:18 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:18.783+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:19.416 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:19 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3[65432]: 2024-03-24T12:53:19.092+0000 7fc8fd1f4700 -1 osd.3 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.187724+0000 front 2024-03-24T12:52:55.187841+0000 (oldest deadline 2024-03-24T12:53:17.487803+0000) 2024-03-24T12:53:19.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:19 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:19.330+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:19.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:19 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:19.205+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:20.131 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:19 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:19.719+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:20.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:19 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:19.783+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:20.416 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:20 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3[65432]: 2024-03-24T12:53:20.046+0000 7fc8fd1f4700 -1 osd.3 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.187724+0000 front 2024-03-24T12:52:55.187841+0000 (oldest deadline 2024-03-24T12:53:17.487803+0000) 2024-03-24T12:53:20.631 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:20 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:20.341+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:20.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:20 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:20.217+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:21.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:20 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:20.747+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:21.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:20 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:20.753+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:21.417 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:21 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3[65432]: 2024-03-24T12:53:21.020+0000 7fc8fd1f4700 -1 osd.3 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.187724+0000 front 2024-03-24T12:52:55.187841+0000 (oldest deadline 2024-03-24T12:53:17.487803+0000) 2024-03-24T12:53:21.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:21.334+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:21.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:21.198+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:21.916 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:21 smithi029 podman[229129]: 2024-03-24 12:53:21.64938061 +0000 UTC m=+5.683683610 container died 72efdab1954e1a0dbe64cf19ea7d15d4e8d8e259d1d546070c57412934f010f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, org.label-schema.license=GPLv2) 2024-03-24T12:53:22.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:21.766+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:22.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:21 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:21.704+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:22.406 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:22 smithi029 podman[229129]: 2024-03-24 12:53:22.134337179 +0000 UTC m=+6.168640178 container cleanup 72efdab1954e1a0dbe64cf19ea7d15d4e8d8e259d1d546070c57412934f010f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T12:53:22.406 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:22 smithi029 bash[229129]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3 2024-03-24T12:53:22.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:22 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:22.371+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:22.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:22 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:22.226+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:22.666 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:22 smithi029 podman[229153]: 2024-03-24 12:53:22.411799362 +0000 UTC m=+0.749650737 container remove 72efdab1954e1a0dbe64cf19ea7d15d4e8d8e259d1d546070c57412934f010f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3, RELEASE=squid-db0330b, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-24T12:53:23.071 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:22 smithi029 podman[229252]: 2024-03-24 12:53:22.649893171 +0000 UTC m=+0.018298019 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:53:23.072 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:22 smithi029 podman[229252]: 2024-03-24 12:53:22.763351332 +0000 UTC m=+0.131756169 container create ebff95e33ce5ef71fd00c394f9c502f951f20ecbfc2fceeb8fc01c87cfee406c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T12:53:23.072 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:22 smithi029 podman[229252]: 2024-03-24 12:53:22.984647931 +0000 UTC m=+0.353052781 container init ebff95e33ce5ef71fd00c394f9c502f951f20ecbfc2fceeb8fc01c87cfee406c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T12:53:23.072 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:22 smithi029 podman[229252]: 2024-03-24 12:53:22.990534169 +0000 UTC m=+0.358939009 container start ebff95e33ce5ef71fd00c394f9c502f951f20ecbfc2fceeb8fc01c87cfee406c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:53:23.072 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:23 smithi029 podman[229252]: 2024-03-24 12:53:23.077261372 +0000 UTC m=+0.445666223 container attach ebff95e33ce5ef71fd00c394f9c502f951f20ecbfc2fceeb8fc01c87cfee406c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-03-24T12:53:23.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:22 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:22.706+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:23.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:22 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:22.809+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:23.416 INFO:journalctl@ceph.osd.3.smithi029.stdout:Mar 24 12:53:23 smithi029 podman[229252]: 2024-03-24 12:53:23.286329836 +0000 UTC m=+0.654734669 container died ebff95e33ce5ef71fd00c394f9c502f951f20ecbfc2fceeb8fc01c87cfee406c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:53:23.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:23 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:23.345+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:23.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:23 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:23.270+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:23.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T12:53:23.897 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:53:23.897 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T12:53:23.897 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T12:53:23.897 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.4 2024-03-24T12:53:24.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:23 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:23.765+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:24.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:23 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:23.744+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:24.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:23 smithi190 systemd[1]: Stopping Ceph osd.4 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:53:24.631 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:24 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:24.343+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:24.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:24 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:24.284+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:24.632 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:24 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:24.234+0000 7f9f54c56700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:53:24.632 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:24 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:24.234+0000 7f9f54c56700 -1 osd.4 485 *** Got signal Terminated *** 2024-03-24T12:53:24.632 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:24 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:24.234+0000 7f9f54c56700 -1 osd.4 485 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:53:25.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:24 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:24.776+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:25.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:24 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:24.765+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:25.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:25 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:25.318+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:25.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:25 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:25.261+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:26.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:25 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:25.738+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:26.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:25 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:25.742+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:26.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:26 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:26.277+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:26.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:26 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:26.238+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:27.131 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:26 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:26.709+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:27.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:26 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:26.741+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:27.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:27.196+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:27.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:27.228+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:27.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:27.228+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:28.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:27.770+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:28.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:27.770+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:04.179150+0000 front 2024-03-24T12:53:04.179178+0000 (oldest deadline 2024-03-24T12:53:27.079392+0000) 2024-03-24T12:53:28.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:27.726+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:28.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:27 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:27.726+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:28.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:28.231+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:28.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:28.231+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:28.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:28.188+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:28.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:28.188+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:02.790776+0000 front 2024-03-24T12:53:02.790832+0000 (oldest deadline 2024-03-24T12:53:27.490705+0000) 2024-03-24T12:53:29.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:28.805+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:52.378500+0000 front 2024-03-24T12:52:52.378542+0000 (oldest deadline 2024-03-24T12:53:18.278154+0000) 2024-03-24T12:53:29.132 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4[39277]: 2024-03-24T12:53:28.805+0000 7f9f5124a700 -1 osd.4 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:04.179150+0000 front 2024-03-24T12:53:04.179178+0000 (oldest deadline 2024-03-24T12:53:27.079392+0000) 2024-03-24T12:53:29.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:28.688+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:29.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:28 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:28.688+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:29.411 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:29 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:29.263+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:29.411 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:29 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:29.263+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:29.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:29 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:29.191+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:29.412 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:29 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:29.192+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:02.790776+0000 front 2024-03-24T12:53:02.790832+0000 (oldest deadline 2024-03-24T12:53:27.490705+0000) 2024-03-24T12:53:30.129 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:29 smithi190 podman[160161]: 2024-03-24 12:53:29.881773554 +0000 UTC m=+5.788494654 container died 97ca2d9f5555eb717c61813e4e7c03c4059cb722af0a2972dc49bd164693b954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T12:53:30.130 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:29 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:29.702+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:30.130 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:29 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:29.702+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:30.382 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:30 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:30.222+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:30.382 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:30 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:30.222+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:30.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:30 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:30.225+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:30.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:30 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:30.225+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:02.790776+0000 front 2024-03-24T12:53:02.790832+0000 (oldest deadline 2024-03-24T12:53:27.490705+0000) 2024-03-24T12:53:30.383 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:30 smithi190 podman[160161]: 2024-03-24 12:53:30.299832244 +0000 UTC m=+6.206553339 container cleanup 97ca2d9f5555eb717c61813e4e7c03c4059cb722af0a2972dc49bd164693b954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T12:53:30.383 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:30 smithi190 bash[160161]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4 2024-03-24T12:53:30.686 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:30 smithi190 podman[160194]: 2024-03-24 12:53:30.559729194 +0000 UTC m=+0.680378333 container remove 97ca2d9f5555eb717c61813e4e7c03c4059cb722af0a2972dc49bd164693b954 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:53:30.954 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:30 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:30.691+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:30.954 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:30 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:30.691+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:31.254 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:30 smithi190 podman[160287]: 2024-03-24 12:53:30.862412976 +0000 UTC m=+0.017827675 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:53:31.254 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:30 smithi190 podman[160287]: 2024-03-24 12:53:30.960362326 +0000 UTC m=+0.115777020 container create 90eded9868206f0a60eaf6fa5a9e34caf47f7ce836c3566309ea66787d2d7e7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T12:53:31.254 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:31 smithi190 podman[160287]: 2024-03-24 12:53:31.179938571 +0000 UTC m=+0.335353272 container init 90eded9868206f0a60eaf6fa5a9e34caf47f7ce836c3566309ea66787d2d7e7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-24T12:53:31.255 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:31 smithi190 podman[160287]: 2024-03-24 12:53:31.186004634 +0000 UTC m=+0.341419345 container start 90eded9868206f0a60eaf6fa5a9e34caf47f7ce836c3566309ea66787d2d7e7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-24T12:53:31.522 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:31 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:31.258+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:31.522 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:31 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:31.258+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:31.522 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:31 smithi190 podman[160287]: 2024-03-24 12:53:31.287804479 +0000 UTC m=+0.443219178 container attach 90eded9868206f0a60eaf6fa5a9e34caf47f7ce836c3566309ea66787d2d7e7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:53:31.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:31 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:31.259+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:31.523 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:31 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:31.259+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:02.790776+0000 front 2024-03-24T12:53:02.790832+0000 (oldest deadline 2024-03-24T12:53:27.490705+0000) 2024-03-24T12:53:31.844 INFO:journalctl@ceph.osd.4.smithi190.stdout:Mar 24 12:53:31 smithi190 podman[160287]: 2024-03-24 12:53:31.527791603 +0000 UTC m=+0.683206300 container died 90eded9868206f0a60eaf6fa5a9e34caf47f7ce836c3566309ea66787d2d7e7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T12:53:31.845 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:31 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:31.732+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:31.845 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:31 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:31.732+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:32.126 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T12:53:32.170 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:53:32.170 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T12:53:32.170 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T12:53:32.170 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.5 2024-03-24T12:53:32.381 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:32 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:32.295+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:32.382 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:32 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:32.295+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:32.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:32 smithi190 systemd[1]: Stopping Ceph osd.5 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:53:32.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:32 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:32.305+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:32.382 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:32 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:32.306+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:02.790776+0000 front 2024-03-24T12:53:02.790832+0000 (oldest deadline 2024-03-24T12:53:27.490705+0000) 2024-03-24T12:53:32.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:32 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:32.521+0000 7f1918dc4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:53:32.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:32 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:32.521+0000 7f1918dc4700 -1 osd.5 485 *** Got signal Terminated *** 2024-03-24T12:53:32.882 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:32 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:32.521+0000 7f1918dc4700 -1 osd.5 485 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:53:32.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:32 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:32.713+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:32.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:32 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:32.713+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:33.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:33 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:33.281+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:33.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:33 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:33.281+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:33.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:33 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:33.293+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:33.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:33 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:33.293+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:02.790776+0000 front 2024-03-24T12:53:02.790832+0000 (oldest deadline 2024-03-24T12:53:27.490705+0000) 2024-03-24T12:53:34.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:33 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:33.717+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:34.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:33 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:33.717+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:34.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:34 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:34.293+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:34.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:34 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:34.293+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:34.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:34 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:34.331+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:34.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:34 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:34.331+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:02.790776+0000 front 2024-03-24T12:53:02.790832+0000 (oldest deadline 2024-03-24T12:53:27.490705+0000) 2024-03-24T12:53:34.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:34 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:34.331+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:07.491283+0000 front 2024-03-24T12:53:07.491335+0000 (oldest deadline 2024-03-24T12:53:33.391152+0000) 2024-03-24T12:53:35.131 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:34 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:34.745+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:35.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:34 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:34.745+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:35.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:35 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:35.315+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:35.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:35 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:35.315+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:35.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:35 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:35.315+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:35.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:35 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:35.335+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:35.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:35 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:35.335+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:02.790776+0000 front 2024-03-24T12:53:02.790832+0000 (oldest deadline 2024-03-24T12:53:27.490705+0000) 2024-03-24T12:53:35.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:35 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:35.335+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:07.491283+0000 front 2024-03-24T12:53:07.491335+0000 (oldest deadline 2024-03-24T12:53:33.391152+0000) 2024-03-24T12:53:36.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:35 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:35.755+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:36.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:35 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:35.755+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:36.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:35 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:35.755+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:36.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:36 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:36.306+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:36.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:36 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:36.306+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:36.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:36 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:36.306+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:36.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:36 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:36.286+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:36.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:36 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:36.286+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:02.790776+0000 front 2024-03-24T12:53:02.790832+0000 (oldest deadline 2024-03-24T12:53:27.490705+0000) 2024-03-24T12:53:36.633 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:36 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:36.286+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:07.491283+0000 front 2024-03-24T12:53:07.491335+0000 (oldest deadline 2024-03-24T12:53:33.391152+0000) 2024-03-24T12:53:37.131 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:36 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:36.723+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:37.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:36 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:36.723+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:37.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:36 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:36.723+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:37.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:36 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:36.723+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:37.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:37.348+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:37.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:37.348+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:37.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:37.348+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:37.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:37.246+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:54.688997+0000 front 2024-03-24T12:52:54.689057+0000 (oldest deadline 2024-03-24T12:53:16.989126+0000) 2024-03-24T12:53:37.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:37.246+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:02.790776+0000 front 2024-03-24T12:53:02.790832+0000 (oldest deadline 2024-03-24T12:53:27.490705+0000) 2024-03-24T12:53:37.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5[156566]: 2024-03-24T12:53:37.246+0000 7f19153b8700 -1 osd.5 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:07.491283+0000 front 2024-03-24T12:53:07.491335+0000 (oldest deadline 2024-03-24T12:53:33.391152+0000) 2024-03-24T12:53:37.990 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:37.740+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:37.991 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:37.740+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:37.991 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:37.740+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:37.991 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:37 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:37.740+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:38.265 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:37 smithi190 podman[160415]: 2024-03-24 12:53:37.996219846 +0000 UTC m=+5.582417856 container died a6cac83b1f7c6cd2622fa0de1b641595ef77cb5bbed34b5a43387e92342b4608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T12:53:38.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:38 smithi190 podman[160415]: 2024-03-24 12:53:38.454901261 +0000 UTC m=+6.041099274 container cleanup a6cac83b1f7c6cd2622fa0de1b641595ef77cb5bbed34b5a43387e92342b4608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-db0330b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:53:38.632 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:38 smithi190 bash[160415]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5 2024-03-24T12:53:38.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:38.319+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:38.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:38.319+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:38.633 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:38.319+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:39.089 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:38 smithi190 podman[160438]: 2024-03-24 12:53:38.789868399 +0000 UTC m=+0.786883512 container remove a6cac83b1f7c6cd2622fa0de1b641595ef77cb5bbed34b5a43387e92342b4608 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-24T12:53:39.089 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:38.718+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:39.090 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:38.718+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:39.090 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:38.718+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:39.090 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:38 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:38.718+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:39.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:39 smithi190 podman[160534]: 2024-03-24 12:53:38.99773571 +0000 UTC m=+0.017802800 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:53:39.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:39 smithi190 podman[160534]: 2024-03-24 12:53:39.098812077 +0000 UTC m=+0.118879152 container create 5f3da6a65e0dc02ee84ff76cc4bb3b1425b234a8c3d212a935557d9f3472b3c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-deactivate, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T12:53:39.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:39 smithi190 podman[160534]: 2024-03-24 12:53:39.310075037 +0000 UTC m=+0.330142134 container init 5f3da6a65e0dc02ee84ff76cc4bb3b1425b234a8c3d212a935557d9f3472b3c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-deactivate, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:53:39.347 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:39 smithi190 podman[160534]: 2024-03-24 12:53:39.31569 +0000 UTC m=+0.335757079 container start 5f3da6a65e0dc02ee84ff76cc4bb3b1425b234a8c3d212a935557d9f3472b3c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T12:53:39.598 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:39 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:39.351+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:39.598 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:39 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:39.351+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:39.598 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:39 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:39.351+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:39.599 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:39 smithi190 podman[160534]: 2024-03-24 12:53:39.392914405 +0000 UTC m=+0.412981481 container attach 5f3da6a65e0dc02ee84ff76cc4bb3b1425b234a8c3d212a935557d9f3472b3c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-03-24T12:53:39.881 INFO:journalctl@ceph.osd.5.smithi190.stdout:Mar 24 12:53:39 smithi190 podman[160534]: 2024-03-24 12:53:39.604118699 +0000 UTC m=+0.624185780 container died 5f3da6a65e0dc02ee84ff76cc4bb3b1425b234a8c3d212a935557d9f3472b3c1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-5-deactivate, RELEASE=squid-db0330b, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-24T12:53:39.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:39 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:39.676+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:39.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:39 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:39.676+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:39.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:39 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:39.676+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:39.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:39 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:39.676+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:40.248 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T12:53:40.291 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:53:40.291 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T12:53:40.291 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T12:53:40.291 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.6 2024-03-24T12:53:40.483 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:40.367+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:40.483 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:40.367+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:40.483 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:40.367+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:40.483 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:40 smithi190 systemd[1]: Stopping Ceph osd.6 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:53:40.882 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:40.635+0000 7f2bdd6b6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:53:40.882 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:40.635+0000 7f2bdd6b6700 -1 osd.6 485 *** Got signal Terminated *** 2024-03-24T12:53:40.882 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:40.635+0000 7f2bdd6b6700 -1 osd.6 485 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:53:40.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:40.631+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:40.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:40.631+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:40.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:40.631+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:40.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:40 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:40.631+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:41.631 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:41 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:41.360+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:41.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:41 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:41.360+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:41.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:41 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:41.360+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:42.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:41 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:41.675+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:42.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:41 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:41.675+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:42.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:41 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:41.675+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:42.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:41 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:41.675+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:42.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:42.376+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:42.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:42.376+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:42.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:42.376+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:42.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:42.376+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:19.741434+0000 front 2024-03-24T12:53:19.741445+0000 (oldest deadline 2024-03-24T12:53:41.942033+0000) 2024-03-24T12:53:43.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:42.671+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:43.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:42.671+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:43.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:42.671+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:43.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:42 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:42.671+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:43.659 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:43.388+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:43.660 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:43.388+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:43.660 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:43.388+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:43.660 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:43.388+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:19.741434+0000 front 2024-03-24T12:53:19.741445+0000 (oldest deadline 2024-03-24T12:53:41.942033+0000) 2024-03-24T12:53:44.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:43.664+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:44.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:43.664+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:44.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:43.664+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:44.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:43 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:43.664+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:44.631 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:44.372+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:44.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:44.372+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:44.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:44.372+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:44.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:44.372+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:19.741434+0000 front 2024-03-24T12:53:19.741445+0000 (oldest deadline 2024-03-24T12:53:41.942033+0000) 2024-03-24T12:53:45.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:44.699+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:45.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:44.699+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:45.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:44.699+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:45.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:44.699+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:20.885441+0000 front 2024-03-24T12:53:20.885476+0000 (oldest deadline 2024-03-24T12:53:43.785429+0000) 2024-03-24T12:53:45.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:44 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:44.699+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:45.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:45.332+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:53.037328+0000 front 2024-03-24T12:52:53.037396+0000 (oldest deadline 2024-03-24T12:53:17.037612+0000) 2024-03-24T12:53:45.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:45.332+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:01.039293+0000 front 2024-03-24T12:53:01.039313+0000 (oldest deadline 2024-03-24T12:53:26.939180+0000) 2024-03-24T12:53:45.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:45.332+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:12.740703+0000 front 2024-03-24T12:53:12.740731+0000 (oldest deadline 2024-03-24T12:53:35.040568+0000) 2024-03-24T12:53:45.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6[51120]: 2024-03-24T12:53:45.332+0000 7f2bd5ca2700 -1 osd.6 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:19.741434+0000 front 2024-03-24T12:53:19.741445+0000 (oldest deadline 2024-03-24T12:53:41.942033+0000) 2024-03-24T12:53:46.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:45.709+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:46.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:45.709+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:46.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:45.710+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:46.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:45.710+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:20.885441+0000 front 2024-03-24T12:53:20.885476+0000 (oldest deadline 2024-03-24T12:53:43.785429+0000) 2024-03-24T12:53:46.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:45 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:45.710+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:46.546 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:46 smithi190 podman[160664]: 2024-03-24 12:53:46.23981509 +0000 UTC m=+5.706957981 container died 57e2ec2e6ed15300b65f8125d0a184c54e7c03e50a20d4707c36ea344198bae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b) 2024-03-24T12:53:46.881 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:46 smithi190 podman[160664]: 2024-03-24 12:53:46.699724843 +0000 UTC m=+6.166867775 container cleanup 57e2ec2e6ed15300b65f8125d0a184c54e7c03e50a20d4707c36ea344198bae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T12:53:46.882 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:46 smithi190 bash[160664]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6 2024-03-24T12:53:46.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:46.690+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:46.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:46.690+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:46.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:46.690+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:46.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:46.690+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:20.885441+0000 front 2024-03-24T12:53:20.885476+0000 (oldest deadline 2024-03-24T12:53:43.785429+0000) 2024-03-24T12:53:46.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:46 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:46.690+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:47.374 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:47 smithi190 podman[160689]: 2024-03-24 12:53:47.017998977 +0000 UTC m=+0.765986765 container remove 57e2ec2e6ed15300b65f8125d0a184c54e7c03e50a20d4707c36ea344198bae3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T12:53:47.631 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:47 smithi190 podman[160786]: 2024-03-24 12:53:47.282557371 +0000 UTC m=+0.017997038 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:53:47.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:47 smithi190 podman[160786]: 2024-03-24 12:53:47.385270023 +0000 UTC m=+0.120709685 container create 4b56e2a078decdb6270398be8642d080df6d7c94172aaacf10baebd203f00373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T12:53:47.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:47 smithi190 podman[160786]: 2024-03-24 12:53:47.60625029 +0000 UTC m=+0.341689957 container init 4b56e2a078decdb6270398be8642d080df6d7c94172aaacf10baebd203f00373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-24T12:53:47.632 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:47 smithi190 podman[160786]: 2024-03-24 12:53:47.612036282 +0000 UTC m=+0.347475944 container start 4b56e2a078decdb6270398be8642d080df6d7c94172aaacf10baebd203f00373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-db0330b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:53:48.132 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:47 smithi190 podman[160786]: 2024-03-24 12:53:47.68801262 +0000 UTC m=+0.423452310 container attach 4b56e2a078decdb6270398be8642d080df6d7c94172aaacf10baebd203f00373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-db0330b) 2024-03-24T12:53:48.132 INFO:journalctl@ceph.osd.6.smithi190.stdout:Mar 24 12:53:47 smithi190 podman[160786]: 2024-03-24 12:53:47.903647585 +0000 UTC m=+0.639087246 container died 4b56e2a078decdb6270398be8642d080df6d7c94172aaacf10baebd203f00373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-db0330b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T12:53:48.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:47.702+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:48.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:47.703+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:48.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:47.703+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:48.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:47.703+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:20.885441+0000 front 2024-03-24T12:53:20.885476+0000 (oldest deadline 2024-03-24T12:53:43.785429+0000) 2024-03-24T12:53:48.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:47 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:47.703+0000 7fa362020700 -1 osd.7 485 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24427.0:1608 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e485) 2024-03-24T12:53:48.645 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T12:53:48.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:53:48.688 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T12:53:48.688 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T12:53:48.688 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.7 2024-03-24T12:53:48.881 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:48.674+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:48.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:48.674+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:48.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:48.674+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:48.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:48 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:48.674+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:20.885441+0000 front 2024-03-24T12:53:20.885476+0000 (oldest deadline 2024-03-24T12:53:43.785429+0000) 2024-03-24T12:53:48.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:48 smithi190 systemd[1]: Stopping Ceph osd.7 for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:53:49.381 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:49.021+0000 7fa365a2c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:53:49.382 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:49.021+0000 7fa365a2c700 -1 osd.7 485 *** Got signal Terminated *** 2024-03-24T12:53:49.382 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:49.021+0000 7fa365a2c700 -1 osd.7 485 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T12:53:50.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:49.683+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:50.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:49.683+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:50.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:49.683+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:50.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:49 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:49.683+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:20.885441+0000 front 2024-03-24T12:53:20.885476+0000 (oldest deadline 2024-03-24T12:53:43.785429+0000) 2024-03-24T12:53:51.070 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:50 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:50.656+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:51.070 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:50 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:50.656+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:51.070 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:50 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:50.656+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:51.070 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:50 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:50.656+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:20.885441+0000 front 2024-03-24T12:53:20.885476+0000 (oldest deadline 2024-03-24T12:53:43.785429+0000) 2024-03-24T12:53:51.382 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:53:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:53:51.075Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-03-24T12:53:51.382 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:53:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:53:51.077Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-03-24T12:53:51.382 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:53:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:53:51.078Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-03-24T12:53:51.382 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:53:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:53:51.078Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.29:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.29:8765: connect: connection refused" 2024-03-24T12:53:52.131 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:51.693+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:52.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:51.693+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:52.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:51.693+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:52.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:51.693+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:20.885441+0000 front 2024-03-24T12:53:20.885476+0000 (oldest deadline 2024-03-24T12:53:43.785429+0000) 2024-03-24T12:53:52.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:51 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:51.693+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.190:6802 osd.4 since back 2024-03-24T12:53:29.086353+0000 front 2024-03-24T12:53:29.086389+0000 (oldest deadline 2024-03-24T12:53:50.686660+0000) 2024-03-24T12:53:53.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:52 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:52.668+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:53.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:52 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:52.668+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:53.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:52 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:52.668+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:53.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:52 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:52.668+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:20.885441+0000 front 2024-03-24T12:53:20.885476+0000 (oldest deadline 2024-03-24T12:53:43.785429+0000) 2024-03-24T12:53:53.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:52 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:52.668+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.190:6802 osd.4 since back 2024-03-24T12:53:29.086353+0000 front 2024-03-24T12:53:29.086389+0000 (oldest deadline 2024-03-24T12:53:50.686660+0000) 2024-03-24T12:53:54.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:53 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:53.690+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6803 osd.0 since back 2024-03-24T12:52:55.781931+0000 front 2024-03-24T12:52:55.781915+0000 (oldest deadline 2024-03-24T12:53:18.582177+0000) 2024-03-24T12:53:54.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:53 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:53.690+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6807 osd.1 since back 2024-03-24T12:53:03.283212+0000 front 2024-03-24T12:53:03.283285+0000 (oldest deadline 2024-03-24T12:53:26.783168+0000) 2024-03-24T12:53:54.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:53 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:53.690+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6811 osd.2 since back 2024-03-24T12:53:11.484300+0000 front 2024-03-24T12:53:11.484412+0000 (oldest deadline 2024-03-24T12:53:34.984358+0000) 2024-03-24T12:53:54.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:53 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:53.690+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.29:6815 osd.3 since back 2024-03-24T12:53:20.885441+0000 front 2024-03-24T12:53:20.885476+0000 (oldest deadline 2024-03-24T12:53:43.785429+0000) 2024-03-24T12:53:54.132 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:53 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7[57192]: 2024-03-24T12:53:53.690+0000 7fa362020700 -1 osd.7 485 heartbeat_check: no reply from 172.21.15.190:6802 osd.4 since back 2024-03-24T12:53:29.086353+0000 front 2024-03-24T12:53:29.086389+0000 (oldest deadline 2024-03-24T12:53:50.686660+0000) 2024-03-24T12:53:54.882 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:54 smithi190 podman[160916]: 2024-03-24 12:53:54.61936596 +0000 UTC m=+5.703798074 container died bb17c5ac783eba1212355d08fa2121c304d558265fa999fe82e812fae46900d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-db0330b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T12:53:55.357 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:55 smithi190 podman[160916]: 2024-03-24 12:53:55.069502586 +0000 UTC m=+6.153934750 container cleanup bb17c5ac783eba1212355d08fa2121c304d558265fa999fe82e812fae46900d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:53:55.357 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:55 smithi190 bash[160916]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7 2024-03-24T12:53:55.631 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:55 smithi190 podman[160942]: 2024-03-24 12:53:55.362260658 +0000 UTC m=+0.742132684 container remove bb17c5ac783eba1212355d08fa2121c304d558265fa999fe82e812fae46900d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-db0330b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T12:53:55.984 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:55 smithi190 podman[161039]: 2024-03-24 12:53:55.600403675 +0000 UTC m=+0.019299958 image pull 4f325f8440e9c0f506d7739caefa90a7992ee8ebee3932602ec2cb63e901d08d quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 2024-03-24T12:53:55.985 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:55 smithi190 podman[161039]: 2024-03-24 12:53:55.696353272 +0000 UTC m=+0.115249549 container create 50667046107b2d5e8697fd439e67a216ceaff5335ad92306e79e68a91ed24142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T12:53:55.985 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:55 smithi190 podman[161039]: 2024-03-24 12:53:55.90715649 +0000 UTC m=+0.326052775 container init 50667046107b2d5e8697fd439e67a216ceaff5335ad92306e79e68a91ed24142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T12:53:55.985 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:55 smithi190 podman[161039]: 2024-03-24 12:53:55.912805223 +0000 UTC m=+0.331701501 container start 50667046107b2d5e8697fd439e67a216ceaff5335ad92306e79e68a91ed24142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-24T12:53:55.985 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:55 smithi190 podman[161039]: 2024-03-24 12:53:55.990201516 +0000 UTC m=+0.409097804 container attach 50667046107b2d5e8697fd439e67a216ceaff5335ad92306e79e68a91ed24142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-db0330b) 2024-03-24T12:53:56.381 INFO:journalctl@ceph.osd.7.smithi190.stdout:Mar 24 12:53:56 smithi190 podman[161039]: 2024-03-24 12:53:56.200266868 +0000 UTC m=+0.619163152 container died 50667046107b2d5e8697fd439e67a216ceaff5335ad92306e79e68a91ed24142 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=squid-db0330b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:53:56.833 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T12:53:56.874 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:53:56.875 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T12:53:56.875 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-24T12:53:56.875 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@rgw.foo.a 2024-03-24T12:53:57.166 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:53:56 smithi029 systemd[1]: Stopping Ceph rgw.foo.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:53:57.666 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:53:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-rgw-foo-a[70573]: 2024-03-24T12:53:57.187+0000 7f3b40f68700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T12:53:57.666 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:53:57 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-rgw-foo-a[70573]: 2024-03-24T12:53:57.187+0000 7f3b4f7d8ac0 -1 shutting down 2024-03-24T12:54:07.492 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:54:07 smithi029 bash[229383]: time="2024-03-24T12:54:07Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-bc017092-e9d8-11ee-95cd-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-24T12:54:07.492 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:54:07 smithi029 podman[229383]: 2024-03-24 12:54:07.217127953 +0000 UTC m=+10.131442176 container died e3dc4a015866345f50ac4ad0fa6c83782b04f4ada54dc023a46b6b9d996c8953 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:54:07.894 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:54:07 smithi029 podman[229383]: 2024-03-24 12:54:07.670964912 +0000 UTC m=+10.585279143 container cleanup e3dc4a015866345f50ac4ad0fa6c83782b04f4ada54dc023a46b6b9d996c8953 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-rgw-foo-a, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T12:54:07.894 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:54:07 smithi029 bash[229383]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-rgw-foo-a 2024-03-24T12:54:08.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@rgw.foo.a.service' 2024-03-24T12:54:08.145 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:54:07 smithi029 podman[229420]: 2024-03-24 12:54:07.899574244 +0000 UTC m=+0.664555011 container remove e3dc4a015866345f50ac4ad0fa6c83782b04f4ada54dc023a46b6b9d996c8953 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-rgw-foo-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-db0330b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T12:54:08.145 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:54:07 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-24T12:54:08.145 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:54:08 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-24T12:54:08.145 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:54:08 smithi029 systemd[1]: Stopped Ceph rgw.foo.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:54:08.145 INFO:journalctl@ceph.rgw.foo.a.smithi029.stdout:Mar 24 12:54:08 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@rgw.foo.a.service: Consumed 3.969s CPU time. 2024-03-24T12:54:08.155 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:54:08.156 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-24T12:54:08.156 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-24T12:54:08.156 DEBUG:teuthology.orchestra.run.smithi190:> sudo systemctl stop ceph-bc017092-e9d8-11ee-95cd-87774f69a715@prometheus.a 2024-03-24T12:54:08.502 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 systemd[1]: Stopping Ceph prometheus.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:54:08.807 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.506Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-24T12:54:08.807 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.507Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.507Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.507Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.507Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.507Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.507Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.507Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.526Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.527Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.527Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a[79113]: ts=2024-03-24T12:54:08.527Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-24T12:54:08.808 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 podman[161169]: 2024-03-24 12:54:08.537844827 +0000 UTC m=+0.145204647 container died 46683ac055d1e2a14266996917c17018fbb5bb9208453c9c1431df51b0f85b2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:54:09.131 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 podman[161169]: 2024-03-24 12:54:08.987322812 +0000 UTC m=+0.594682641 container cleanup 46683ac055d1e2a14266996917c17018fbb5bb9208453c9c1431df51b0f85b2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:54:09.131 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:08 smithi190 bash[161169]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a 2024-03-24T12:54:09.433 DEBUG:teuthology.orchestra.run.smithi190:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bc017092-e9d8-11ee-95cd-87774f69a715@prometheus.a.service' 2024-03-24T12:54:09.458 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:09 smithi190 podman[161188]: 2024-03-24 12:54:09.330183258 +0000 UTC m=+0.792325724 container remove 46683ac055d1e2a14266996917c17018fbb5bb9208453c9c1431df51b0f85b2d (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-24T12:54:09.459 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:09 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-24T12:54:09.459 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:09 smithi190 systemd[1]: Stopped Ceph prometheus.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:54:09.459 INFO:journalctl@ceph.prometheus.a.smithi190.stdout:Mar 24 12:54:09 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@prometheus.a.service: Consumed 4.442s CPU time. 2024-03-24T12:54:09.473 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T12:54:09.473 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-24T12:54:09.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bc017092-e9d8-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T12:54:09.664 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:54:12.166 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:54:12 smithi029 systemd[1]: Stopping Ceph alertmanager.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:54:12.628 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:54:12 smithi029 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a[95443]: ts=2024-03-24T12:54:12.283Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-24T12:54:12.629 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:54:12 smithi029 podman[229907]: 2024-03-24 12:54:12.334582082 +0000 UTC m=+0.193721206 container died d88a22cde729f6720b901e219a3336342391ca4779bc0e11785eeaa08598a689 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:54:12.916 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:54:12 smithi029 podman[229907]: 2024-03-24 12:54:12.740604486 +0000 UTC m=+0.599743618 container cleanup d88a22cde729f6720b901e219a3336342391ca4779bc0e11785eeaa08598a689 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:54:12.916 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:54:12 smithi029 bash[229907]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a 2024-03-24T12:54:13.261 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:54:13 smithi029 podman[229926]: 2024-03-24 12:54:13.002793602 +0000 UTC m=+0.668055044 container remove d88a22cde729f6720b901e219a3336342391ca4779bc0e11785eeaa08598a689 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-24T12:54:13.262 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:54:13 smithi029 podman[229926]: 2024-03-24 12:54:13.087242411 +0000 UTC m=+0.752503864 volume remove 09a600bc2c5d3918b47c613da5c4172a23c2fcea36f59df03f75ea1a21d11e07 2024-03-24T12:54:13.262 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:54:13 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-24T12:54:13.262 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:54:13 smithi029 systemd[1]: Stopped Ceph alertmanager.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:54:13.262 INFO:journalctl@ceph.alertmanager.a.smithi029.stdout:Mar 24 12:54:13 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@alertmanager.a.service: Consumed 1.899s CPU time. 2024-03-24T12:54:13.666 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:54:13 smithi029 systemd[1]: Stopping Ceph node-exporter.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:54:13.969 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:54:13 smithi029 podman[230046]: 2024-03-24 12:54:13.69570846 +0000 UTC m=+0.119028199 container died e5bbcb3d32819411e67d5d0f52ebb1244199e1c7dcd8f0c1fee6b342ca4f715f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:54:14.306 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:54:14 smithi029 podman[230046]: 2024-03-24 12:54:14.07524339 +0000 UTC m=+0.498563163 container cleanup e5bbcb3d32819411e67d5d0f52ebb1244199e1c7dcd8f0c1fee6b342ca4f715f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:54:14.306 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:54:14 smithi029 bash[230046]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a 2024-03-24T12:54:14.588 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:54:14 smithi029 podman[230065]: 2024-03-24 12:54:14.312059299 +0000 UTC m=+0.621186868 container remove e5bbcb3d32819411e67d5d0f52ebb1244199e1c7dcd8f0c1fee6b342ca4f715f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-24T12:54:14.588 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:54:14 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T12:54:14.588 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:54:14 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-24T12:54:14.588 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:54:14 smithi029 systemd[1]: Stopped Ceph node-exporter.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:54:14.588 INFO:journalctl@ceph.node-exporter.a.smithi029.stdout:Mar 24 12:54:14 smithi029 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@node-exporter.a.service: Consumed 4.788s CPU time. 2024-03-24T12:54:15.151 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bc017092-e9d8-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T12:54:15.344 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T12:54:17.382 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:17 smithi190 systemd[1]: Stopping Ceph iscsi.iscsi.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:54:17.881 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:17 smithi190 ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a[62231]: debug Shutdown received 2024-03-24T12:54:27.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:27 smithi190 bash[161602]: time="2024-03-24T12:54:27Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-24T12:54:27.838 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:27 smithi190 podman[161602]: 2024-03-24 12:54:27.520609691 +0000 UTC m=+10.158618960 container died d64804989f550cbafeb03a7184d7f6801a7169e011b27acaab5bcbec3bbc81c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-db0330b, GIT_CLEAN=True) 2024-03-24T12:54:28.131 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:27 smithi190 podman[161602]: 2024-03-24 12:54:27.915710513 +0000 UTC m=+10.553719787 container cleanup d64804989f550cbafeb03a7184d7f6801a7169e011b27acaab5bcbec3bbc81c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-db0330b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T12:54:28.131 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:27 smithi190 bash[161602]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a 2024-03-24T12:54:28.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:28 smithi190 podman[161629]: 2024-03-24 12:54:28.17522661 +0000 UTC m=+0.654264351 container remove d64804989f550cbafeb03a7184d7f6801a7169e011b27acaab5bcbec3bbc81c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, RELEASE=squid-db0330b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T12:54:28.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:28 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-24T12:54:28.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:28 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-24T12:54:28.476 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:28 smithi190 systemd[1]: Stopped Ceph iscsi.iscsi.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:54:28.477 INFO:journalctl@ceph.iscsi.iscsi.a.smithi190.stdout:Mar 24 12:54:28 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@iscsi.iscsi.a.service: Consumed 3.623s CPU time. 2024-03-24T12:54:29.932 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:54:29 smithi190 systemd[1]: Stopping Ceph grafana.a for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:54:30.186 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:54:29 smithi190 podman[161909]: 2024-03-24 12:54:29.937197173 +0000 UTC m=+0.109573267 container died 86dfe5a4032f20268d81ffe775b26556ee313288affa70e896aff605e96a0a86 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a, io.buildah.version=1.32.0, distribution-scope=public, description=Ceph Grafana Container, version=8.5, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, maintainer=Paul Cuzner , com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-type=git, io.openshift.expose-services=, name=ubi8, release=239.1651231664, build-date=2022-04-29T11:48:11.055897) 2024-03-24T12:54:30.552 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:54:30 smithi190 podman[161909]: 2024-03-24 12:54:30.282312763 +0000 UTC m=+0.454688855 container cleanup 86dfe5a4032f20268d81ffe775b26556ee313288affa70e896aff605e96a0a86 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a, name=ubi8, release=239.1651231664, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8.5, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, io.openshift.expose-services=, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.openshift.tags=base rhel8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, vendor=Red Hat, Inc., io.buildah.version=1.32.0) 2024-03-24T12:54:30.553 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:54:30 smithi190 bash[161909]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a 2024-03-24T12:54:30.854 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:54:30 smithi190 podman[161928]: 2024-03-24 12:54:30.558359663 +0000 UTC m=+0.623919686 container remove 86dfe5a4032f20268d81ffe775b26556ee313288affa70e896aff605e96a0a86 (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=ubi8-container, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, io.buildah.version=1.32.0, distribution-scope=public, vcs-type=git, name=ubi8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, io.openshift.tags=base rhel8, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, vendor=Red Hat, Inc.) 2024-03-24T12:54:30.854 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:54:30 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-24T12:54:30.854 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:54:30 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-24T12:54:30.854 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:54:30 smithi190 systemd[1]: Stopped Ceph grafana.a for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:54:30.854 INFO:journalctl@ceph.grafana.a.smithi190.stdout:Mar 24 12:54:30 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@grafana.a.service: Consumed 13.765s CPU time. 2024-03-24T12:54:31.110 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:54:30 smithi190 systemd[1]: Stopping Ceph node-exporter.b for bc017092-e9d8-11ee-95cd-87774f69a715... 2024-03-24T12:54:31.365 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:54:31 smithi190 podman[162046]: 2024-03-24 12:54:31.116007182 +0000 UTC m=+0.118243778 container died e5177deff2c0bb4af4de3cb5bf272d0f901250fff2d2fd84c1f13bfef08f73ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:54:31.631 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:54:31 smithi190 podman[162046]: 2024-03-24 12:54:31.460820562 +0000 UTC m=+0.463057158 container cleanup e5177deff2c0bb4af4de3cb5bf272d0f901250fff2d2fd84c1f13bfef08f73ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:54:31.631 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:54:31 smithi190 bash[162046]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b 2024-03-24T12:54:32.024 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:54:31 smithi190 podman[162065]: 2024-03-24 12:54:31.728573589 +0000 UTC m=+0.613725878 container remove e5177deff2c0bb4af4de3cb5bf272d0f901250fff2d2fd84c1f13bfef08f73ca (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-bc017092-e9d8-11ee-95cd-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-24T12:54:32.024 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:54:31 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-24T12:54:32.024 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:54:31 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-24T12:54:32.024 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:54:31 smithi190 systemd[1]: Stopped Ceph node-exporter.b for bc017092-e9d8-11ee-95cd-87774f69a715. 2024-03-24T12:54:32.024 INFO:journalctl@ceph.node-exporter.b.smithi190.stdout:Mar 24 12:54:31 smithi190 systemd[1]: ceph-bc017092-e9d8-11ee-95cd-87774f69a715@node-exporter.b.service: Consumed 4.362s CPU time. 2024-03-24T12:54:32.569 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:54:32.603 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T12:54:32.632 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T12:54:32.634 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999/remote/smithi029/crash 2024-03-24T12:54:32.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/crash -- . 2024-03-24T12:54:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:tar: /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T12:54:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:tar: Error is not recoverable: exiting now 2024-03-24T12:54:32.701 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999/remote/smithi190/crash 2024-03-24T12:54:32.702 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/crash -- . 2024-03-24T12:54:32.755 INFO:teuthology.orchestra.run.smithi190.stderr:tar: /var/lib/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T12:54:32.755 INFO:teuthology.orchestra.run.smithi190.stderr:tar: Error is not recoverable: exiting now 2024-03-24T12:54:32.755 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T12:54:32.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T12:54:32.871 INFO:teuthology.orchestra.run.smithi029.stdout:2024-03-24T12:30:00.000318+0000 mon.a (mon.0) 922 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-03-24T12:54:32.872 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T12:54:32.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[SEC\]' /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T12:54:32.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]' /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T12:54:32.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[WRN\]' /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-24T12:54:33.043 INFO:teuthology.orchestra.run.smithi029.stdout:2024-03-24T12:30:00.000318+0000 mon.a (mon.0) 922 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull 2024-03-24T12:54:33.043 INFO:tasks.cephadm:Compressing logs... 2024-03-24T12:54:33.043 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T12:54:33.085 DEBUG:teuthology.orchestra.run.smithi190:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T12:54:33.125 INFO:teuthology.orchestra.run.smithi029.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T12:54:33.125 INFO:teuthology.orchestra.run.smithi029.stderr:: No such file or directory 2024-03-24T12:54:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T12:54:33.134 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/objectstore_tool.7.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr: 57.8% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T12:54:33.135 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T12:54:33.138 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mon.c.log: 93.8% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/objectstore_tool.7.log.gz 2024-03-24T12:54:33.141 INFO:teuthology.orchestra.run.smithi190.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T12:54:33.141 INFO:teuthology.orchestra.run.smithi190.stderr:: No such file or directory 2024-03-24T12:54:33.146 INFO:teuthology.orchestra.run.smithi029.stderr: 91.7% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T12:54:33.153 INFO:teuthology.orchestra.run.smithi190.stderr:gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T12:54:33.153 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T12:54:33.153 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T12:54:33.154 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/objectstore_tool.7.log 2024-03-24T12:54:33.154 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T12:54:33.154 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/objectstore_tool.7.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T12:54:33.154 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T12:54:33.154 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/tcmu-runner.log 2024-03-24T12:54:33.154 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T12:54:33.154 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.log 2024-03-24T12:54:33.154 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T12:54:33.154 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T12:54:33.162 INFO:teuthology.orchestra.run.smithi190.stderr:/var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mon.b.log: 67.6% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/tcmu-runner.log.gz 2024-03-24T12:54:33.286 INFO:teuthology.orchestra.run.smithi190.stderr: 88.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T12:54:33.325 INFO:teuthology.orchestra.run.smithi190.stderr: 81.4% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T12:54:33.360 INFO:teuthology.orchestra.run.smithi190.stderr: 85.2% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T12:54:33.378 INFO:teuthology.orchestra.run.smithi190.stderr: 90.2% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T12:54:33.379 INFO:teuthology.orchestra.run.smithi029.stderr: 94.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T12:54:33.447 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T12:54:33.455 INFO:teuthology.orchestra.run.smithi190.stderr: 91.1% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T12:54:33.548 INFO:teuthology.orchestra.run.smithi190.stderr: 92.3% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T12:54:33.570 INFO:teuthology.orchestra.run.smithi029.stderr: 91.2% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T12:54:33.682 INFO:teuthology.orchestra.run.smithi029.stderr: 94.0% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T12:54:34.430 INFO:teuthology.orchestra.run.smithi190.stderr: 94.5% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/objectstore_tool.7.log.gz 2024-03-24T12:54:35.837 INFO:teuthology.orchestra.run.smithi190.stderr: 91.3% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T12:54:39.182 INFO:teuthology.orchestra.run.smithi029.stderr: 89.6% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T12:54:48.371 INFO:teuthology.orchestra.run.smithi029.stderr: 90.4% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T12:54:48.501 INFO:teuthology.orchestra.run.smithi029.stderr: 91.2% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T12:55:18.086 INFO:teuthology.orchestra.run.smithi190.stderr: 93.8% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T12:55:50.389 INFO:teuthology.orchestra.run.smithi190.stderr: 93.8% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T12:55:50.834 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T12:55:58.354 INFO:teuthology.orchestra.run.smithi190.stderr: 93.9% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T12:55:58.356 INFO:teuthology.orchestra.run.smithi190.stderr: 2024-03-24T12:55:58.356 INFO:teuthology.orchestra.run.smithi190.stderr:real 1m25.255s 2024-03-24T12:55:58.356 INFO:teuthology.orchestra.run.smithi190.stderr:user 3m34.870s 2024-03-24T12:55:58.356 INFO:teuthology.orchestra.run.smithi190.stderr:sys 0m8.575s 2024-03-24T13:15:58.491 INFO:teuthology.orchestra.run.smithi029.stderr: 93.5% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T13:15:58.610 INFO:teuthology.orchestra.run.smithi029.stderr: 93.2% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T13:17:20.985 INFO:teuthology.orchestra.run.smithi029.stderr: 92.9% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T13:18:49.077 INFO:teuthology.orchestra.run.smithi029.stderr: 92.5% -- replaced with /var/log/ceph/bc017092-e9d8-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T13:18:49.079 INFO:teuthology.orchestra.run.smithi029.stderr: 2024-03-24T13:18:49.079 INFO:teuthology.orchestra.run.smithi029.stderr:real 24m15.983s 2024-03-24T13:18:49.079 INFO:teuthology.orchestra.run.smithi029.stderr:user 7m34.513s 2024-03-24T13:18:49.079 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m23.067s 2024-03-24T13:18:49.080 INFO:tasks.cephadm:Archiving logs... 2024-03-24T13:18:49.080 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999/remote/smithi029/log 2024-03-24T13:18:49.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T13:19:56.858 DEBUG:teuthology.misc:Transferring archived files from smithi190:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999/remote/smithi190/log 2024-03-24T13:19:56.860 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T13:20:29.259 INFO:tasks.cephadm:Removing cluster... 2024-03-24T13:20:29.260 DEBUG:teuthology.orchestra.run.smithi029:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bc017092-e9d8-11ee-95cd-87774f69a715 --force 2024-03-24T13:20:29.774 INFO:teuthology.orchestra.run.smithi029.stdout:Deleting cluster with fsid: bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T13:20:30.529 DEBUG:teuthology.orchestra.run.smithi190:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bc017092-e9d8-11ee-95cd-87774f69a715 --force 2024-03-24T13:20:30.726 INFO:teuthology.orchestra.run.smithi190.stdout:Deleting cluster with fsid: bc017092-e9d8-11ee-95cd-87774f69a715 2024-03-24T13:20:31.270 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T13:20:31.270 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T13:20:31.298 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T13:20:31.314 INFO:tasks.cephadm:Teardown complete 2024-03-24T13:20:31.314 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T13:20:31.324 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T13:20:31.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T13:20:31.341 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T13:20:31.445 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T13:20:31.453 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T13:20:31.453 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T13:20:31.456 DEBUG:teuthology.orchestra.run.smithi190:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T13:20:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2024-03-24T13:20:31.470 INFO:teuthology.orchestra.run.smithi190.stderr:bash: line 1: ntpq: command not found 2024-03-24T13:20:31.499 INFO:teuthology.orchestra.run.smithi190.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T13:20:31.499 INFO:teuthology.orchestra.run.smithi190.stdout:=============================================================================== 2024-03-24T13:20:31.499 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +569us[ +569us] +/- 94ms 2024-03-24T13:20:31.499 INFO:teuthology.orchestra.run.smithi190.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -83us[ -83us] +/- 73ms 2024-03-24T13:20:31.500 INFO:teuthology.orchestra.run.smithi190.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 -398us[ -398us] +/- 82ms 2024-03-24T13:20:31.500 INFO:teuthology.orchestra.run.smithi190.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T13:20:31.517 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T13:20:31.517 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2024-03-24T13:20:31.517 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +541us[ +541us] +/- 94ms 2024-03-24T13:20:31.517 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -112us[ -113us] +/- 73ms 2024-03-24T13:20:31.517 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 -411us[ -411us] +/- 81ms 2024-03-24T13:20:31.517 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T13:20:31.518 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T13:20:31.556 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T13:20:31.556 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T13:20:31.721 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T13:20:31.889 DEBUG:teuthology.orchestra.run.smithi190:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T13:20:31.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T13:20:31.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T13:20:31.995 DEBUG:teuthology.orchestra.run.smithi190:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T13:20:32.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T13:20:32.024 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T13:20:32.033 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T13:20:32.093 INFO:teuthology.task.internal:Duration was 4163.905893 seconds 2024-03-24T13:20:32.093 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T13:20:32.205 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T13:20:32.205 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T13:20:32.208 DEBUG:teuthology.orchestra.run.smithi190:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T13:20:32.288 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T13:20:32.292 INFO:teuthology.orchestra.run.smithi190.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T13:20:32.745 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T13:20:32.746 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2024-03-24T13:20:32.746 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T13:20:32.769 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi190.front.sepia.ceph.com 2024-03-24T13:20:32.769 DEBUG:teuthology.orchestra.run.smithi190:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T13:20:32.796 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T13:20:32.797 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T13:20:32.812 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T13:20:32.865 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T13:20:32.865 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T13:20:32.888 DEBUG:teuthology.orchestra.run.smithi190:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T13:20:34.253 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T13:20:34.263 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T13:20:34.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T13:20:34.290 DEBUG:teuthology.orchestra.run.smithi190:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T13:20:34.325 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T13:20:34.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T13:20:34.481 DEBUG:teuthology.orchestra.run.smithi190:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T13:20:34.515 INFO:teuthology.orchestra.run.smithi190.stdout:kernel.core_pattern = core 2024-03-24T13:20:34.542 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2024-03-24T13:20:34.569 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T13:20:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T13:20:34.582 DEBUG:teuthology.orchestra.run.smithi190:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T13:20:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T13:20:34.597 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T13:20:34.616 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T13:20:34.617 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999/remote/smithi029 2024-03-24T13:20:34.617 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T13:20:34.684 DEBUG:teuthology.misc:Transferring archived files from smithi190:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999/remote/smithi190 2024-03-24T13:20:34.684 DEBUG:teuthology.orchestra.run.smithi190:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T13:20:34.725 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T13:20:34.726 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T13:20:34.728 DEBUG:teuthology.orchestra.run.smithi190:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T13:20:34.774 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T13:20:34.785 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T13:20:34.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T13:20:34.952 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T13:20:34.952 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T13:20:34.955 DEBUG:teuthology.orchestra.run.smithi190:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T13:20:34.966 INFO:teuthology.orchestra.run.smithi029.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 13:20 /home/ubuntu/cephtest 2024-03-24T13:20:34.969 INFO:teuthology.orchestra.run.smithi190.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 13:20 /home/ubuntu/cephtest 2024-03-24T13:20:34.970 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T13:20:35.362 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T13:20:35.690 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T13:20:35.715 DEBUG:teuthology.nuke:shortname: smithi029 2024-03-24T13:20:35.716 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T13:20:35.733 DEBUG:teuthology.nuke:shortname: smithi190 2024-03-24T13:20:35.733 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T13:20:35.737 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 12:00:39.711460', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCL40OM2DBOoWWg9YLz+PMTjeo1NtoyPYRIzlHs+Xs6di299FV0gjNRv8vZznlwC129g1h+CqMQEaWtG5tcPhPM='} 2024-03-24T13:20:35.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi190.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617999', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 12:00:39.713185', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDemyXDXAATj1s5F0ACac++1oAZMyNmqRLmaZbHcx6bjd1Fy2d+wcDa4wN6WmHh17bY4eAAEiWlxYgPOo1/zppo='} 2024-03-24T13:20:35.784 INFO:teuthology.orchestra.console.smithi029:Power off 2024-03-24T13:20:35.784 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T13:20:35.809 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2024-03-24T13:20:35.809 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T13:20:35.824 INFO:teuthology.orchestra.console.smithi190:Power off 2024-03-24T13:20:35.825 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T13:20:35.833 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-24T13:20:35.848 DEBUG:teuthology.orchestra.console.smithi190:power off output: Chassis Power Control: Down/Off 2024-03-24T13:20:35.848 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T13:20:35.871 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-03-24T13:20:39.834 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T13:20:39.873 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T13:20:39.945 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2024-03-24T13:20:39.983 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is on 2024-03-24T13:20:43.950 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T13:20:43.983 DEBUG:teuthology.orchestra.console.smithi190:pexpect command: ipmitool -H smithi190.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T13:20:44.060 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2024-03-24T13:20:44.093 DEBUG:teuthology.orchestra.console.smithi190:check power output: Chassis Power is off 2024-03-24T13:20:44.161 INFO:teuthology.orchestra.console.smithi029:Power off completed 2024-03-24T13:20:44.194 INFO:teuthology.orchestra.console.smithi190:Power off completed 2024-03-24T13:20:44.298 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v2only root} duration: 4163.905893325806 failure_reason: '"2024-03-24T12:30:00.000318+0000 mon.a (mon.0) 922 : cluster 3 [WRN] PG_BACKFILL_FULL: Low space hindering backfill (add storage if this doesn''t resolve itself): 2 pgs backfill_toofull" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-24T13:20:44.298 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T13:20:44.385 INFO:teuthology.run:FAIL